The problem is not producing a mapping — it’s believing one
Give a language model two field lists and it will map them. It will map them even when it shouldn’t: when the source field doesn’t exist, when two candidates are equally plausible, when the spec is silent. The output looks identical either way — a tidy row asserting that Country feeds Region — and a reviewer scanning two hundred tidy rows has no signal for which ones are load-bearing guesses. That is the entire engineering problem of LLM data mapping: not generation, but reviewability. A mapping document is a promise someone else implements; every mechanism below exists to keep that promise inspectable.
Mechanism one: every suggestion cites its source, verbatim
Each drafted row must carry the exact quote from the input documents it is based on — not a paraphrase, not a summary, the passage itself. This does three jobs at once. It makes review fast: the reviewer reads the quote next to the rule and judges the source, not the model. It makes fabrication visible: a row that cannot cite a passage is suspect by construction, and rendering the quote verbatim means a weak or off-topic citation exposes itself. And it survives the export — the finished workbook carries the evidence column, so the “why” of every mapping is still answerable months later, by someone who never saw the AI run. Provenance stays honest too: a row a reviewer authored by hand, or one reused from a prior approved decision, says so — it is never passed off as a fresh AI suggestion with fresh evidence.
Mechanism two: confidence is triage, and calibration is checked, not assumed
Every suggestion carries a self-rated confidence. The operative word is self-rated: it is the model’s own assessment, and models can be miscalibrated — on our benchmark cases, confidence has at times been inverted, with medium-confidence rows outperforming high-confidence ones. So the product treats confidence as a triage cue that orders the review queue, never as a verification signal, and the copy says so everywhere it appears. Our benchmark suite explicitly checks calibration monotonicity and warns when it breaks. The honest framing costs some marketing shine and buys the thing that matters: reviewers who read the evidence instead of trusting a number.
Mechanism three: abstention is a first-class output
The most important token an LLM can produce in this domain is the one that says “I don’t know.” A schema for mapping suggestions must make that expressible: in ours, a row can have a null source, an unknown mapping type, no confidence — and a clarifying question in its place. Two design rules follow. The prompt must make abstaining cheap and respectable, because a model penalized for gaps will fill them. And the evaluation must never count an abstention as a failure to produce: our over-production metric counts fabricated rows — confident inventions — as the defect, while an explicit abstention is scored as what it is, honest declining. Teams get this backwards by optimizing recall first; recall optimized against a spec that genuinely lacks the answer is just fabrication with better numbers.
The unglamorous parts that make it hold
- Honest ingestion. If chunking or retrieval dropped part of a document, the run records it and the output says so — a mapping drafted from 80% of a spec must never present itself as a mapping of the spec.
- Schema-validated outputs, or nothing. Model responses are strict-schema constrained and validated before anything touches the database; a parse failure throws, it never persists a best-effort fragment.
- Deterministic checks stay deterministic. Validation — required-target coverage, source-field references, type and cardinality risks — is pure code, zero AI. Using a model to verify a model adds agreement, not correctness.
- Benchmarks gate releases. AI-touching changes run a strict benchmark on curated real-world cases before shipping, and a gate breach is a deliberate, documented decision — never a silent one. The grader penalizes over-production and rewards honest gaps, so the incentive points the same way the product does.
- The human is the correctness gate. Nothing is “done” because the AI said so: every row is reviewed, and the exported workbook records the human sign-off — a record of who reviewed it, not a cryptographic signature.
The takeaway
If you are building or buying LLM-assisted mapping, the questions that separate systems are not “which model” but: can every suggestion show the passage it came from? Is the model allowed — encouraged — to abstain, and does the evaluation reward that? Is confidence presented as triage or laundered into verification? And does a human decision sit between the draft and anything downstream? A wrong mapping is worse than an honest gap; the architecture either encodes that or it doesn’t.