Human-in-the-loop review

Human-in-the-loop review starts from an arithmetic problem. A 40,000-SKU catalog with 60 attributes each is 2.4 million values. Nobody is reading them.

So the working definition is narrower than "a person checks the AI's work." The system decides which values a human needs to see and routes only those. Everything else publishes on the strength of its own evidence.

That reframing is the whole game. Three things have to exist for it to work:

• A confidence signal per value. Pulling 3/8-16 out of a labeled spec-table cell is a different evidence event from inferring finish = zinc because the bolt looks silver in a product photo. One score for the whole SKU hides both.
• A threshold per attribute. The line should sit in a different place for thread_pitch than for long_description. One is a fitment fact; the other is prose.
• An escalation path with a name on it. A queue, an owner, and a turnaround expectation. "Someone will look at it" is not a path.

Your PIM stores the reviewed value and its workflow state. The judgment about what deserves scrutiny lives upstream, in the enrichment process. It is the part a buyer evaluating AI enrichment should interrogate hardest.

Confidence should be a function of evidence type and source agreement. Both are inspectable, which is what separates a threshold you can defend from a number you have to trust.

Here is how that plays out on a single real part, a 3/8-16 x 2" Grade 8 hex cap screw:

Evidence type	Example on this SKU	Default handling
Direct extraction, labeled field	thread_size = 3/8-16 from a spec table cell headed "Thread"	Auto-publish
Multi-source agreement	Manufacturer page and two distributor PDPs all state grade = 8	Auto-publish
Single unverified source	One reseller listing says finish = yellow zinc	Review
Inference from category norms	Assigning drive_type = external hex because most of the class is	Review
Source conflict	Manufacturer says length = 2 in; supplier feed says 2.5 in	Escalate
Regulated or safety claim	UL listed, 600V rated, RoHS compliant, Prop 65 status	Always human

Notice that the last row ignores the score entirely. Certification claims are a liability question rather than a confidence question, and the bar for them is 100% or a person.

Tuning is a two-number exercise: the escalation rate (how much is queuing) and the override rate (how often reviewers change what was proposed), where near-zero overrides mean the bar sits too conservatively and routine overrides mean it is too loose and something upstream is broken.

Every catalog has a set of fields where being wrong is expensive in a way that no confidence score can price. Those get a mandatory review gate rather than a threshold.

• Certification and compliance claims. UL, CSA, ETL, RoHS, REACH, NSF, Prop 65. Publishing an unearned "UL listed" is legal exposure.
• Ratings that govern safe use. Voltage, amperage, pressure, torque, load capacity, temperature range.
• Fitment-critical dimensions. Thread pitch, bore, flange class, tolerance. A wrong 3/8-16 versus 3/8-24 is a return and a lost account.
• Identifiers. GTIN, UPC, MPN, ASIN. These are match keys, and a bad one corrupts everything downstream that joins on it.
• Regulatory and logistics classification. HS code, hazmat class, country of origin.
• Commercial fields. Price, MAP, minimum order quantity, lead time.

Everything else is a threshold decision: descriptive copy, feature bullets, search keywords, secondary attributes, category assignment within a known taxonomy.

A gate only holds if reviewers can finish the queue. When too much escalates, approve becomes a reflex, and the audit trail fills with signatures on values nobody read. That leaves you worse off than having no gate at all, because the record still looks trustworthy.

Human-in-the-loop review answers "can I trust AI enrichment" because every value can explain itself afterward. A human touch on its own proves very little. What holds up under challenge is the record left behind.

Each accepted attribute should carry:

Field	Example
Value	600
Attribute	voltage_rating_v
Source	Manufacturer datasheet, page 2, revision C
Method	Direct extraction
Confidence	High
Reviewer	Named person, if it was gated
Timestamp	Date accepted

With that record, a merchandiser disputing a spec has something to check. A supplier claiming you misrepresented their part has something to answer to. When a datasheet is revised, you can find every value sourced from the old revision and re-check those, instead of re-enriching the whole category blind.

The record also changes what the gate costs over time. The first pass through a category is the expensive one, because every ambiguous evidence pattern arrives new and each judgment has to be made from scratch.

Once those decisions are logged, the patterns become recognizable: an unlabeled 50.8 in a supplier feed, a 600V claim with no datasheet page behind it, a finish inferred from photo color. The threshold for each pattern can then be set from what reviewers already decided rather than from a guess, which is why review effort concentrates in the early batches of a category and thins out afterward.

This is also the honest test to put to any enrichment vendor: show me a value, and show me why it is there. If the answer is a score with no provenance behind it, the loop has a number in it where the human should be.