SpecSure

Is the spec a shopping agent tells you actually true? SpecSure checks product claims made by AI shopping agents against attribute-level spec snapshots with a deterministic, model-free verifier, and returns PASS / WARN / BLOCK with the evidence behind every verdict.

This is a static, pre-computed demo — it does not accept input. The results below were produced by running the real verifier locally on the public synthetic benchmark (fictional brands and products). SpecSure is not connected to any store and cannot stop a purchase; it only returns a verdict with evidence.

Headline numbers

Unseen handwritten challenge set v1 (120 claims, sealed in git before its single run). Full tables: BENCHMARKS.md.

94.7%

BLOCK precision — deterministic verifier

naive string matcher: 60.8%

0 / 15

safety-critical (S1) errors passed — deterministic

naive string matcher: 6 / 15

P 97.6% / R 91.1%

claim extraction from free text, extractor v2

15-answer synthetic hold-out, single run

not run

LLM-judge comparison

author's decision (no API spend)

How to read the results

PASS FAIL UNKNOWN — per claim. FAIL always cites the snapshot evidence; UNKNOWN means the snapshot cannot decide it.

Severity: S1 safety/legal (ingredients, allergens), S2 invalidates the purchase (compatibility, model, variant, voltage…), S3 degrades the experience.

Per product: S1 FAIL or UNKNOWN → BLOCK; S2 FAIL → BLOCK, S2 UNKNOWN → WARN; S3 FAIL → WARN.

Pre-computed checks on synthetic products

Each card is one product and a set of claims an agent might make about it: true claims from the benchmark plus known, labelled errors (E1 model/generation, E2 variant, E4 compatibility, E5 bundle, E7 ingredient/allergen, E8 regional spec, E3 number/unit).

All claims check out

Mizunagi IC80 Ink Cartridge — electronics.ink_cartridge, JP market · model MZ-IC80-BK · GTIN 2010401000002 · snapshot snap-jp-mz-ic80-bk-2026-07-22

Decision PASS

Claim (as stated)	Product reference	Benchmark label	Verdict	Severity	Outcome	Evidence (path = gold value)
numeric — ink_volume: 0.27 fl oz	`model_number=MZ-IC80-BK`	PASS	PASS within_tolerance	S3	PASS	attributes.ink_volume = {"unit": "mL", "value": 8.0}
model — MZ-IC80-BK	`model_number=MZ-IC80-BK`	PASS	PASS model_matches	S2	PASS	product.model_number = MZ-IC80-BK
variant — color: black	`gtin=2 010401 000002`	PASS	PASS variant_matches	S2	PASS	variant_axes.color = black

A minor bundle error (S3) gives WARN

Kestrelon Power Bank — electronics.power_bank, US market · model KP-20K-BK-US · GTIN 210303000007 · snapshot snap-us-kp-20k-bk-2026-10-12

Decision WARN

Claim (as stated)	Product reference	Benchmark label	Verdict	Severity	Outcome	Evidence (path = gold value)
numeric — capacity_charge: 20000 mAh	`model_number=KP-20K-BK-US`	PASS	PASS within_tolerance	S2	PASS	attributes.capacity_charge = {"unit": "mAh", "value": 20000}
model — KP-20K-BK-US	`gtin=210303000007, model_number=KP-20K-BK-US`	PASS	PASS model_matches	S2	PASS	product.model_number = KP-20K-BK-US
variant — capacity: 20000 mAh	`model_number=Kestrelon KP-20K-BK-US`	PASS	PASS variant_matches	S2	PASS	variant_axes.capacity = 20000 mAh
bundle — includes: キャリングケース	`gtin=2 103030 00007`	FAIL (E5)	FAIL not_in_exhaustive_bundle diagnosis: E5	S3	WARN	bundle.items = ["USB-C cable", "travel pouch", "user manual"]

Fabricated compatibility (S2) blocks the purchase

Halvane Slim Case for Corvel Nova 5 series — electronics.phone_case, US market · model HV-N5-CL · GTIN 210901000003 · snapshot snap-us-hv-n5-cl-2026-08-11

Decision BLOCK

Claim (as stated)	Product reference	Benchmark label	Verdict	Severity	Outcome	Evidence (path = gold value)
numeric — thickness: 0.1 cm	`model_number=hv n5 cl`	PASS	PASS within_tolerance	S3	PASS	attributes.thickness = {"unit": "mm", "value": 1.2}
model — HV-N5	`model_number=ＨＶ－Ｎ５－ＣＬ`	PASS	PASS model_matches_partial	S2	PASS	product.model_number = HV-N5-CL
variant — color: クリア	`gtin=210901000003`	PASS	PASS variant_matches	S2	PASS	variant_axes.color = clear
compatibility — compatible_with: corvel nova 4	`gtin=2 109010 00003`	FAIL (E4)	FAIL listed_incompatible diagnosis: E4	S2	BLOCK	compatibility.incompatible = Corvel Nova 4

Another region's voltage (S2) blocks the purchase

Yukibana Ion Hair Dryer — appliance.hair_dryer, JP market · model YK-HD12-W · GTIN 2011201000001 · snapshot snap-jp-yk-hd12-w-2026-08-18

Decision BLOCK

Claim (as stated)	Product reference	Benchmark label	Verdict	Severity	Outcome	Evidence (path = gold value)
numeric — power_consumption: 1.2 kw	`gtin=2 011201 000001`	PASS	PASS within_tolerance	S3	PASS	attributes.power_consumption = {"unit": "W", "value": 1200}
model — YK-HD12	`model_number=Yukibana YK-HD12-W`	PASS	PASS model_matches_partial	S2	PASS	product.model_number = YK-HD12-W
variant — color: ホワイト	`model_number=YK-HD12-W`	PASS	PASS variant_matches	S2	PASS	variant_axes.color = white
region — voltage: 120 V	`gtin=2 011201 000001`	FAIL (E8)	FAIL out_of_tolerance diagnosis: E8	S2	BLOCK	region_spec.voltage = {"unit": "V", "value": 100}

Previous-generation value and a sibling's model number

Kestrelon 65W USB-C Charger — electronics.charger, JP market · model KC-PD65-BK · GTIN 2010201000004 · snapshot snap-jp-kc-pd65-bk-2026-08-20

Decision BLOCK

Claim (as stated)	Product reference	Benchmark label	Verdict	Severity	Outcome	Evidence (path = gold value)
numeric — depth: 3.1 cm	`gtin=2010201000004`	PASS	PASS within_tolerance	S3	PASS	attributes.depth = {"unit": "mm", "value": 31}
model — KC-PD65-BK	`gtin=02010201000004`	PASS	PASS model_matches	S2	PASS	product.model_number = KC-PD65-BK
variant — color: ブラック	`gtin=2 010201 000004`	PASS	PASS variant_matches	S2	PASS	variant_axes.color = black
numeric — depth: 30 mm	`gtin=2 010201 000004`	FAIL (E1)	FAIL out_of_tolerance diagnosis: E1	S3	WARN	attributes.depth = {"unit": "mm", "value": 31}
model — KC-PD65-WH	`model_number=KC-PD65-BK`	FAIL (E2)	FAIL model_refers_to_other_product diagnosis: E2	S2	BLOCK	product.model_number = KC-PD65-BK product.model_number = KC-PD65-WH

An allergen claim (S1) that contradicts the label

Lumora Chocolate Bar — food.confectionery, JP market · GTIN 2020301000002 · snapshot snap-jp-choco-milk-2026-08-07

Decision BLOCK

Claim (as stated)	Product reference	Benchmark label	Verdict	Severity	Outcome	Evidence (path = gold value)
numeric — net_weight: 1.76 oz	`gtin=2020301000002`	PASS	PASS within_tolerance	S3	PASS	attributes.net_weight = {"unit": "g", "value": 50}
variant — flavor: milk	`gtin=02020301000002`	PASS	PASS variant_matches	S2	PASS	variant_axes.flavor = ミルク
region — market: JP	`gtin=2 020301 000002`	PASS	PASS region_matches	S3	PASS	region_spec.market = {"value": "JP"}
ingredient — contains: NOT peanut	`gtin=2 020301 000002`	FAIL (E7)	FAIL may_contain_conflict diagnosis: E2	S1	BLOCK	ingredients.may_contain = 落花生
numeric — net_weight: 50 oz	`gtin=02020301000002`	FAIL (E3)	FAIL out_of_tolerance diagnosis: E3	S3	WARN	attributes.net_weight = {"unit": "g", "value": 50}

Handwritten development claims (challenge v0)

Paraphrases, rounding, and cases the snapshot cannot decide. The verifier was tuned on this development set, and it still contains a known false alarm (last row), shown here on purpose.

Claim	Product	Note	Expected	Verdict	Severity	Outcome	Evidence
compatibility — compatible_with: ミズナギ PX‐830F	`jp-mz-ic80-c`	カタカナブランド＋Unicode ハイフン	PASS	PASS listed_compatible	S2	PASS	compatibility.compatible = Mizunagi PX-830F
numeric — length: 6.6 ft	`jp-pw-h21-2m`	2 m = 6.56 ft の丸め	PASS	PASS within_tolerance	S2	PASS	attributes.length = {"unit": "m", "value": 2}
ingredient — contains: gluten	`jp-senbei-shoyu`	小麦をグルテンと言う	PASS	PASS declared	S1	PASS	ingredients.items[しょうゆ].contains = 小麦 ingredients.allergens = 小麦
ingredient — contains: NOT milk	`jp-choco-dark`	「乳成分を含む可能性」表示がある	FAIL	FAIL may_contain_conflict	S1	BLOCK	ingredients.may_contain = 乳成分
ingredient — contains: NOT lactose	`jp-choco-milk`	乳糖は辞書外。全粉乳は乳糖を含むが語彙では判断不能	FAIL	UNKNOWN absence_unverifiable(claim_term_unrecognized) · differs from expected	S1	BLOCK	—
region — pa_rating: PA++++	`us-sunscreen-1-4oz`	US 版は PA 表示なし（属性が無いので断定不能）	FAIL	UNKNOWN region_attribute_not_in_snapshot · differs from expected	S2	WARN	—
numeric — volume: 400 mL	`us-shampoo-floral`	13.5 fl oz (400 mL) 表記の丸め	PASS	FAIL out_of_tolerance · differs from expected	S3	WARN	attributes.volume = {"unit": "fl oz", "value": 13.5}