Jev approved a product claim that turned a 10 kg table-load limit into 100,000 grams. That is 100 kg: ten times the limit in the supplied evidence. It made the same mistake in English and Swedish, with reported confidence of 0.95 and 0.92.
The overall score was 98 out of 100. The wrong approvals tell me more about using this model in an e-commerce workflow than that headline number does.
In my first TypeSafe Jev review, I had not used the API. After getting access, I tried 24 simple product-copy checks. Jev matched all 24 assigned labels. I then made the task harder: more products, mixed claims, missing evidence, variant traps, Swedish copy, and instructions hidden inside the text being checked.
This article reports that second experiment. The product references are real; the candidate descriptions are constructed test cases. None of the incorrect copy below is an allegation about a manufacturer's own listing.
The harder test
From easy checks to edge cases
The score needs
the context.
100 recorded answers. 50 paired scenarios. English and Swedish. Constructed cases, not a live store audit.
Different datasets. This is a difficulty comparison, not a before-and-after model improvement.
Labels were fixed before inference and reviewed by AI agents, not independently adjudicated by humans. Paired translations are correlated observations.
74,486 reported input tokens. Estimated cost uses published input pricing; it is not an invoice amount.
I prepared 50 scenarios and paired each English version with a Swedish version, giving 100 snippets across four product references. Both versions received the same English source facts. These are 50 paired scenarios, not 100 independent products or a random sample of a shop's catalog.
The source set covered an Apple 20W adapter, one exact Logitech M185 variant, the Raspberry Pi 5 board, and an IKEA UK LACK table. I supplied selected facts, not entire product pages.
The test asked Jev to return one of three decisions:
| Decision | Evidence rule |
|---|---|
| --- | --- |
| Supported | Every product claim follows from the supplied facts. |
| Contradicted | At least one claim conflicts with an unambiguous applicable fact. |
| Insufficient evidence | At least one claim is unestablished, without an unambiguous contradiction. |
An unsupported warranty promise should not become a proven falsehood merely because the source omits warranty terms. An actual contradiction, however, should still count even when the same paragraph also contains an unsupported claim.
I fixed the labels before inference. A second agent pass reviewed the cases and English–Swedish meaning before the run. That is AI-assisted label review, not independent human annotation. The questions explicitly covered unresolved source conflicts and embedded commands; this clarified prompt differs from the first 24-case test.
What made these cases harder?
Each category included correct controls. Rejecting everything would have performed poorly: 28 of the 100 snippets were assigned “supported.” The longer examples were ordinary paragraphs, not a context-window stress test.
The 10 kg mistake
The table source states a maximum load of 10 kg. The English candidate claimed a maximum of 100,000 grams. I assigned “contradicted” before inference because 100,000 grams equals 100 kg.
Jev returned “supported.” Its probability for that answer was 0.97 and its reported confidence was 0.95. For the corresponding Swedish text, it returned “supported” with probability 0.94 and confidence 0.92.
These are two erroneous responses to one underlying numeric trap. They are not two independent discoveries of different failure modes.
After seeing the error, I made five additional requests with the exact English request body and five with the exact Swedish body. All ten again returned “supported.” Reported confidence ranged from 0.90 to 0.96. I kept these post-hoc repeats separate from the original 100-case score.
The repeat check does not explain the model's internal cause. It does show that asking the same question again did not repair this error in those ten requests.
TypeSafe already documents numeric precision as a weakness of Jev 1.13 and recommends doing arithmetic in code. This result supports taking that guidance seriously. The experiment deliberately stressed a naive whole-description checker; it was not a test of a full system with deterministic numeric checks.
Inspect the recorded decisions
Read the claim.
Inspect the decision.
2 matching cases · showing 1
IKEA LACK white side table 55x55 cm, article 304.499.08, UK listing
The top accepts a maximum of 100,000 grams of items. Keep the total within that limit.
Source facts, label rationale & probabilities
Exact facts supplied to the model
This white LACK table, article 304.499.08, is 55 cm long, 55 cm wide, and 45 cm high. Maximum tabletop load is 10 kg. Its top contains particleboard, fibreboard, paint, plastic edging and honeycomb paper filling; the paper filling is 100% recycled. The smaller 35x35 cm LACK is sold separately and can be nested underneath. Care instructions are to wipe with a damp cloth and then dry, and check and retighten assembly fastenings when necessary. Packaging is 55 cm wide, 75 cm long, 5 cm high and 3.59 kg. Product dimensions and package dimensions are distinct.
Manufacturer reference ↗Assigned-label rationale
10000g limit is falsely converted to100000g.
This rationale belongs to the test fixture. Jev did not generate a written explanation.
The explorer shows the candidate copy, assigned label, actual API decision, and probability distribution. The label rationale comes from the fixture; Jev did not write that explanation.
For each case, the supplied facts define the evidence boundary. A claim can be true in the outside world and still be unestablished by that input. One partial-feed example deliberately removed cable information to test whether the checker respected that boundary.
The missing-evidence cases matched all 16 assigned labels, including the two simulated conflict scenarios in both languages. The embedded-instruction cases matched all 14. Those are encouraging observations on this specific set. Fourteen straightforward instruction attempts do not establish security against prompt injection in production.
English and Swedish each matched 49 of 50 labels. Both missed the same numeric scenario. Equal scores here do not establish equal language ability: the Swedish texts were paired translations and the reference facts remained English.
What would a confidence gate actually do?
Higher confidence.
Enough protection?
Only a “supported” answer at or above this threshold would pass this hypothetical gate. All other answers stay in review.
4 supported cases would also be held. Confidence describes the answer distribution; it is not a guarantee of correctness. Moving this slider reuses recorded answers and does not validate a threshold on new data.
Separate repeat check: the same numeric error returned “supported” in 10/10 additional requests, reaching 0.96 confidence. These repeats are excluded from the slider counts.
A common implementation would accept a description only when Jev says “supported” and its confidence clears a threshold. I applied that rule to the recorded answers without changing or rerunning them.
| Minimum confidence | Would pass | Wrongly passed | Held for review |
|---|---|---|---|
| --- | ---: | ---: | ---: |
| 0.90 | 26 | 2 | 74 |
| 0.95 | 22 | 1 | 78 |
| 0.96 | 18 | 0 | 82 |
| 1.00 | 4 | 0 | 96 |
At 0.90, the gate would still accept both incorrect load claims while sending four correctly supported snippets to review. At 1.00, it would hold 24 of the 28 supported snippets.
Choosing 0.96 after inspecting the original answers would appear to remove the two wrong approvals. That is a result on the data used to choose the threshold. In the separate repeat check, the same wrong English answer reached 0.96 and would pass that gate too.
I would not present any of these settings as a validated operating threshold. TypeSafe describes confidence as a statistic derived from the answer distribution. It cannot replace domain-specific evaluation or checks for known failure modes.
Speed and cost, with the measurement boundary intact
The main run used jev-1.13.0 on September 19, 2026. I sent 100 sequential HTTP requests, with one Choice question per snippet. The API reported the pinned model version throughout.
| Main-run measurement | Result |
|---|---|
| --- | ---: |
| Matching assigned labels | 98 / 100 |
| Supported labels matched | 28 / 28 |
| Contradicted labels matched | 34 / 36 |
| Insufficient-evidence labels matched | 36 / 36 |
| Median request time | 304 ms |
| 95th percentile, nearest rank | 401 ms |
| Slowest request | 804 ms |
| Input tokens | 74,486 |
| Output tokens | 5,318 |
Elapsed time starts immediately before the client calls fetch and ends after it reads the response body. It includes network overhead. It excludes source gathering, writing fixtures, human review, and publishing. I retained the first request, which was also the slowest. All 100 requests completed without automatic retries.
At the $0.042 per million input tokens listed by TypeSafe, the main run's calculated input charge is:
74,486 / 1,000,000 × $0.042 = $0.003128412That is approximately $0.00313, not a reconciled invoice. The ten post-hoc repeats used another 7,845 input tokens, adding a calculated $0.00032949 at the same rate. Output tokens were listed as free. Neither figure includes source maintenance, engineering or editorial labor.
The first, simpler run had a 279 ms median over 24 requests. The harder run had longer inputs and a revised question, so I would not attribute the timing difference to any one factor. I did not compare latency or accuracy against another model.
Where this fits in a catalog workflow
Select current facts for the exact product, variant and seller bundle.
Inference is one part of the cost. Source maintenance, false alarms and editor time determine whether this helps a business.
I would start with the exact SKU, variant and seller bundle. A manufacturer's statement about a board does not establish which accessories a reseller packs with it. A specification for a nearby mouse variant does not establish the receiver in this one.
Next, code should handle comparisons that already have structured values: dimensions, quantities, units, and allowed combinations. In the failed load example, converting 100,000 g to 100 kg and comparing it with 10 kg is an arithmetic check. It should not depend on a model's interpretation.
Free-form copy makes extracting the right number its own problem. A real system still needs to identify which number refers to load, distinguish a quotation from a claim, and handle ambiguous units. This experiment did not implement that extraction pipeline. Passing normalized values to ordinary validation code is the design I would test next.
Jev could then review the semantic parts: whether wording drops an accessory condition, broadens compatibility, or promises a feature without evidence. I would expose the claim and source to the catalog editor rather than return a single unexplained quality score.
The first deployment should run alongside the existing review process. Editors would keep making publication decisions while the checker records suggested flags. The business measurements would be missed errors, unnecessary flags, review minutes per product, and time spent maintaining evidence.
A cheap model call can still create expensive work if it sends too many correct descriptions to review. A fast approval can create a worse problem if it silently accepts a wrong promise. Neither consequence appears in the token bill.
My verdict after the harder test
I would continue the pilot. Jev handled most of these bounded claim-versus-source checks, including mixed claims and Swedish variants, and the measured request times were short enough to make an editorial assistant worth investigating.
I would not let this whole-description checker approve product pages on its own. The numeric failure survived ten repeat requests and had high reported confidence. A human review step and ordinary numeric validation address different parts of that risk.
The next meaningful evidence would come from actual catalog edits, independently labeled by a product editor before inference. I would compare a rules-based baseline, Jev, and another model on that untouched set. Until then, 98/100 is a result from a controlled experiment, not a production accuracy claim or demonstrated return on investment.
Method and limits
The API calls used the current HTTP contract and guidance from TypeSafe's official agent skill. I saved the non-secret request bodies, responses, preassigned labels, and timing measurements. The response checks verified allowed verdicts, probability values, probability sums, confidence bounds, and reported input usage.
The source summaries were curated and short. Some explicitly identified fields that they did not establish, which makes the evidence boundary clearer than a messy supplier feed. The test has only four product references and correlated translation pairs. AI agents authored and reviewed the labels; no independent human adjudicator verified them. The examples do not cover full pages, images, merchant-specific policies, or sustained production traffic.
There was one request per main-run snippet. Only the failed pair received the additional ten requests, so that follow-up does not measure repeatability of the whole dataset. I did not tune the prompt after viewing results or change the labels to improve the score.
Frequently asked questions
Is Jev good at checking product descriptions?
It matched 98 of 100 assigned labels in this controlled test. The two mistakes were the English and Swedish versions of the same unit-conversion error. That supports further testing as an editorial aid, not automatic approval of a live catalog.
Can Jev rewrite incorrect copy?
Jev returns typed decisions rather than replacement prose. An editor or text-generation model would write a correction, then check it against the approved facts.
Is a confidence threshold of 0.96 safe?
This experiment does not establish a safe threshold. The original run had no wrong passes at 0.96, but a later exact repeat returned the same wrong answer at that confidence. A threshold needs validation on separate representative data.
Was any customer data involved?
No. The inputs contained public product references, selected facts, and constructed candidate text. The experiment needed no customer identities or orders.
