Question review agent

ONC2-7195 · Replaces the report fixer (fix-question-task-v2) with one pipeline for every course and format. All numbers from prod data, read-only, on 2026-09-28.

What was wrong

• New formats never got reviewed. The fixer threw for every format except mcq_single and skipped every non-medical course, so reports on CFA, CPA, LSAT, Bar and NCLEX's new formats failed or were ignored. A report also sets the question to invalid, and several practice queries only serve validated questions, so a failed or skipped review took the question out of those pools for good.
• It edited the wrong fields. New formats render and grade from content; the old agent only edited question_text, the option rows and the explanation, so a key it "fixed" on a new-format question kept grading against the old key.
• Its key changes were unchecked. It changed the key after 568 of 2650 reports (21%). 67 questions had their key changed after two separate reports, and in 30 of them the second fix undid the first.
• Old machinery. A Claude Agent SDK subprocess with six database-writing tools and a medical-only prompt on Sonnet 4.6; no benchmark.

How it works now

1. Load the whole question. The reported row plus its set (case study, TBS, question set, reading set), hidden parts dropped, images checked to be reachable.
2. Code checks the structure. Per-format checks, and the stored key is submitted to the real grader and must score 100%. Code findings are facts the review must resolve.
3. Blind solve. GPT-6.1-sol answers every keyed part with the key, explanation and report withheld. (The approach the September explanation backfill used for key disputes.)
4. Review agent. An AI SDK ToolLoopAgent on GPT-6.1-sol via AI Gateway (each call capped at 8,000 output tokens, retried and finally sent to GPT-6-sol if it returns nothing parseable), with a calculator and Perplexity search, judges the report against the key, the blind answer and the course's exam conventions, and returns a typed verdict: no change, fix, disable or escalate.
5. Deterministic, checked writes. Fixes are exact text replacements, a new key in the format's own terms, or a redrawn image (below), written into every copy (content, option rows, flattened text, explanation copies). A fix goes live only if: a key changes only when the blind solve of the original question (at 0.6+ confidence) or a structural check disagreed with it; every changed key is reached by two blind solvers from different model families (GPT-6.1-sol and Claude Opus 5.5) at 0.8+ confidence; every part whose text changed is re-solved from the new text and still reaches its key; the question lints clean; and the rebuilt explanation passes the house-format checks. Disabling needs a structural finding or a confident blind-solve disagreement on the item the reviewer blames. Otherwise a person decides.
6. Record and notify. One transaction that refuses a question edited since it was read, never moves a disabled question back into practice, and keeps the previous fields and status in metadata.question_review so a run can be reverted; then audit rows, a Slack summary and the learner email.

Vercel's eve was considered and not used: it is a separate agent server with its own workflow store, sandbox and auth, which would duplicate Trigger.dev for a job that is one bounded review per report.

Image repair

About one learner note in five is about an image (171 of 817). The old fixer could search for, edit or draw a replacement. The new pipeline does it inside the same checks:

1. The reviewer asks. It sees every image with its URL. If one is dead, wrong, unreadable or gives the answer away, it asks for a redrawn one and writes a complete brief: what it shows, and every label, unit and value a chart needs.
2. gpt-image-2.5-flare draws it, another model family checks it. Flare is called through OpenAI directly (the AI Gateway does not serve it); Gemini 3 Pro Image through the gateway is the fallback. Claude Opus 5.5 checks the picture against the brief for missing or wrong labels, bars that do not match their numbers, unreadable text and anything that names the answer. A rejected image is drawn once more with the complaint.
3. The question is solved again with the new image. The blind solve must still reach the key and no image on the question may be broken. Until then the image exists only in memory: nothing is stored or linked.
4. Stored with the write. The image is stored, the row's linked asset and image_details point at it, and the previous ones are kept in the backup, so a revert restores them.

Not redrawn: image_hot_spot questions (their answer regions belong to the current image's pixels) and questions with no image URL at all (nothing says what belonged there). Those are still taken out of practice.

Three runs of the real pipeline on real questions

Each used a real benchmark question with a fault put in by hand, and the real models. Nothing was written.

In the benchmark

Both dead-image questions were redrawn and fixed, and all 19 questions whose image had been removed were still taken out of practice, since nothing says what belonged there. Of the 16 image questions labelled sound because the report was mistaken, 9 were left alone, 4 went to a person and 3 were redrawn. I looked at those three: one figure is titled "Long Straddle Expiration Profit Diagram" on a question asking which position Figure 1 shows, so it named the answer; one shows propylbenzene where the stem says ethylbenzene; the third has tranche labels the reviewer says do not match the question's classes, which I did not check myself. The benchmark counts them as edits to a sound question, but the first two are real defects the label missed.

The redrawn diagram is qualitative (positions, no numbers), which is all the explanation and the question support. The benchmark has only 2 dead-image cases, so its numbers say little about repair; the unit tests and these runs are the evidence.

Benchmark: 363 frozen cases

Built read-only from prod (scripts/question-review/build-benchmark.ts, data in scripts/question-review/data/benchmark.jsonl.gz). Scored per case as correct, to a person (held or escalated when a fix or no change was expected) or harmful (a wrong key applied, a sound question disabled, or a known defect left live).

Every harmful case, read by hand

The scorer calls a case harmful when the final key differs from the label, a sound question is disabled, or a known defect stays live. Each one was checked against the question itself:

Real reports: old agent vs new

Labelled real reports only (both independent solvers agree). These are Indian PG MCQs, the old agent's home ground; on every other course and format the old agent did nothing.

Models on AI Gateway

The same pipeline with each model doing the blind solve and the review, on a stratified 120-case subset (12 per case type). Key changes are confirmed by Claude Opus 5.5 in every run. The three alternatives ran on an earlier build of the checks, so treat the comparison as rough.

By course and format

Every report since 2024

2650 reports (1843 from 480 learners, 807 from the system user) on 2065 questions; 381 questions were reported more than once. 817 learner reports carry a written note, 325 a screenshot. Jev classified every note by what the learner claims (200 name a specific answer; 7 cite a source):

Not in this PR: validating the whole bank

This PR reviews one question at a time, when a learner reports it. A pass over all 113K live question sets is a separate PR: a targeted set of Jev checks per course and question type, run across the bank, with the full review reserved for what Jev flags. Jev (TypeSafe's System One model) told wrong keys from right ones with AUC 0.86 in a first measurement on this benchmark, without reasoning, at about $0.04 per million input tokens.

What else this found

• Fill-in-the-blank grading. The app gives a numeric keypad whenever a key is a number after dropping "$", "," and "%", but the grader kept the "%", so keys like "0.52%" could never be matched. Fixed in the grader in this PR. Keys that embed words ("USD 3.0 million", "163 contracts (sell)") are content defects the review fixes when they are reported.
• Dead images. Some image questions link to files that return 400 from storage; the review records these, redraws the image when the question and explanation say enough to draw it from, and otherwise takes the question out of practice.
• Empty keys. 12 live NCLEX fill-in-the-blank items had an empty answer, so no learner could get them right.