Question review agent

ONC2-7195 · Replaces the report fixer (fix-question-task-v2) with one pipeline for every course and format. All numbers from prod data, read-only, on 2026-09-28.

What was wrong

• New formats never got reviewed. The fixer threw for every format except mcq_single and skipped every non-medical course, so reports on CFA, CPA, LSAT, Bar and NCLEX's new formats failed or were ignored. A report also sets the question to invalid, and several practice queries only serve validated questions, so a failed or skipped review took the question out of those pools for good.
• It edited the wrong fields. New formats render and grade from content; the old agent only edited question_text, the option rows and the explanation, so a key it "fixed" on a new-format question kept grading against the old key.
• Its key changes were unchecked. It changed the key after 568 of 2650 reports (21%). 67 questions had their key changed after two separate reports, and in 30 of them the second fix undid the first.
• Old machinery. A Claude Agent SDK subprocess with six database-writing tools and a medical-only prompt on Sonnet 4.6; no benchmark.

How it works now

1. Load the whole question. The reported row plus its set (case study, TBS, question set, reading set), hidden parts dropped, images checked to be reachable.
2. Code checks the structure. Per-format checks, and the stored key is submitted to the real grader and must score 100%. Code findings are facts the review must resolve.
3. Blind solve. GPT-6-sol answers every keyed part with the key, explanation and report withheld. (The approach the September explanation backfill used for key disputes.)
4. Review agent. An AI SDK ToolLoopAgent on GPT-6-sol via AI Gateway, with a calculator and Perplexity search, judges the report against the key, the blind answer and the course's exam conventions, and returns a typed verdict: no change, fix, disable or escalate.
5. Deterministic, checked writes. Fixes are exact text replacements or a new key in the format's own terms, written into every copy (content, option rows, flattened text, explanation copies). A fix goes live only if: a key changes only when the blind solve of the original question (at 0.6+ confidence) or a structural check disagreed with it; every changed key is reached by two blind solvers from different model families (GPT-6-sol and Claude Opus 5.5) at 0.8+ confidence; every part whose text changed is re-solved from the new text and still reaches its key; the question lints clean; and the rebuilt explanation passes the house-format checks. Disabling needs a structural finding or a confident blind-solve disagreement on the item the reviewer blames. Otherwise a person decides.
6. Record and notify. One transaction that refuses a question edited since it was read, never moves a disabled question back into practice, and keeps the previous fields and status in metadata.question_review so a run can be reverted; then audit rows, a Slack summary and the learner email.

Vercel's eve was considered and not used: it is a separate agent server with its own workflow store, sandbox and auth, which would duplicate Trigger.dev for a job that is one bounded review per report.

Benchmark: 363 frozen cases

Built read-only from prod (scripts/question-review/build-benchmark.ts, data in scripts/question-review/data/benchmark.jsonl.gz). Scored per case as correct, to a person (held or escalated when a fix or no change was expected) or harmful (a wrong key applied, a sound question disabled, or a known defect left live).

Every harmful case, read by hand

The scorer calls a case harmful when the final key differs from the label, a sound question is disabled, or a known defect stays live. Each one was checked against the question itself:

Real reports: old agent vs new

Labelled real reports only (both independent solvers agree). These are Indian PG MCQs, the old agent's home ground; on every other course and format the old agent did nothing.

Models on AI Gateway

The same pipeline with each model doing the blind solve and the review, on a stratified 120-case subset (12 per case type). Key changes are confirmed by Claude Opus 5.5 in every run. The three alternatives ran on an earlier build of the checks, so treat the comparison as rough.

By course and format

Screening every question

A full review costs about $0.035, so the sweep screens first and reviews only what is flagged. Measured on the benchmark's key errors (131) and sound keys (187):

GPT-6-luna costs about $0.00023 per question. Its misses are almost all key-copy mismatches, which the code checks catch, so lint plus the luna solve catch about 96% of key errors. Jev (TypeSafe's System One model) scores AUC 0.86 at telling wrong keys from right ones without reasoning; it misses calculation items, so it adds the non-key defect signal rather than being the key screen. Learners' answer data adds a third signal: 181 questions with 30+ answers where one distractor is picked more than twice as often as the key.

Every report since 2024

2650 reports (1843 from 480 learners, 807 from the system user) on 2065 questions; 381 questions were reported more than once. 817 learner reports carry a written note, 325 a screenshot. Jev classified every note by what the learner claims (200 name a specific answer; 7 cite a source):

Sweep of live questions (partial)

The sweep was stopped after CFA and most of CPA to save gateway credit; nothing from it has been written to prod. Its fixes were proposed while the checks were still being tightened, so the full sweep will review them again before anything is applied.

Proposed fixes: 90 key corrections (mostly CFA numeric keys stored with units, such as "USD 360,000" or "4.0x", that learners could not enter), 108 text corrections and 198 rebuilt explanations. Screen cost $1.0, review cost $12.19 for 3254 sets.

What else this found

• Fill-in-the-blank grading. The app gives a numeric keypad whenever a key is a number after dropping "$", "," and "%", but the grader kept the "%", so keys like "0.52%" could never be matched. Fixed in the grader in this PR. Keys that embed words ("USD 3.0 million", "163 contracts (sell)") are content defects the sweep fixes.
• Dead images. Some image questions link to files that return 400 from storage; the review now records these and takes the question out of practice when it depends on the image.
• Empty keys. 12 live NCLEX fill-in-the-blank items had an empty answer, so no learner could get them right.