Question review agent

ONC2-7195 · Replaces the report fixer (fix-question-task-v2) with one pipeline for every course and format. All numbers from prod data, read-only, on 2026-09-28.

What was wrong

• New formats never got reviewed. The fixer threw for every format except mcq_single and skipped every non-medical course, so reports on CFA, CPA, LSAT, Bar and NCLEX's new formats failed or were ignored. A report also sets the question to invalid, and several practice queries only serve validated questions, so a failed or skipped review took the question out of those pools for good.
• It edited the wrong fields. New formats render and grade from content; the old agent only edited question_text, the option rows and the explanation, so a key it "fixed" on a new-format question kept grading against the old key.
• Its key changes were unchecked. It changed the key after 568 of 2650 reports (21%). 67 questions had their key changed after two separate reports, and in 30 of them the second fix undid the first.
• Old machinery. A Claude Agent SDK subprocess with six database-writing tools and a medical-only prompt on Sonnet 4.6; no benchmark.

How it works now

1. Load the whole question. The reported row plus its set (case study, TBS, question set, reading set), hidden parts dropped, images checked to be reachable.
2. Code checks the structure. Per-format checks, and the stored key is submitted to the real grader and must score 100%. Code findings are facts the review must resolve.
3. Blind solve. GPT-6.1-sol answers every keyed part with the key, explanation and report withheld. (The approach the September explanation backfill used for key disputes.)
4. Review agent. An AI SDK ToolLoopAgent on GPT-6.1-sol via AI Gateway (each call capped at 8,000 output tokens, retried and finally sent to GPT-6-sol if it returns nothing parseable), with a calculator and Perplexity search, judges the report against the key, the blind answer and the course's exam conventions, and returns a typed verdict: no change, fix, disable or escalate.
5. Deterministic, checked writes. Fixes are exact text replacements or a new key in the format's own terms, written into every copy (content, option rows, flattened text, explanation copies). A fix goes live only if: a key changes only when the blind solve of the original question (at 0.6+ confidence) or a structural check disagreed with it; every changed key is reached by two blind solvers from different model families (GPT-6.1-sol and Claude Opus 5.5) at 0.8+ confidence; every part whose text changed is re-solved from the new text and still reaches its key; the question lints clean; and the rebuilt explanation passes the house-format checks. Disabling needs a structural finding or a confident blind-solve disagreement on the item the reviewer blames. Otherwise a person decides.
6. Record and notify. One transaction that refuses a question edited since it was read, never moves a disabled question back into practice, and keeps the previous fields and status in metadata.question_review so a run can be reverted; then audit rows, a Slack summary and the learner email.

Vercel's eve was considered and not used: it is a separate agent server with its own workflow store, sandbox and auth, which would duplicate Trigger.dev for a job that is one bounded review per report.

Benchmark: 363 frozen cases

Built read-only from prod (scripts/question-review/build-benchmark.ts, data in scripts/question-review/data/benchmark.jsonl.gz). Scored per case as correct, to a person (held or escalated when a fix or no change was expected) or harmful (a wrong key applied, a sound question disabled, or a known defect left live).

Every harmful case, read by hand

The scorer calls a case harmful when the final key differs from the label, a sound question is disabled, or a known defect stays live. Each one was checked against the question itself:

Real reports: old agent vs new

Labelled real reports only (both independent solvers agree). These are Indian PG MCQs, the old agent's home ground; on every other course and format the old agent did nothing.

Models on AI Gateway

The same pipeline with each model doing the blind solve and the review, on a stratified 120-case subset (12 per case type). Key changes are confirmed by Claude Opus 5.5 in every run. The three alternatives ran on an earlier build of the checks, so treat the comparison as rough.

By course and format

Every report since 2024

2650 reports (1843 from 480 learners, 807 from the system user) on 2065 questions; 381 questions were reported more than once. 817 learner reports carry a written note, 325 a screenshot. Jev classified every note by what the learner claims (200 name a specific answer; 7 cite a source):

Not in this PR: validating the whole bank

This PR reviews one question at a time, when a learner reports it. A pass over all 113K live question sets is a separate PR: a targeted set of Jev checks per course and question type, run across the bank, with the full review reserved for what Jev flags. Jev (TypeSafe's System One model) told wrong keys from right ones with AUC 0.86 in a first measurement on this benchmark, without reasoning, at about $0.04 per million input tokens.

What else this found

• Fill-in-the-blank grading. The app gives a numeric keypad whenever a key is a number after dropping "$", "," and "%", but the grader kept the "%", so keys like "0.52%" could never be matched. Fixed in the grader in this PR. Keys that embed words ("USD 3.0 million", "163 contracts (sell)") are content defects the review fixes when they are reported.
• Dead images. Some image questions link to files that return 400 from storage; the review now records these and takes the question out of practice when it depends on the image.
• Empty keys. 12 live NCLEX fill-in-the-blank items had an empty answer, so no learner could get them right.