What it is
You get paged, dig through monitoring, logs, deploys and chat, and fix the cause before the error budget runs out. Then you read a postmortem of your shift. The nine incidents come from real public postmortems: a leaking connection pool, a full disk, an expired certificate, a retry storm, replica lag and more.
How it ships
- Pull request: lint, type checks, 700+ tests, Playwright, shellcheck and tests for the deploy script. Both Docker images build.
- Merge to main: API and web images are tagged with the commit SHA and pushed to GHCR.
- Deploy: the job joins my tailnet with a tag that reaches one host on port 22, and runs
deploy.sh. Players come in through a Cloudflare Tunnel, so no port is open at home. deploy.sh: migrates the database, starts the release, replays a known game through the whole stack as a smoke test, and rolls back to the last good release if anything fails.
Game day, 28 September 2026
I merged a release whose readiness check always failed, with the hypotheses written in the pull request first.
| Hypothesis | Result | Evidence |
|---|---|---|
| The smoke test rejects the broken release | Pass | smoke test failed for 5d73169 |
deploy.sh rolls back to the last good release |
Pass | rollback ok: fd8bcf9 |
| The deploy job goes red | Pass | exit code 1 |
| Players see no impact | Pass | the site served fd8bcf9 throughout |
Only the API and web containers were recreated. Postgres kept running. The revert went out as a normal pull request and deployed green.
Scores the browser can’t fake
Each run is a seed plus a list of actions. The API replays the list with the same deterministic engine and stores its own score. The browser’s score is ignored, and impossible actions are rejected.
What I’d do next
- A public status page for the game, fed by the checks that gate the deploy.
- An SSH forced command for the deploy user, so the CI key can only run the deploy script.