Situation
Releasing a service meant someone with server access opened SSH, ran git pull, and did the rest by hand. The steps differed per person and per service, and developers waited for an operator.
My role
I built the pipelines, debugged them, standardized the shared workflow templates and wrote the release SOPs.
Results
- All 20+ production services release through GitHub Actions. Nobody logs into a server to deploy anymore.
- About 60% deploy automatically on merge. For the rest, I start the production run (
workflow_dispatch) and confirm it. Same pipeline, same checks; the only manual part is the decision to go. - Under 2% of production deploy runs failed on core services, measured from GitHub Actions run history.
- Release steps are written down, in the workflow and in the SOP, instead of living in one person’s head.
What the pipeline does today (v1)
Before
- developer asks
- operator with server access
- SSH into VPS
- git pull
- remaining steps by hand
By hand, per service; steps varied by person.
v1 · today
- trigger / dispatch
- build + gates
- package
- copy over SSH
- activate on server
- health check
All 20+ services. ~60% trigger automatically; production runs are dispatched and confirmed by me.
From the real workflow files, names and hosts removed. This combines the two most complete pipelines.
- Gate. Production runs start from
workflow_dispatchand stop at once unless the operator types a confirmation word. A concurrency group stops two production deploys from overlapping, and the workflow token is read-only. - Build. PHP and Node.js 22 are set up. Composer and npm dependencies are installed with caching, and Vite builds the frontend assets.
- Check. Laravel Pint, Prettier, ESLint with zero warnings allowed, a TypeScript type check, and the Pest test suite.
- Package. Production-only Composer packages go into one release tarball, excluding
.git, tests,node_modulesand.env. - Deploy. The package is copied to the VPS over SSH. On the server the pipeline:
- backs up the previous
vendor/and built assets (keeping the last ten) - stops the queue workers and Horizon, then extracts the release
- resets
storage/andbootstrap/cachepermissions - rebuilds the config, route, view and event caches
- runs migrations
- reloads PHP-FPM and restarts the workers
- backs up the previous
- Verify. It primes the scheduler heartbeat and then calls the health endpoint. Anything other than HTTP 200 or 302 fails the run. The SSH key is deleted in an always-run step, and a summary (commit, branch, who deployed) is written to the run page.
Sixteen failures that shaped the template
Almost every step above exists because something broke without it. Each failure became a permanent step, so the fix no longer depends on someone remembering it.
Before it ships · 7 failures
Tests crashed on a missing Vite manifest
- Symptom
- Feature tests that render Inertia pages failed with 'Vite manifest not found'. They passed locally because my machine had a build folder.
- Fix
- public/build isn't committed, so CI never had one. The pipeline now builds the frontend before running the tests.
- Step
npm run build → pest
Type checks failed on route helpers that didn't exist yet
- Symptom
- ESLint and the TypeScript check failed on a fresh checkout, complaining about missing route imports.
- Fix
- Wayfinder's generated route files aren't in git either. The pipeline now generates them before linting and building.
- Step
php artisan wayfinder:generate
The test suite died halfway
- Symptom
- Larger suites stopped with PHP's 'Allowed memory size exhausted', seemingly at random.
- Fix
- The test runner gets an explicit memory limit and stops at the first failure, so a real failure is clear and quick to find.
- Step
php -d memory_limit=512M pest --stop-on-failure
Formatting and lint drift kept breaking builds later
- Symptom
- Code merged with style or lint problems passed review, then failed a later build.
- Fix
- Quality gates run before anything is packaged: Pint, Prettier, ESLint with zero warnings allowed, and type checks.
- Step
pint --test · prettier --check · eslint --max-warnings=0
The release tarball tried to include itself
- Symptom
- After an app moved to the repository root, the archive step started packing its own output file into itself.
- Fix
- The tarball is written to /tmp, outside the directory being archived.
- Step
tar -czf /tmp/laravel-release.tar.gz -C . .
Dev packages and tests ended up in production
- Symptom
- Release artifacts carried dev dependencies, test files and node_modules. They were bigger than needed and exposed things production doesn't need.
- Fix
- Composer is reinstalled with --no-dev before packaging, and tests, node_modules, .git and .env are excluded.
- Step
composer install --no-dev --optimize-autoloader
The pipeline still built a service that had moved out
- Symptom
- When the FastAPI backend was split into its own repository, the Laravel pipeline still carried its build and deploy steps.
- Fix
- Each repository now owns exactly one service's pipeline, and the old steps were removed.
While it ships · 6 failures
Production deploys shouldn't be one click away
- Symptom
- A manual trigger with no guard makes an accidental production run too easy.
- Fix
- A guard job fails the run unless the operator types the confirmation word. v2 replaces this with a real reviewer approval.
- Step
if inputs.confirm != 'DEPLOY' → exit 1
Two deploys racing each other
- Symptom
- Two runs that overlap on one server can extract over each other halfway through.
- Fix
- A concurrency group queues production deploys one at a time and never cancels one mid-flight.
- Step
concurrency: { group: production, cancel-in-progress: false }
Permission denied on storage/ and bootstrap/cache
- Symptom
- The deploy failed, or the app threw errors right after, because Laravel couldn't write its cache or logs.
- Fix
- Files there were owned by a different user, left by an earlier manual or elevated step. Every deploy now creates the storage directories, removes stale compiled cache files and resets ownership and permissions.
New code, old config
- Symptom
- After a release the app behaved as if the previous .env or routes were still live.
- Fix
- Cached config, routes, views and events are cleared, then rebuilt from the new release on every deploy.
- Step
artisan optimize:clear → artisan optimize
Queue workers kept running last release's code
- Symptom
- The deploy looked successful, but queued jobs still behaved like the old version.
- Fix
- Horizon and queue workers are long-running processes under Supervisor. The pipeline now signals them to stop and Supervisor starts them on the new code.
- Step
artisan queue:restart · artisan horizon:terminate
PHP served the old files after extraction
- Symptom
- Some requests still hit the previous code for a while after a release.
- Fix
- OPcache was holding the old compiled files. PHP-FPM is config-tested and then gracefully reloaded after every release.
- Step
php-fpm -t && reload php-fpm
After it ships · 3 failures
Green pipeline, broken site
- Symptom
- Every step passed, and the app still returned errors.
- Fix
- The final step calls the app's health endpoint (with a fallback to the root URL) and fails the run on anything but 200/302.
- Step
curl -w '%{http_code}' https://…/up
The scheduler looked dead right after a deploy
- Symptom
- Health checks flagged the scheduler as not running for the first minutes after a release.
- Fix
- The pipeline primes the scheduler heartbeat before running health checks.
- Step
artisan health:schedule-check-heartbeat
Secrets outliving the job
- Symptom
- The deploy key was written to disk on the runner and stayed there when a job failed.
- Fix
- An always-run cleanup step deletes the key whether the job succeeds or fails.
- Step
if: always() → rm -f ~/.ssh/deploy_key
How I narrow a failed run down: first the failing step in the Actions log. Then, on the server, laravel.log and the worker status in Supervisor/Horizon. After that, file ownership on storage/ and bootstrap/cache. Finally memory, on the Grafana dashboard and in the system logs.
What v1 still gets wrong
Known problems in v1:
| v1 behaviour | Why it matters |
|---|---|
Typed DEPLOY confirmation |
A speed bump, not access control: anyone who can run workflows can type it. |
| Extracts over the live directory | A failing step can leave a half-updated app, and there’s no clean rollback. |
Several steps end in || true |
Real failures, including the app’s own health check, get silently ignored. |
| Failed health check only fails the job | The broken release stays live until someone steps in. |
ssh-keyscan on every run |
Trusts whichever host answers first. |
| Some pipelines deploy without tests | Build and deploy share one job, so nothing forces the tests to run first. |
Next iteration: v2 template
Draft v2 release path
- pull requestCI on every PR: Pint, ESLint, types, dependency audit, tests on PostgreSQL
- approveGitHub Environment "production": a required reviewer approves; deploy secrets unlock only then
- release dirTested artifact unpacked to releases/<id>, linked to shared .env and storage/
- migrateBackward-compatible migrations, run before traffic moves
- switchcurrent → releases/<id> by atomic rename; PHP-FPM reload; workers restarted
- healthPoll /up up to 10 times
200 release stays live · old releases pruned (keep 5)
fail current → previous release, automatically
- Real approval. A GitHub Environment with a required reviewer replaces the typed word. The deploy secrets are scoped to that environment, so they don’t exist for the job until someone approves.
- Ship only what CI tested. One reusable CI workflow runs on every pull request, testing against PostgreSQL rather than SQLite. The deploy calls that workflow and ships its artifact.
- Release directories and an atomic switch. Each deploy goes to
releases/<id>, andcurrentflips with a single rename. If the health check fails,currentis pointed back at the previous release automatically. - Fail loudly.
set -Eeuo pipefailreplaces|| true, and the host key is pinned.sudois limited to a singlesystemctl reload.
Deploy script excerpt
# remote-deploy.sh (excerpt): one rename, no half-deployed state
ln -sfn "$new" "$APP_ROOT/.current.tmp"
mv -Tf "$APP_ROOT/.current.tmp" "$CURRENT"
# from here on, any failure puts the previous release back
trap 'switch_to "$previous"; reload_runtime || true' ERR
reload_runtime
trap - ERRThe containerized variant
For services that run in Docker, the same shape uses an image instead of a directory:
- A multi-stage
Dockerfileseparates Composer dependencies, the Vite asset build and the runtime. PHP-FPM runs as a non-root user, and an unprivileged nginx image servespublic/. - One app image runs PHP-FPM, the queue worker (or Horizon) and the scheduler. Docker’s restart policy does the job Supervisor does on a plain VPS.
- Images are tagged with the commit SHA, scanned with Trivy before they’re pushed, and pulled from GitHub’s container registry.
docker compose up --waitblocks until healthchecks pass, and rolling back means starting the previous tag.
Known limit, in both variants: rolling back restores the code, not the database. Migrations have to stay backward compatible with the previous release, so schema changes are split across two deploys.