Zero-Downtime Deployment: CI/CD, Health Checks and Safe Rollbacks

A green CI run proves that a workflow completed, not that every user received a healthy response. Zero-downtime deployment requires an application architecture that can serve requests while a new version becomes ready. A single-process restart on one server may still create a brief interruption. Be precise about the availability guarantee your infrastructure can actually provide.

Build once and deploy a known artifact

Compile and test in CI, then deploy the exact artifact that passed. Record the commit SHA, build identifier and checksum. Do not rebuild different code on the production server after approval. Immutable release directories make it easier to inspect a failed release and return to a known-good version.

Separate preparation from activation

Upload the release into a new directory, install production dependencies if required and validate configuration before switching traffic. Keep the previous release intact. For applications behind a reverse proxy, bring up new instances and wait for readiness before routing traffic to them. A symlink swap alone is atomic at the filesystem level, but it does not make a service restart interruption-free.

Use readiness and liveness correctly

Liveness answers whether a process needs restarting. Readiness answers whether it can serve real requests. A readiness check should verify essential dependencies without becoming an expensive end-to-end test. After activation, inspect HTTP status, error rate, latency and logs. A successful health endpoint is necessary but not always sufficient.

Make systemd part of a controlled rollout

For a simple Linux deployment, systemd can supervise web, API and worker processes. Restart only the services affected by the release, and confirm that each returns to an active state. Workers need special care: pause or drain jobs before replacing code if task execution cannot safely overlap versions. Preserve logs and release metadata for diagnosis.

Rollback is a tested operation

Rollback should restore the previous application revision quickly, but database schema compatibility must be planned in advance. Use expand-and-contract migrations when possible: add compatible fields, deploy new code, migrate data, then remove old fields in a later release. Never promise that reverting a symlink can undo destructive data changes.

Gate releases on evidence

A useful pipeline checks tests, artifact integrity, deployment permissions, readiness and post-release smoke tests. Add a rollback trigger based on objective failure conditions. Capture the release SHA, activation time, health evidence and operator identity. A deployment is complete when the intended service is verified, not when a script prints success.

A release directory layout

A simple layout might use /srv/app/releases/<commit-sha> for immutable code and /srv/app/current as the active pointer. Keep configuration and uploaded files outside release directories. Prepare the next release without touching the active one. Only switch the pointer after its dependencies and configuration have passed checks. Record both the old and new SHA before activation.

Understand the single-server limitation

On one server with a single web process, restarting that process can interrupt active requests even if the symlink changes instantly. To avoid that interruption, run two compatible instances behind a proxy, wait for the new instance to become ready, then drain connections from the old instance. If that architecture is unavailable, describe the deployment as low-downtime rather than zero-downtime.

A deployment gate with real evidence

Before activation, verify tests, artifact checksum, required environment variables, free disk space and migration compatibility. After activation, call the application through the same reverse proxy used by users, not just localhost. Check a representative authenticated workflow if safe test credentials are available. Preserve response codes, timestamps and logs as release evidence.

What rollback can and cannot fix

Rollback restores application code and process configuration, but it cannot automatically undo emails sent, payments captured or destructive migrations. Use backward-compatible schema changes and idempotent operations to keep the previous release functional. When a rollback is triggered, record the failure and prevent an automatic redeploy of the same broken artifact.

Operational ownership and alerts

A healthy release should remain healthy after the deploy script exits. Watch error rates, latency, queue backlog and service restarts for a defined observation window. Alerts need a responsible owner and a documented response. Treat deployment as a reversible state transition with evidence, not a shell command that happens to return exit code zero.

Start simple, then improve

For a small application, reliable releases with a short maintenance window may be more honest than claiming zero downtime. When availability requirements justify it, introduce rolling instances, load-balancer draining and backward-compatible migrations. Engineering discipline comes before infrastructure complexity.

A two-instance deployment walkthrough

Suppose an API serves traffic through Nginx and currently runs version A. The pipeline builds version B and starts it on a separate port, leaving A untouched. A readiness endpoint confirms B has loaded its configuration, can reach essential dependencies and has applied compatible migrations. Nginx then routes new connections to B while existing sessions on A drain. After the observation window, A can be stopped. If B fails, route traffic back to A without waiting for another build.

Deployment invariants worth writing down

Every release should answer five questions: which commit is running, what artifact was built, where the previous version lives, which database migrations ran and what evidence proves the new version is healthy. Record the answer mechanically in deployment metadata. If an operator cannot identify the active SHA, rollback becomes guesswork. If the same artifact was not used in staging and production, test results lose much of their value.

Database compatibility is the hard part

A reversible code release does not guarantee reversible data changes. Use expand-and-contract migrations for fields, indexes and API responses that old and new instances must share during a rolling transition. First introduce compatible structures, then deploy writers and readers, then backfill and finally retire obsolete fields in a later release. Destructive schema changes belong in a separate, explicitly reviewed step, not a hidden startup hook.

Queue workers need their own cutover plan

Web traffic can drain through a proxy; background jobs require separate handling. A worker may be halfway through sending a payment request or generating a report when a process restarts. Store job state durably and make handlers idempotent. Pause job acquisition before deployment when versions cannot safely coexist, let in-flight operations finish or resume, then start the new workers. Never use process termination as a substitute for business-level recovery.

Make health checks representative

A GET /health endpoint that only returns a static JSON object proves the web framework is listening. Readiness checks should cover only essential dependencies, and post-deploy smoke checks should exercise representative public paths and one authorized workflow. Track rate of HTTP errors and queue delays for a short window after activation. Define failure thresholds before releasing so the team is not tempted to reinterpret red metrics under pressure.

Know when the honest answer is a short interruption

A single server with a single systemd service may offer reliable, fast releases but not mathematically zero downtime. Atomic symlink changes do not preserve in-flight requests through a restart. Communicate the real service-level objective and invest in parallel instances only when the product warrants them. Small, observable, reversible deployments are already a serious improvement over manual file replacement.

Related engineering notes

For more on the connected decisions, explore Nginx, Redis and CDN performance, Core Web Vitals field fixes, Modernizing without a rewrite.