Overview
Ansible rolling deployments with zero downtime solve one of the most persistent operational problems in web infrastructure: how do you update a multi-node application without dropping a single request? The answer is not a maintenance window — it is a controlled, sequential process where each node is removed from the load balancer, updated, health-checked, and returned to rotation before the next batch begins.
This tutorial builds a production-ready playbook that combines three mechanisms: Ansible’s serial directive to define batch size, HAProxy runtime socket commands to drain and restore nodes, and a block/rescue structure that triggers an automatic symlink rollback when post-deploy health checks fail. The result is a deployment pipeline that is both safe and fully idempotent.
The approach described here applies equally to bare-metal fleets and cloud VM groups. If you are running containerized workloads, the same rolling logic translates well — see the related Kubernetes deployment and Argo CD articles on kuryzhev.cloud for that path. For pure application servers managed over SSH, this playbook is the right tool.
Before You Start
Before running anything in production, confirm the following prerequisites are in place.
Ansible version. This playbook uses ansible.builtin fully-qualified collection names (FQCNs) and relies on loop behavior introduced in Ansible 2.10. Run ansible --version and confirm you are on 2.10 or later. Ansible 2.14+ is recommended for stable any_errors_fatal behavior across serial batches.
Inventory structure. Your inventory must define an app_servers group containing all target nodes. Each host entry should carry ansible_host set to its private IP or resolvable hostname. We will reference inventory_hostname inside HAProxy drain commands, so the name must match the backend server name registered in /etc/haproxy/haproxy.cfg.
SSH access. The Ansible control node needs passwordless SSH access to all app_servers hosts and to the HAProxy host referenced as lb_host. The HAProxy commands run via delegate_to: "{{ lb_host }}", so that host must also be reachable and have socat installed.
Load balancer requirements. This tutorial targets HAProxy with the runtime API socket enabled. Confirm the socket path — typically /var/run/haproxy/admin.sock — is accessible by the SSH user on the load balancer host. The socket must have write permissions; running the drain command as root or via sudo is common in restricted environments.
Variables file. Create vars/deploy_config.yml and populate it with at minimum: health_check_url, drain_wait_seconds, artifact_url, artifact_checksum, release_version, previous_version, app_port, lb_backend, haproxy_socket, and lb_host. Centralizing these values prevents scattered overrides and makes promotion across environments straightforward.
Refer to the official Ansible documentation on playbook strategies for a complete reference on how serial interacts with execution strategies before proceeding.
Step 1: Configure Inventory Groups and Serial Batches
The serial directive at the play level is what transforms a standard Ansible play into a rolling deployment. Without it, Ansible runs every task across all hosts simultaneously — which is exactly what you do not want when the goal is zero downtime.
You have two practical options. Using serial: 2 processes exactly two hosts per batch, regardless of fleet size. Using serial: "25%" processes one quarter of the inventory at a time, which scales automatically as the fleet grows. For most production fleets of ten or more nodes, the percentage form is preferable because it keeps batch duration predictable as capacity changes.
Pair serial with max_fail_percentage: 0 and any_errors_fatal: true. The first aborts the current batch if any host in it fails. The second ensures that no subsequent batch begins after a failure is detected anywhere in the run. Together, they enforce a strict fail-fast policy that prevents a bad release from propagating beyond the first affected batch.
A minimal inventory for this setup looks like this:
# inventory/production.ini
[app_servers]
web01 ansible_host=10.0.1.11
web02 ansible_host=10.0.1.12
web03 ansible_host=10.0.1.13
web04 ansible_host=10.0.1.14
[lb]
haproxy01 ansible_host=10.0.1.5
[all:vars]
ansible_user=deploy
ansible_ssh_private_key_file=~/.ssh/deploy_rsa
One important strategy note: do not mix strategy: free with serial. The free strategy allows hosts to advance through tasks independently, which breaks the drain/restore ordering entirely — a node could be re-added to the load balancer before its health check has completed on a slower sibling in the same batch. Ansible’s default linear strategy is correct here and requires no explicit declaration.
Step 2: Integrate Load-Balancer Drain and Restore Tasks
The drain and restore operations are the mechanical heart of zero-downtime deployment. Every node must be invisible to the load balancer during its update window, and it must only return to rotation after it has passed a readiness check.
HAProxy exposes a runtime API via a Unix socket. The drain command disables the backend server entry without requiring a configuration reload or service restart. Setting backend weight to zero via the runtime API is one approach, but the cleaner production pattern is the disable server command, which immediately stops new connection routing to that node while allowing in-flight requests to complete.
The critical implementation detail is delegate_to: "{{ lb_host }}". Because the play runs in the context of each app_servers host, the HAProxy socket command must be explicitly redirected to the load balancer host. Without delegate_to, Ansible would attempt to run the socat command on the application server itself, where neither the socket nor the HAProxy process exists.
Here is the complete playbook, including pre_tasks for drain, the deployment block with rescue, and post_tasks for restore:
# playbooks/rolling_deploy.yml
---
- name: Zero-downtime rolling deployment
hosts: app_servers
serial: "25%"
max_fail_percentage: 0
any_errors_fatal: true
vars_files:
- ../vars/deploy_config.yml
pre_tasks:
- name: Drain node from load balancer
ansible.builtin.command: >
echo "disable server {{ lb_backend }}/{{ inventory_hostname }}"
| socat stdio {{ haproxy_socket }}
delegate_to: "{{ lb_host }}"
changed_when: true
- name: Wait for active connections to drain
ansible.builtin.pause:
seconds: "{{ drain_wait_seconds | default(10) }}"
tasks:
- name: Deploy application artifact
block:
- name: Pull latest release tarball
ansible.builtin.get_url:
url: "{{ artifact_url }}"
dest: "/opt/releases/{{ release_version }}.tar.gz"
checksum: "sha256:{{ artifact_checksum }}"
- name: Extract release to versioned directory
ansible.builtin.unarchive:
src: "/opt/releases/{{ release_version }}.tar.gz"
dest: "/opt/app/releases/{{ release_version }}"
remote_src: true
creates: "/opt/app/releases/{{ release_version }}/app.py"
- name: Atomically symlink current release
ansible.builtin.file:
src: "/opt/app/releases/{{ release_version }}"
dest: /opt/app/current
state: link
- name: Restart application service
ansible.builtin.service:
name: myapp
state: restarted
- name: Wait for application health endpoint
ansible.builtin.uri:
url: "http://{{ ansible_host }}:{{ app_port }}/health"
status_code: 200
register: health_result
retries: 6
delay: 5
until: health_result.status == 200
rescue:
- name: Rollback symlink to previous release
ansible.builtin.file:
src: "/opt/app/releases/{{ previous_version }}"
dest: /opt/app/current
state: link
- name: Restart service on rolled-back code
ansible.builtin.service:
name: myapp
state: restarted
- name: Fail play to halt remaining batches
ansible.builtin.fail:
msg: "Deployment failed on {{ inventory_hostname }}; rolled back to {{ previous_version }}"
post_tasks:
- name: Re-enable node in load balancer
ansible.builtin.command: >
echo "enable server {{ lb_backend }}/{{ inventory_hostname }}"
| socat stdio {{ haproxy_socket }}
delegate_to: "{{ lb_host }}"
changed_when: true
The drain_wait_seconds pause is intentionally simple. For high-traffic environments, replace the fixed pause with an active polling loop that queries the HAProxy stats socket for current session count on the backend server, waiting until it reaches zero before proceeding. The fixed pause is a safe default for most deployments.
Step 3: Validate the Deployment with Smoke Tests and Rollback Hooks
The block/rescue structure in Step 2 already handles the mechanical rollback. This step focuses on making the validation logic robust enough to catch real-world failure modes before they affect the full fleet.
The health check in the block uses ansible.builtin.uri with retries: 6 and delay: 5, giving the application up to 30 seconds to become ready after a service restart. Adjust these values based on your application’s measured cold-start time. For JVM-based services, 60 seconds of retry window is not unusual.
For deeper validation, extend the block with additional URI checks against critical application endpoints — a database connectivity probe, a cache warm-up endpoint, or a version endpoint that confirms the deployed artifact matches the expected release_version. Each additional check increases confidence that the node is genuinely healthy before it re-enters the load balancer pool.
The rescue block performs three actions in sequence: repoint the symlink to the previous release, restart the service on the rolled-back code, and then explicitly call ansible.builtin.fail. That final fail task is essential. Without it, Ansible considers the rescue block a recovery and continues to the next batch — which means a partially failed deployment could silently propagate. The explicit fail ensures any_errors_fatal: true triggers and the entire play halts.
To test rollback behavior without a real failure, temporarily point artifact_url to a known-broken artifact and run the playbook against a staging inventory. Confirm that the rescue block fires, the symlink reverts, the service restarts cleanly, and no subsequent batch begins.
For emergency single-node re-runs after a partial failure, use the --limit flag to target the affected host and pass a corrected release version:
ansible-playbook playbooks/rolling_deploy.yml \
--inventory inventory/production.ini \
--limit web02 \
--extra-vars "release_version=1.4.2 previous_version=1.4.1"
This allows you to remediate a single node without re-running the entire fleet, which is particularly valuable when a deployment partially succeeded before a network interruption caused it to abort mid-run.
Troubleshooting
Drain command hangs or returns a permission error. The most common cause is that the SSH user on the load balancer host does not have read/write access to the HAProxy socket. Check socket permissions with ls -la /var/run/haproxy/admin.sock on the HAProxy host. The socket is typically owned by the haproxy group. Either add the deploy user to that group or configure a sudoers entry for the specific socat command. Confirm socat is installed: which socat.
SSH timeouts mid-run on large fleets. When Ansible processes a batch, it holds SSH connections open for the duration of that batch. Long drain waits or slow artifact downloads can exceed ServerAliveInterval defaults. Add the following to your ansible.cfg to keep connections alive:
[ssh_connection]
ssh_args = -o ServerAliveInterval=30 -o ServerAliveCountMax=10
pipelining = true
Idempotency breaks when re-running after a partial failure. If a previous run updated the symlink but failed before the service restart, re-running the playbook will skip the get_url and unarchive tasks (because the release directory already exists) but will still restart the service — which is the correct behavior. The idempotency break to watch for is using ansible.builtin.command: systemctl restart myapp instead of ansible.builtin.service with state: restarted. The command module always reports changed, which masks real state and can produce misleading output in subsequent runs.
Rescue block fires but service does not come up on rolled-back code. This usually means previous_version is not set correctly, so the symlink points to a non-existent directory. Validate that /opt/app/releases/{{ previous_version }} exists on the target host before the deployment run begins. A pre-flight task that asserts the previous release directory is present is worth adding to pre_tasks for any production deployment.
Play aborts on the first batch but remaining nodes are still serving old code. This is correct and expected behavior when any_errors_fatal: true is set. The nodes that were not yet processed remain on the previous release. After diagnosing and fixing the root cause, re-run with --limit targeting only the failed host, then re-run the full play to complete the remaining fleet.
Going Further
The playbook as built handles the core rolling deployment contract reliably. Several natural extensions are worth considering as your deployment process matures.
Canary batch as first serial step. Ansible supports passing a list to serial, allowing you to define a progressive batch schedule in a single play. For example, serial: [1, "25%", "100%"] deploys to one node first, then 25% of the remaining fleet, then all remaining nodes. This gives you a built-in canary stage without a separate playbook or pipeline step. Monitor error rates on the canary node before the larger batches begin — pair this with an external monitoring check task between serial groups using ansible.builtin.uri against your metrics API.
Ansible Tower or AWX for scheduled and audited runs. Running this playbook manually is acceptable for infrequent deployments, but at scale you want scheduling, credential management, and audit logging. Ansible Tower and its open-source equivalent AWX provide a workflow layer on top of this playbook with no changes required to the YAML itself. Job templates in Tower can enforce which inventory and variable files are used, reducing the risk of a misconfigured manual run.
Blue-green migration path. The rolling model updates nodes in place, which means there is always a period where different nodes serve different release versions simultaneously. If your application cannot tolerate mixed-version traffic — for example, due to database schema changes that are not backward compatible — consider migrating to a blue-green model where the entire inactive environment is updated before any traffic shifts. The drain and restore patterns in this playbook translate directly to blue-green; the difference is that you drain the entire blue group before beginning updates, rather than one batch at a time.
For teams already running Kubernetes, the equivalent of this pattern is a Deployment rollout strategy with maxUnavailable and maxSurge tuning. The Kubernetes rolling update documentation covers the container-native equivalent in detail. If you are managing both VM fleets and Kubernetes clusters, aligning the conceptual model between the two helps maintain consistent operational practices across your infrastructure.
For more on production-grade deployment patterns and infrastructure automation on this site, browse the DevOps tutorials archive on kuryzhev.cloud — particularly the Helm rollback strategy and Argo CD GitOps sync articles, which address the release promotion and rollback problem from the Kubernetes side of the stack.
