What We’re Building
A Helm release rollback strategy with safe values promotion is the kind of operational discipline that separates teams who discover broken production releases from their users from teams who catch them in the pipeline. This post builds a complete workflow: structured per-environment values files, pre-promotion diffing and schema validation, and a CI/CD-integrated fallback that automatically reverts to the last known good revision when a release goes sideways.
The core problem is straightforward but surprisingly common. Engineers promote a values change from staging to production, the rollout stalls or surfaces errors, and they’re left with a failed Helm release that wasn’t automatically rolled back because --atomic was never wired in. Helm doesn’t clean up failed upgrades by default. Without an explicit rollback strategy, that release stays broken until someone intervenes manually.
What we’re building solves this at three layers: a structured values promotion convention, a shell-based validation gate that runs before any helm upgrade, and a pipeline pattern that ties health checks to automatic revision reversion. By the end, you’ll have a safe-promote.sh script and a CI pattern you can drop into any Helm-managed cluster.
Prerequisites
Before working through this tutorial, confirm the following are in place:
- A running Kubernetes cluster with
kubectlcontext configured — local (kind, minikube) or remote both work fine. - Helm 3.x installed. Verify with
helm version. This guide does not apply to Helm 2. - The
helm-diffplugin installed. If you haven’t added it yet, run:
helm plugin install https://github.com/databus23/helm-diff- A sample Helm chart you can deploy and modify. Any chart works — we’ll use a generic app chart with a configurable image tag and replica count to demonstrate failures cleanly.
- Basic familiarity with
helm upgrade,helm history, and Kubernetes Deployments. If you’re newer to Helm’s release model, the official Helm usage documentation is worth a read before continuing.
Deploying Your Initial Helm Release with Versioned Values
The foundation of any reliable rollback strategy is a consistent values file convention. Ad-hoc --set flags passed at the command line are invisible to history and impossible to diff meaningfully. Instead, we use named files per environment.
Structure your chart directory like this:
my-app/
├── Chart.yaml
├── values.yaml # base defaults
├── values-dev.yaml # dev overrides
├── values-staging.yaml # staging overrides
├── values-prod.yaml # production overrides
└── values.schema.json # JSON Schema for validationThe values.schema.json file is frequently overlooked but critical for catching type mismatches and missing required keys before they reach the cluster. Helm validates values against this schema during helm lint and helm upgrade when the file is present.
Deploy the initial release to your staging namespace, pinning the chart version explicitly so the revision history is traceable:
export HELM_NAMESPACE=staging
helm upgrade --install my-app ./my-app \
--values ./my-app/values-staging.yaml \
--version 1.0.0 \
--wait \
--timeout 3m0s \
--history-max 10The HELM_NAMESPACE environment variable scopes all subsequent Helm commands to the target namespace without repeating -n on every invocation. The --history-max 10 flag caps the number of stored revisions — each revision is stored as a Kubernetes secret, and unbounded history accumulates quietly in clusters that see frequent deployments.
After the deployment succeeds, verify the revision baseline:
helm history my-app --max 5You should see revision 1 with a deployed status. This is the state we’ll roll back to in the next section.
Simulating a Bad Release and Triggering a Rollback
Understanding what a failed Helm release actually looks like in practice is more useful than reading about it in the abstract. We’ll intentionally push a broken change and walk through the recovery path.
Modify values-staging.yaml to reference a container image tag that doesn’t exist — this is one of the most common real-world failure modes:
image:
repository: my-registry/my-app
tag: "this-tag-does-not-exist"Now attempt the upgrade without --atomic, which is the mistake we’re specifically demonstrating:
helm upgrade my-app ./my-app \
--namespace staging \
--values ./my-app/values-staging.yaml \
--wait \
--timeout 2m0sAfter the timeout expires, Helm marks the release as failed but does not revert it. Check the history to confirm:
helm history my-app --namespace staging --max 5You’ll see revision 2 in a failed state, with revision 1 still listed as the previous superseded release. To recover, extract the values from revision 1 to confirm what you’re rolling back to, then execute the rollback:
# Inspect the values that were active in revision 1
helm get values my-app --namespace staging --revision 1
# Roll back to revision 1
helm rollback my-app 1 --namespace staging --waitThe --wait flag on rollback is important — without it, the command returns immediately and you have no confirmation that the rollback actually completed successfully. After this runs, helm history will show a new revision (revision 3) with a deployed status, representing the successful rollback.
The takeaway here is that helm rollback <release-name> <revision> targets a specific revision stored in Kubernetes secrets — it’s not a destructive operation, and it creates a new revision in the history rather than deleting the failed one.
Implementing a Safe Values Promotion Gate
Manual rollback is useful to understand, but the goal is to never need it in production because the promotion gate caught the problem first. The script below runs schema validation, diffs the source and target values files, previews the live Helm diff against the running release, and only proceeds if all checks pass. If the upgrade still fails, it rolls back automatically.
Save this as safe-promote.sh at the root of your repository:
#!/usr/bin/env bash
# safe-promote.sh — validate and promote Helm values across environments
# Usage: ./safe-promote.sh <chart-dir> <release-name> <from-env> <to-env> <namespace>
set -euo pipefail
CHART_DIR="${1:?chart directory required}"
RELEASE="${2:?release name required}"
FROM_ENV="${3:?source environment required}"
TO_ENV="${4:?target environment required}"
NAMESPACE="${5:?namespace required}"
FROM_VALUES="${CHART_DIR}/values-${FROM_ENV}.yaml"
TO_VALUES="${CHART_DIR}/values-${TO_ENV}.yaml"
SCHEMA="${CHART_DIR}/values.schema.json"
echo "==> Checking prerequisites"
command -v helm >/dev/null 2>&1 || { echo "helm not found"; exit 1; }
helm plugin list | grep -q "diff" || { echo "helm-diff plugin required"; exit 1; }
echo "==> Validating schema for target values: ${TO_VALUES}"
if [[ -f "${SCHEMA}" ]]; then
helm lint "${CHART_DIR}" -f "${TO_VALUES}" --strict
echo " Schema lint passed"
else
echo " WARNING: no values.schema.json found — skipping schema check"
fi
echo "==> Diffing ${FROM_ENV} → ${TO_ENV} values"
diff_output=$(diff "${FROM_VALUES}" "${TO_VALUES}" || true)
if [[ -z "${diff_output}" ]]; then
echo " No value differences detected — promotion skipped"
exit 0
fi
echo "${diff_output}"
echo "==> Previewing Helm diff against live release"
helm diff upgrade "${RELEASE}" "${CHART_DIR}" \
--namespace "${NAMESPACE}" \
-f "${TO_VALUES}" \
--allow-unreleased
echo "==> Capturing current revision before upgrade"
CURRENT_REVISION=$(helm history "${RELEASE}" \
--namespace "${NAMESPACE}" \
--max 1 \
--output json | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['revision'])")
echo " Current revision: ${CURRENT_REVISION}"
echo "==> Promoting release to ${TO_ENV}"
helm upgrade --install "${RELEASE}" "${CHART_DIR}" \
--namespace "${NAMESPACE}" \
--values "${TO_VALUES}" \
--atomic \
--wait \
--timeout 3m0s \
--history-max 10 \
|| {
echo "ERROR: upgrade failed — rolling back to revision ${CURRENT_REVISION}"
helm rollback "${RELEASE}" "${CURRENT_REVISION}" --namespace "${NAMESPACE}" --wait
exit 1
}
echo "==> Promotion to ${TO_ENV} succeeded"
helm history "${RELEASE}" --namespace "${NAMESPACE}" --max 5A few design decisions worth noting. The script captures the current revision before the upgrade attempt so the rollback target is explicit — relying on Helm to infer “previous revision” can be ambiguous when a release has a complex history. The --atomic flag on the upgrade handles the common case automatically, but the explicit helm rollback in the error block is a safety net for edge cases where --atomic itself encounters an error before completing its own rollback.
Skipping helm diff upgrade before applying changes is the most common mistake in Helm-based pipelines. The diff plugin makes the delta between the current live state and the proposed upgrade visible before anything is applied — catching accidental namespace changes, resource deletions, or configuration regressions that aren’t obvious from reading a values file diff alone.
For more patterns around structuring Kubernetes workloads for reliable deployments, see the DevOps_DayS archive for related posts on Kubernetes deployment manifests and GitOps workflows.
Automating Rollback in CI/CD with Health Checks
The script works locally, but the real value comes from integrating it into a pipeline where it runs automatically on every promotion event. The pattern below applies to GitHub Actions, GitLab CI, and Jenkins equally — the logic is the same regardless of the runner.
The key pipeline structure for a staging-to-production promotion looks like this:
# Example: GitHub Actions promotion job (simplified)
promote-to-prod:
runs-on: ubuntu-latest
environment: production
needs: [deploy-staging, integration-tests]
steps:
- uses: actions/checkout@v4
- name: Install Helm
uses: azure/setup-helm@v3
with:
version: '3.14.0'
- name: Install helm-diff plugin
run: helm plugin install https://github.com/databus23/helm-diff
- name: Configure kubeconfig
run: |
echo "${KUBECONFIG_PROD_B64}" | base64 -d > /tmp/kubeconfig
echo "KUBECONFIG=/tmp/kubeconfig" >> $GITHUB_ENV
- name: Run safe-promote
run: |
chmod +x ./safe-promote.sh
./safe-promote.sh \
./charts/my-app \
my-app \
staging \
prod \
production
- name: Post-deploy health check
run: |
kubectl rollout status deployment/my-app \
--namespace production \
--timeout=3mThe post-deploy health check step using kubectl rollout status is a second verification layer on top of --atomic. In practice, --atomic catches failures during the Helm-managed rollout window, but there are scenarios — particularly with readiness probes that have long initial delays — where a deployment appears healthy to Helm but hasn’t actually finished rolling out. The explicit kubectl rollout status call closes that gap.
For pipelines where you need more granular control over what constitutes a healthy release, consider adding a smoke test step between the promotion and the health check. A simple HTTP probe against a known endpoint is often enough to confirm the application is serving traffic before the pipeline marks the deployment as successful.
One operational note on revision history: the --history-max 10 flag limits stored revisions to 10 per release. In a high-frequency deployment environment, this means older revisions become unavailable for rollback. If your team needs to roll back across more than 10 revisions, increase this value or implement an external revision audit log. The Helm upgrade documentation covers all available flags in detail.
What to Do Next
This workflow covers the core promotion and rollback loop, but there are several natural extensions worth exploring depending on your environment’s complexity.
Helm hooks let you run pre-upgrade and post-upgrade Jobs — database migrations, smoke tests, or cache warm-up steps — as part of the Helm release lifecycle. Pairing hooks with --atomic means a failing migration automatically triggers a rollback before the application is updated, which is significantly safer than running migrations out-of-band.
ArgoCD sync waves extend this pattern into a GitOps-native model. Rather than scripting promotion logic in shell, sync waves let you declare the order in which resources are applied and validated within a single sync operation. Combined with ArgoCD’s built-in health checks and the argocd app rollback command, you get revision-aware rollback without maintaining pipeline scripts. The ArgoCD sync waves documentation is the right starting point.
GitOps-native rollback patterns take this further by treating every values change as a Git commit, making rollback equivalent to reverting a commit and letting the GitOps operator reconcile the desired state. This eliminates the gap between what’s in Git and what’s running in the cluster — a gap that manual helm upgrade invocations quietly introduce over time.
If you’re running Helm-managed workloads alongside autoscaling policies, the post on Kubernetes HPA and VPA rightsizing for production autoscaling covers how scaling configuration interacts with deployment rollouts in ways that affect rollback timing and health check thresholds.
