Reducing alert fatigue with Prometheus Alertmanager routing is one of the highest-leverage improvements you can make to an on-call workflow — yet it’s routinely deferred until an engineer misses a critical page because it was lost in a flood of warnings. This post treats the problem systematically: we identify the routing patterns that create noise, build a working configuration that addresses them, and walk through real scenarios where grouping and inhibition do the heavy lifting.
When to use this
The symptoms are familiar: your on-call rotation starts filtering Slack notifications by habit, PagerDuty acknowledgment times creep up, and post-mortems occasionally reveal that a critical alert fired but was dismissed as routine. These are not process failures — they are configuration failures.
Alertmanager’s routing model is built specifically to address this. The core levers are:
- Grouping — collapsing multiple related alerts into a single notification so one flapping node doesn’t produce fifty pages.
- Inhibition — suppressing lower-severity alerts when a parent condition already covers them. If the cluster is down, you don’t need individual pod restart alerts.
- Tiered routing — sending critical alerts to PagerDuty with tight timing, warnings to Slack with generous batching, and informational noise to a null receiver.
You should invest in this configuration when your team operates more than two or three Kubernetes clusters, when a single deployment event produces more than a handful of simultaneous alerts, or when repeat_interval on your root route is set to anything under an hour. That last condition alone — a root-level repeat_interval of one or two minutes — is the single most common source of alert floods we see in production environments.
If you’re also working on the Kubernetes side of observability, the DevOps_DayS archive covers autoscaling, deployment wiring, and GitOps patterns that complement this routing setup.
Setup
Alertmanager ships as a standalone binary and is typically deployed alongside Prometheus, either as a sidecar in Kubernetes or as a systemd-managed service on a dedicated host. For the purposes of this tutorial we assume a working Prometheus installation that is already scraping targets and evaluating rules. What we’re adding is a properly wired Alertmanager instance.
The default configuration file lives at /etc/alertmanager/alertmanager.yml. If you’re running Alertmanager in Kubernetes via the kube-prometheus-stack Helm chart, this file is managed through a ConfigMap — but the YAML structure is identical.
Before touching any routing logic, install amtool, which ships with Alertmanager and is indispensable for validation and silence management:
# Verify amtool can reach your Alertmanager instance
amtool --alertmanager.url=http://localhost:9093 config show
# Validate a config file before applying it
amtool check-config /etc/alertmanager/alertmanager.ymlMake this validation step a habit. A malformed YAML file will cause Alertmanager to reject the reload and continue running the previous configuration — silently, from the operator’s perspective, unless you’re watching the reload endpoint’s response code.
Once you’ve made a configuration change, reload without restarting the process:
curl -X POST http://localhost:9093/-/reloadThe official Alertmanager documentation covers installation options in detail, including the flags needed to enable the reload endpoint (--web.enable-lifecycle).
Configuration (with code)
The configuration below reflects a production-grade routing tree with three receiver tiers, two inhibition rules, and conservative timing values. Read through it as a whole first — the annotations explain the reasoning behind each decision.
# /etc/alertmanager/alertmanager.yml
# Purpose: reduce alert fatigue via grouping, inhibition, and tiered routing
global:
resolve_timeout: 5m
templates:
- '/etc/alertmanager/templates/*.tmpl'
inhibit_rules:
# Suppress instance-level alerts when the entire cluster is down
- source_matchers:
- 'alertname = "ClusterDown"'
- 'severity = "critical"'
target_matchers:
- 'severity =~ "warning|info"'
equal:
- 'cluster'
- 'namespace'
# Suppress warning when a critical for the same service already fired
- source_matchers:
- 'severity = "critical"'
target_matchers:
- 'severity = "warning"'
equal:
- 'job'
- 'instance'
route:
# Root defaults — conservative repeat to avoid flooding
receiver: 'null'
group_by: ['cluster', 'alertname', 'namespace']
group_wait: 45s
group_interval: 10m
repeat_interval: 6h
routes:
# Critical pages go to PagerDuty immediately with tight grouping
- matchers:
- 'severity = "critical"'
receiver: 'pagerduty-critical'
group_wait: 10s
group_interval: 5m
repeat_interval: 1h
continue: false
# Warnings route to Slack, batched generously
- matchers:
- 'severity = "warning"'
receiver: 'slack-warnings'
group_wait: 2m
group_interval: 15m
repeat_interval: 8h
continue: false
# Info and heartbeat alerts are silenced (dropped to null receiver)
- matchers:
- 'severity = "info"'
receiver: 'null'
continue: false
receivers:
- name: 'null'
- name: 'pagerduty-critical'
pagerduty_configs:
- routing_key: '<PAGERDUTY_INTEGRATION_KEY>'
description: '{{ template "pagerduty.default.description" . }}'
- name: 'slack-warnings'
slack_configs:
- api_url: '<SLACK_WEBHOOK_URL>'
channel: '#alerts-warning'
title: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
text: >-
{{ range .Alerts }}
*Instance:* {{ .Labels.instance }} | *Job:* {{ .Labels.job }}
{{ .Annotations.summary }}
{{ end }}
send_resolved: trueA few decisions worth calling out explicitly:
group_by at the root level includes cluster, alertname, and namespace. This means alerts from different clusters never collapse into the same notification — a critical distinction in multi-cluster environments where a namespace name may repeat across clusters.
The matchers syntax (introduced in Alertmanager ≥ 0.22) replaces the deprecated match and match_re keys. If you’re running an older version, the equality matcher severity = "critical" needs to be written as match: { severity: critical }. Mixing old and new syntax in the same file will produce a validation error.
continue: false is explicit on every child route. The default is also false, but stating it removes ambiguity when someone reads the config six months later. The only time you want continue: true is when a single alert genuinely needs to reach multiple receivers — for example, sending a critical alert to both PagerDuty and a Slack audit channel simultaneously.
Inhibition rule equality fields require that the source and target alerts share the same label values for the listed keys. In the cluster-down rule, both cluster and namespace must match — this prevents a cluster outage in production from suppressing unrelated warnings in staging.
Examples
Three scenarios illustrate how this configuration behaves under realistic conditions.
Scenario 1: Node failure cascades. A node in cluster=prod-eu goes down. Prometheus fires KubeNodeNotReady (critical), followed within seconds by KubePodCrashLooping (warning) and KubeDeploymentReplicasMismatch (warning) for every workload that was scheduled on that node. Without inhibition, this produces dozens of notifications. With the second inhibition rule in place, the critical node alert suppresses all warning-severity alerts sharing the same job and instance labels. On-call receives one page, not thirty.
Scenario 2: Planned maintenance window. You’re draining a node for kernel patching and want to suppress alerts for that host for 90 minutes without modifying the config file. Use amtool to create a time-bounded silence:
# Silence all alerts for a specific instance for 90 minutes
amtool silence add \
--alertmanager.url=http://localhost:9093 \
--duration=1h30m \
--comment="Kernel patching node-07" \
instance="node-07.prod-eu.internal"
# List active silences
amtool silence query --alertmanager.url=http://localhost:9093The silence is stored in Alertmanager’s state, not in the config file. It expires automatically and leaves no configuration drift behind.
Scenario 3: Multi-cluster grouping verification. After deploying the configuration, confirm that alerts from different clusters are grouped independently. Query the Alertmanager API to inspect current alert groups:
# List current alert groups with their labels
curl -s http://localhost:9093/api/v2/alerts/groups | \
jq '.[] | { groupLabels: .labels, alertCount: (.alerts | length) }'If you see a single group containing alerts from both prod-eu and prod-us, the cluster label is either missing from your alert rules or not included in the group_by key. Both are fixable — the former in your PrometheusRule definitions, the latter in the route block.
Common mistakes
These are the configuration errors that either re-introduce fatigue or cause alerts to disappear without explanation.
Root-level repeat_interval set too low. A catch-all route with repeat_interval: 1m means every unresolved alert re-notifies every minute. For a cluster with fifty firing alerts, that’s fifty notifications per minute until someone acknowledges. The root route should have a repeat_interval of at least four to six hours. Critical routes can be tighter — one hour is reasonable — but the root must be conservative because it catches everything that doesn’t match a more specific route.
Missing continue: true in fan-out routes. If you need an alert to reach both PagerDuty and a logging receiver, you must set continue: true on the first matching route. Without it, Alertmanager stops evaluating routes after the first match. The alert reaches PagerDuty and never reaches the audit log. This failure mode is silent — no error is logged, the alert is simply not delivered to the second receiver.
Inhibition rules with mismatched equal labels. If the equal field references a label that doesn’t exist on either the source or target alert, the inhibition rule never fires. Always verify that the labels listed in equal are actually present on the alerts you intend to suppress. The amtool check-config command validates YAML syntax but does not validate label existence against live alert data — that requires manual verification against the /api/v2/alerts endpoint.
Using deprecated match syntax alongside matchers. Mixing match and matchers in the same configuration file on Alertmanager ≥ 0.22 produces an error on reload. Migrate entirely to the matchers syntax when upgrading. The official configuration reference documents both syntaxes and their compatibility matrix.
Applying configuration changes without validation. The reload endpoint returns HTTP 200 even if the new configuration is partially invalid in ways that amtool would catch. Always run amtool check-config before posting to /-/reload. In CI/CD pipelines, add this as a pre-apply step to prevent silent rollbacks to the previous configuration.
Forgetting send_resolved: true on Slack receivers. Without this flag, your Slack channel accumulates firing alerts with no resolution notifications. Engineers lose track of which alerts are still active and which have cleared, which is a different form of fatigue — confusion fatigue rather than volume fatigue.
