What We’re Building
Setting up Grafana alerting rules for Docker host and container metrics is one of those tasks that looks straightforward until you’re staring at a silent dashboard while a container is quietly crash-looping in production. The goal here is to build a complete, end-to-end alerting pipeline: Prometheus collects and evaluates rule expressions against cAdvisor and Node Exporter data, Alertmanager routes fired alerts to a notification channel, and Grafana provides both visual alert state on dashboards and its own native alerting layer on top.
By the end of this walkthrough, you’ll have alerting coverage across four critical dimensions: host CPU saturation, host memory pressure, per-container CPU throttling, and container restart loops. Each alert is tuned with a for: duration to avoid the noise that comes from single-spike conditions — a detail that’s easy to skip and painful to fix after the fact.
The architecture is intentionally straightforward: Node Exporter exposes host-level metrics on port 9100, cAdvisor exposes container-level metrics on port 8080 at /metrics, Prometheus scrapes both and evaluates rule files, Alertmanager handles routing and deduplication, and Grafana sits in front of it all for visualization and its own alert evaluation engine.
Prerequisites
Before writing a single rule, confirm that your monitoring stack is fully operational. Attempting to debug alert routing while also troubleshooting a broken scrape target is a reliable way to waste an afternoon.
You need the following services running and reachable:
- Prometheus — scraping both Node Exporter and cAdvisor targets. Verify at
http://localhost:9090/targetsthat both show UP. - Node Exporter — running on port
9100. Spot-check withcurl -s http://localhost:9100/metrics | grep node_cpu_seconds_total. - cAdvisor — running on port
8080. Confirm the metrics endpoint returns data:curl -s http://localhost:8080/metrics | grep container_cpu_throttled. - Grafana — connected to Prometheus as a data source. Navigate to Configuration → Data Sources and verify the Prometheus connection test passes.
- Alertmanager — installed and accessible, typically on port
9093. Its default config lives at/etc/alertmanager/alertmanager.yml.
You’ll also need write access to /etc/prometheus/ to create rule files, and the ability to send a POST request to the Prometheus reload endpoint. If you’re running the stack via Docker Compose, make sure your Prometheus container mounts the rules directory as a volume — a rules file written to the host but not mounted into the container will be silently ignored.
For reference on Prometheus configuration options used throughout this post, the official documentation is at prometheus.io — Alerting Rules.
Configuring Prometheus Alerting Rules for Docker Metrics
Prometheus evaluates alerting rules on a schedule defined per rule group. The rules we’re writing target two distinct metric namespaces: node_* metrics from Node Exporter for host-level signals, and container_* metrics from cAdvisor for per-container behavior. Keeping these in separate groups with their own interval settings gives you independent evaluation cadence and cleaner alert labeling.
Create the rule file at /etc/prometheus/rules/docker_alerts.yml. The image!="" filter on all cAdvisor expressions is important — without it, cAdvisor emits metrics for the host itself and for paused containers, which pollutes your alert results with noise from infrastructure-level entries you don’t want to page on.
# /etc/prometheus/rules/docker_alerts.yml
groups:
- name: docker_host
interval: 1m
rules:
- alert: HighHostCPUUsage
expr: |
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 85
for: 5m
labels:
severity: warning
env: production
annotations:
summary: "High CPU on {{ $labels.instance }}"
description: "Host CPU usage is {{ $value | printf \"%.1f\" }}% for more than 5 minutes."
- alert: HighHostMemoryUsage
expr: |
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
labels:
severity: critical
env: production
annotations:
summary: "Memory pressure on {{ $labels.instance }}"
description: "Available memory is below 10% on {{ $labels.instance }}."
- name: docker_containers
interval: 1m
rules:
- alert: ContainerHighCPUThrottle
expr: |
rate(container_cpu_throttled_seconds_total{image!=""}[5m]) > 0.25
for: 3m
labels:
severity: warning
annotations:
summary: "CPU throttling on container {{ $labels.name }}"
description: "Container {{ $labels.name }} is throttled >25% of the time."
- alert: ContainerMemoryNearLimit
expr: |
(container_memory_usage_bytes{image!=""} /
container_spec_memory_limit_bytes{image!=""}) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "Container {{ $labels.name }} memory near limit"
description: "Memory usage is {{ $value | printf \"%.1f\" }}% of the configured limit."
- alert: ContainerRestarting
expr: |
increase(container_restart_count{image!=""}[15m]) > 3
for: 0m
labels:
severity: critical
annotations:
summary: "Container {{ $labels.name }} is crash-looping"
description: "Container restarted more than 3 times in the last 15 minutes."Note that ContainerRestarting uses for: 0m intentionally. A crash loop is already a sustained condition by the time the increase() threshold is crossed — adding a for: duration here would delay the alert without providing any noise-reduction benefit. For the CPU and memory rules, the for: 5m guard is essential: without it, a brief CPU spike during a container startup sequence will fire and resolve within seconds, generating alert fatigue before anyone has time to act.
Now register the rule file in your Prometheus configuration and validate it before reloading:
# In /etc/prometheus/prometheus.yml, under the top-level config:
rule_files:
- "/etc/prometheus/rules/*.yml"Validate the rule syntax using promtool before touching the running process:
promtool check rules /etc/prometheus/rules/docker_alerts.ymlIf promtool reports no errors, reload Prometheus without a restart. This is safe to run against a production instance:
curl -X POST http://localhost:9090/-/reloadNavigate to http://localhost:9090/rules and confirm all five rules appear with a green OK status. If any rule shows an error state, the expression likely references a metric that doesn’t exist in your environment — go back to the Prometheus metrics explorer and verify the exact label names cAdvisor is emitting on your Docker version.
Connecting Alertmanager and Setting Up Notification Channels
Prometheus evaluating a rule and firing an alert is only half the pipeline. Without Alertmanager receiving those alerts and routing them somewhere actionable, you have a monitoring system that generates alerts into a void. The connection between Prometheus and Alertmanager is configured in prometheus.yml under the alerting: block.
Add the Alertmanager target to your Prometheus configuration:
# In /etc/prometheus/prometheus.yml
alerting:
alertmanagers:
- static_configs:
- targets:
- "localhost:9093"Reload Prometheus again after this change. Then open /etc/alertmanager/alertmanager.yml and configure a receiver. The example below uses a Slack webhook, but the same structure applies to email — swap the slack_configs block for email_configs with your SMTP settings.
# /etc/alertmanager/alertmanager.yml
global:
resolve_timeout: 5m
slack_api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
route:
receiver: "docker-alerts"
group_by: ["alertname", "instance"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: "docker-critical"
continue: false
receivers:
- name: "docker-alerts"
slack_configs:
- channel: "#infra-alerts"
title: "[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}"
text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
- name: "docker-critical"
slack_configs:
- channel: "#infra-critical"
title: "CRITICAL: {{ .CommonLabels.alertname }}"
text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
inhibit_rules:
- source_match:
severity: "critical"
target_match:
severity: "warning"
equal: ["instance"]A few decisions worth explaining here. The group_by: ["alertname", "instance"] setting means that if three containers on the same host are all memory-constrained simultaneously, Alertmanager sends one grouped notification rather than three separate ones. The inhibit_rules block suppresses warning-severity alerts on an instance when a critical alert is already firing for the same instance — this prevents a flood of HighHostCPUUsage warnings from cluttering the channel while a more severe condition is already being handled.
Reload Alertmanager after editing its configuration:
curl -X POST http://localhost:9093/-/reloadVerify the configuration was accepted by checking http://localhost:9093/#/status. The Config tab should reflect your updated receiver definitions. You can also send a test alert directly to Alertmanager to confirm the Slack webhook is working before waiting for a real condition to fire:
curl -X POST http://localhost:9093/api/v2/alerts \
-H "Content-Type: application/json" \
-d '[{
"labels": {"alertname": "TestAlert", "severity": "warning", "instance": "localhost"},
"annotations": {"description": "Manual test alert from Alertmanager API"}
}]'For full Alertmanager configuration reference, see the official Alertmanager configuration documentation.
Creating and Testing Grafana Alert Rules on Dashboards
Grafana’s native alerting engine runs independently of Prometheus alerting rules. Both can coexist — Prometheus handles infrastructure-level alerting with fast evaluation, while Grafana alerts are better suited to business-logic thresholds expressed against dashboard panels where you want visual correlation between the alert state and the underlying graph.
Grafana contact points are configured under Alerting → Contact points, not inside dashboard JSON. This is a common point of confusion when migrating from legacy Grafana alert definitions. Navigate there first and create a contact point that targets the same Slack channel you configured in Alertmanager, or a separate one if you want Grafana alerts routed differently.
To create an alert rule on a panel, open a dashboard panel showing container memory usage, click the panel title, and select Edit. Switch to the Alert tab and click Create alert rule from this panel. Grafana will pre-populate the query from the panel’s existing PromQL expression.
Configure the evaluation group settings carefully. The minimum evaluation interval in Grafana is 10 seconds, but setting it that low on a busy Grafana instance can cause missed evaluations under load — 1 minute is a reasonable default for infrastructure alerts. The Pending period field is Grafana’s equivalent of Prometheus’s for: duration; set it to 5 minutes for memory and CPU alerts to match the Prometheus rules.
For the ContainerMemoryNearLimit equivalent in Grafana, the panel query would look like this:
(container_memory_usage_bytes{image!=""} / container_spec_memory_limit_bytes{image!=""}) * 100Set the threshold condition to IS ABOVE 80 with a pending period of 5 minutes. Under Annotations, add a summary field with the value Container memory near limit on {{ $labels.name }} so the notification is immediately actionable without needing to open Grafana.
To trigger a test alert without waiting for a real threshold breach, temporarily lower the threshold to a value you know the current metrics will exceed — for example, set the memory threshold to 1 percent. Save the rule, wait one evaluation interval, and confirm the alert transitions to Firing state in Alerting → Alert rules. Check that the notification arrives in your configured contact point. Then restore the real threshold and save again.
If the alert fires in Grafana but no notification arrives, the issue is almost always in the notification policy routing — verify under Alerting → Notification policies that the default policy points to the contact point you created, or that a specific label matcher routes your alert correctly.
For additional context on how we approach monitoring pipelines on this site, see the DevOps_DayS monitoring and observability posts covering related Grafana and Prometheus setups.
What to Do Next
With alerting rules firing and notifications reaching the right channels, the immediate next steps are about reducing noise and improving on-call ergonomics rather than adding more alerts.
Silencing rules in Alertmanager let you suppress notifications during planned maintenance windows without modifying your alert configuration. Create them via the Alertmanager UI at /api/v2/silences or through the web interface. Scope silences by label — for example, silence all env=production alerts for a two-hour deployment window rather than disabling the rules themselves.
Alert grouping strategy deserves a second pass once you see how your alerts behave in practice. If you’re receiving grouped notifications that bundle too many unrelated alerts together, tighten the group_by labels. If you’re receiving too many separate notifications, broaden them. The repeat_interval value of 4 hours in the example above is conservative — adjust it based on how quickly your team expects to acknowledge and act on alerts.
On-call integration via PagerDuty or OpsGenie is the natural next step for critical severity alerts. Both support Alertmanager receivers natively. PagerDuty integration requires a service integration key configured under pagerduty_configs in Alertmanager; OpsGenie uses an API key under opsgenie_configs. Route only severity: critical alerts to these receivers — routing warnings to an on-call tool is a reliable path to alert fatigue and ignored pages.
Finally, consider adding recording rules alongside your alerting rules for the heavier PromQL expressions. The memory ratio calculation and CPU idle rate are evaluated repeatedly across dashboards, alert rules, and ad-hoc queries. Precomputing them as recording rules reduces Prometheus query load and makes the alert expressions easier to read and maintain over time.
