Your Grafana dashboard shows hundreds of failed SSH logins per minute from a single IP. The graph spikes at 3 a.m. like clockwork. And yet — nobody’s phone buzzes. No PagerDuty ping, no Slack message, nothing. This is exactly the failure mode we hit last quarter while running Loki SSH brute-force detection across a fleet of bastion hosts, and it took longer than I’d like to admit to figure out why the pipeline was silently blind despite the data being right there in Loki.
Symptoms
The first sign wasn’t an incident — it was a curious engineer poking around in Grafana Explore after a routine audit. They ran a quick query against the auth logs and saw a clear pattern: bursts of sshd[...]: Failed password for entries, clustered from a handful of rotating source IPs, mostly hitting during off-hours. Classic credential-stuffing behavior against SSH. Nothing subtle about it.
What was subtle: zero alerts had fired. No rule had triggered, no receiver had been paged, and the on-call rotation had no idea this had been happening for days. We went to the host directly — journalctl -u ssh showed hundreds of Invalid user and Failed password lines per minute during the attack windows. But the equivalent LogQL query in Loki returned either zero results or a partial subset, missing most of the actual volume.
On top of that, when we tried tightening the query to catch more of it, the Loki ruler started throwing too many outstanding requests and rule evaluation timeouts. That’s usually the tell that a query is scanning far more data than it needs to, which is its own symptom worth flagging separately from the missing-alert problem.
Put together: the raw logs exist, the attack is real, but the detection layer between “logs in Loki” and “human gets paged” has multiple silent failure points. That’s the scenario this post walks through.
Root cause
Loki wasn’t broken. The pipeline was just built with three independent gaps that each looked fine in isolation but combined into total silence.
First, sshd log formats vary depending on distro and OpenSSH version. Failed password for invalid user X from IP port P ssh2 and Failed password for X from IP port P ssh2 are both valid lines from the same daemon, and a regex written to match one variant quietly ignores the other. We had a rule that only caught invalid user attempts — meaning attackers guessing valid usernames like root or ubuntu sailed through undetected while we thought coverage was complete.
Second, our Promtail pipeline never extracted source_ip as a field. Without that, LogQL had no cheap way to group and count attempts per attacker — every query fell back to expensive full-text line filtering across the whole stream, which is exactly what triggered the ruler timeouts.
Third — and this is the one nobody checks until it’s too late — there was no alerting rule in the ruler config at all for one of the two log variants, and the rule that did exist had a team: security label that didn’t match anything in the Alertmanager route.match tree. Alertmanager doesn’t complain about unmatched labels; it just routes to whatever default receiver exists (or nothing), silently.
None of these are exotic bugs. They’re the standard gap between “the logs are in Loki” and “someone actually gets notified,” and I’ve seen variations of this exact gap in three different environments now.
Fix #1
Start with the LogQL query itself. The goal is to reliably match failed SSH attempts regardless of variant, extract the source IP without promoting it to a static label, and turn raw line counts into a rate per attacker.
# Base query: match failed password lines and extract IP via LogQL regex parser
{unit="ssh.service"} |= "Failed password" | regex "from (?P\S+) port"
# Wrapped for rate detection: count attempts per IP over a 5m window
sum by (ip) (
count_over_time(
{unit="ssh.service"}
|= "Failed password"
| regex "from (?P\S+) port"
[5m]
)
) > 10
This is query-time extraction, not ingestion-time labeling — that distinction matters and I’ll come back to it in Prevention. Gotcha: it’s tempting to promote ip to a Promtail static label because it makes queries feel snappier in Grafana. Don’t. On Loki 3.0+, use structured metadata instead if you need it indexed; static labels on attacker IPs will wreck your index size fast (more on that below).
Fix #2
The query fix is useless if Promtail’s pipeline is silently dropping or mis-parsing lines before they even hit Loki. We rebuilt the pipeline stage to handle both sshd variants in a single regex, and — critically — tested it locally before touching production.
# promtail-config.yaml snippet — pipeline stage to reliably parse sshd journald output
scrape_configs:
- job_name: sshd-journal
journal:
json: false
max_age: 12h
labels:
job: varlogs
unit: ssh.service # matches Loki query label filter above
pipeline_stages:
- match:
selector: '{unit="ssh.service"}'
stages:
- regex:
# single regex handles both invalid-user and known-user failures
expression: 'Failed password for (invalid user )?(?P<user>\S+) from (?P<ip>\S+) port (?P<port>\d+)'
- labels:
user: # ok as label - low-ish cardinality vs raw IP
# --- Test output from logcli after deploying the fix ---
# $ logcli query '{unit="ssh.service"} |= "Failed password"' --limit=5
# 2024-05-14T03:12:41Z {unit="ssh.service"} sshd[1922]: Failed password for invalid user admin from 203.0.113.5 port 51422 ssh2
# 2024-05-14T03:12:43Z {unit="ssh.service"} sshd[1923]: Failed password for root from 203.0.113.5 port 51430 ssh2
# 2024-05-14T03:12:45Z {unit="ssh.service"} sshd[1924]: Failed password for admin from 198.51.100.9 port 40211 ssh2
We ran promtail --dry-run --config.file=/etc/promtail/config.yaml --client.url=http://localhost:3100/loki/api/v1/push against a sample log file before deploying — this caught a broken capture group the first time around. Watch out for the classic mistake of bolting a json stage onto plaintext syslog output. sshd doesn’t emit JSON, so a stray json stage just no-ops silently. No error in the Promtail logs, no crash — the pipeline just does nothing useful, and you don’t find out until you go looking for data that isn’t there.
Fix #3
Detection is worthless if it doesn’t route to a human. We defined the rule group directly in the Loki ruler and made sure the labels lined up exactly with Alertmanager’s routing tree.
# loki-rules.yaml — Loki ruler rule group for SSH brute-force detection
# Path: /etc/loki/rules/security-tenant/ssh-bruteforce.yaml
groups:
- name: ssh-bruteforce
interval: 1m # evaluation interval - keep low-cost queries at 1m
rules:
- alert: SSHBruteForceDetected
expr: |
sum by (ip, instance) (
count_over_time(
{unit="ssh.service"}
|= "Failed password"
| regex "from (?P<ip>\S+) port"
[5m]
)
) > 10
for: 2m # require sustained activity, avoid single-spike false positives
labels:
severity: critical
team: security # must match Alertmanager route.match tree exactly
annotations:
summary: "Possible SSH brute-force from {{ $labels.ip }} on {{ $labels.instance }}"
description: >
{{ $value }} failed SSH login attempts detected from {{ $labels.ip }}
in the last 5 minutes on host {{ $labels.instance }}.
runbook_url: "https://kuryzhev.cloud/runbooks/ssh-bruteforce"
- alert: SSHBruteForceInvalidUser
expr: |
sum by (ip, instance) (
count_over_time(
{unit="ssh.service"}
|= "Invalid user"
| regex "from (?P<ip>\S+) port"
[5m]
)
) > 5
for: 1m
labels:
severity: warning
team: security
annotations:
summary: "Repeated invalid-user SSH attempts from {{ $labels.ip }}"
Two details here matter more than they look. The for: 2m requirement stops a single retry burst — say, a CI job rotating deploy keys badly — from paging someone at 3 a.m. And grouping by ip, instance in the sum by clause is what prevents one alert per log line; skip it and you flood the receiver with duplicates for a single ongoing attack. We learned that one the hard way after a test run generated 40 near-identical pages in under a minute.
Prevention
Getting Loki SSH brute-force detection working once is easy. Keeping it working — and keeping Loki’s storage bill sane — takes a few standing rules.
Never promote raw attacker IPs to Loki labels at ingestion time. With thousands of unique IPs hitting a public bastion host, that can blow up index size by 10-50x and your storage cost right along with it. Extract IP at query time with regex, or use structured metadata (stable since Loki 2.9.x, default-on in 3.0) if you genuinely need it queryable without a full scan — see the official Loki structured metadata docs.
Add inhibition rules in Alertmanager so a critical brute-force alert suppresses the lower-severity duplicate for the same host — check the Alertmanager inhibit_rule reference for the exact syntax. Pair detection with action: forward the matched IP from an Alertmanager webhook receiver into a script running fail2ban-client set sshd banip so alerts trigger a block, not just a dashboard glance. And audit your sshd log format after every OpenSSH package bump — a minor version upgrade can silently reformat log lines and break your regex without any error surfacing anywhere.
We’ve covered more log pipeline war stories like this over at kuryzhev.cloud — worth a browse if you’re building out your own observability stack from scratch.
