Last quarter our security team asked for “ML-based threat detection” on top of our Loki logging stack. The ask sounded simple until we actually tried to build it. Log volume across our clusters was too big for regex-based LogQL rules to stay useful, and a single global error-rate threshold kept flagging normal traffic bursts in one namespace as anomalies everywhere else. That’s when we built a proper ML anomaly detection pipeline for Loki logs — and learned, the hard way, what actually works and what just burns compute.
What this actually does
Let’s be precise about what “ML for log security” means here, because the phrase gets thrown around loosely. We’re not running deep learning on raw text, and we’re not asking Loki to do anything it wasn’t built for. Loki has zero native ML capability — it’s a log aggregation system with LogQL as its query language. What it gives you is count_over_time, rate, and sum by aggregations that turn unstructured log lines into numeric time series. That’s the feature source.
The actual ML step happens outside Loki: we extract features like error rate, unique IP count, status code entropy, and path cardinality via LogQL, then feed those numbers into something like sklearn.ensemble.IsolationForest or a seasonal ESD decomposition. That’s it. No transformer models reading raw log text, no magic content-understanding.
This approach catches volumetric and behavioral anomalies — brute-force attempts, scanning patterns, exfiltration-shaped traffic spikes. It will not catch a zero-day exploit hidden in a single well-formed request, and it won’t do content-based threat detection without labeled training data. Setting that expectation up front saved us a lot of awkward conversations with security stakeholders who expected something closer to a SIEM’s signature engine. If you’re evaluating this for your stack, check the Loki LogQL documentation first — understanding what aggregations Loki can actually compute determines your entire feature set.
How people use it wrong
The first mistake we made — and I’ve seen at least three other teams make the same one — was piping raw unstructured log lines into an NLP model instead of extracting numeric features via LogQL first. It’s expensive, it’s noisy, and it produces anomaly scores nobody can explain to an auditor. If you can’t tell a security engineer why a score fired, the model is useless in an incident review.
The second mistake: training one global model across every service and namespace. A traffic spike in the payments API during a marketing push looked identical, feature-wise, to a spike in an internal admin tool getting hit by a credential-stuffing bot. One global Isolation Forest flagged both, or neither, depending on which service dominated the training distribution. We stopped using a global model after a genuinely malicious scanning pattern got buried under noise from a legitimate deploy-triggered log burst.
The third mistake is subtler and it bit us in production: ignoring Loki’s ingestion and query limits when pulling training data at the granularity ML needs. Default max_query_series: 500 and ingestion_rate_mb: 4 aren’t generous when you’re querying per-entity time series across hundreds of pods. We hit 429 Too Many Requests mid-pull, more than once, during what we thought was a routine nightly training job. Watch out for this — it fails silently if you don’t check response codes, and you end up training on a truncated, biased dataset without knowing it.
The correct approach
The architecture that actually worked separates three concerns cleanly: querying, feature engineering, and inference. Nothing runs inline inside the log ingestion path — no synchronous inference inside a Promtail pipeline stage. That was mistake number four, actually, before we caught it in staging: doing per-request inference synchronously adds latency to ingestion and risks blocking the pipeline entirely under load.
Instead, we pull metrics-like features via the Loki HTTP API’s /loki/api/v1/query_range endpoint on a fixed schedule — every minute — rather than tailing raw streams. We train one model per entity (service + namespace + pod, tagged consistently via a entity_id label at the Promtail/Vector relabeling stage), not one global model. Here’s the extraction and training script we run as a scheduled job:
#!/usr/bin/env python3
"""
extract_features.py
Pulls aggregated LogQL metrics from Loki and trains a per-entity
Isolation Forest model for anomaly scoring.
Requires: requests, pandas, scikit-learn>=1.3
"""
import requests
import pandas as pd
from datetime import datetime, timedelta
from sklearn.ensemble import IsolationForest
LOKI_URL = "http://loki.internal:3100/loki/api/v1/query_range"
# Read-scoped token only — never use admin creds for this pipeline
HEADERS = {"Authorization": "Bearer <READ_ONLY_TOKEN>"}
def fetch_error_rate(namespace: str, hours: int = 24) -> pd.DataFrame:
end = datetime.utcnow()
start = end - timedelta(hours=hours)
query = f'sum by (entity_id) (rate({{namespace="{namespace}"}} |= "level=error" [1m]))'
params = {
"query": query,
"start": int(start.timestamp()),
"end": int(end.timestamp()),
"step": "60s",
"limit": 5000, # default max_entries_limit_per_query — paginate if hit
}
resp = requests.get(LOKI_URL, headers=HEADERS, params=params, timeout=30)
resp.raise_for_status()
result = resp.json()["data"]["result"]
rows = []
for series in result:
entity = series["metric"].get("entity_id", "unknown")
for ts, val in series["values"]:
rows.append({"entity_id": entity, "ts": int(ts), "error_rate": float(val)})
return pd.DataFrame(rows)
def train_per_entity(df: pd.DataFrame) -> dict:
"""Train one Isolation Forest per entity_id — never one global model."""
models = {}
for entity, group in df.groupby("entity_id"):
if len(group) < 30:
continue # not enough data to train reliably
X = group[["error_rate"]].values
# contamination=0.01 is a reasonable starting default for security logs
model = IsolationForest(contamination=0.01, random_state=42)
model.fit(X)
models[entity] = model
return models
def score_latest(models: dict, df: pd.DataFrame) -> pd.DataFrame:
latest = df.sort_values("ts").groupby("entity_id").tail(1)
latest["anomaly_score"] = latest.apply(
lambda r: models[r["entity_id"]].decision_function([[r["error_rate"]]])[0]
if r["entity_id"] in models else 0.0,
axis=1,
)
return latest
if __name__ == "__main__":
df = fetch_error_rate(namespace="payments")
models = train_per_entity(df)
scored = score_latest(models, df)
print(scored[["entity_id", "error_rate", "anomaly_score"]])
Notice the read-only token in the headers — this is not optional. Log content can carry PII, so granting an ML pipeline broad query access to Loki is itself a data exposure risk worth flagging in your security review. We also persist model output as a labeled metric written back into a time-series store rather than embedding thresholds inline in inference code, so alerting stays declarative and auditable. This is the part teams skip, and it’s exactly why post-incident reviews turn into “why did the model flag this” arguments nobody can win with a black box.
Advanced patterns
Once the base pipeline is stable, the real tuning work starts. Pure ML scores alone produce too many false positives for anyone to trust at 3am. We combine the Isolation Forest score with a rule-based LogQL alert as a weighted vote — the ML score flags “this is statistically weird,” the rule confirms “and it’s actually a rate spike above a sane floor.” Here’s the ruler alert we run alongside the model, deployed straight into Loki’s ruler:
# Loki ruler alert that fires on a simple rate spike — used as a
# rule-based complement to the ML score, NOT a replacement.
# Path: /etc/loki/rules/security-alerts.yaml
groups:
- name: security-anomalies
rules:
- alert: HighErrorRateSpike
expr: |
sum by (entity_id) (
rate({namespace="payments"} |= "level=error" [5m])
) > 5
for: 2m
labels:
severity: warning
source: logql-rule
annotations:
summary: "Error rate spike detected for {{ $labels.entity_id }}"
description: >
Rule-based trigger complements ML anomaly score;
check anomaly_score dashboard before paging on-call.
# Example output row from score_latest() for comparison:
# entity_id error_rate anomaly_score
# payments-api-7f9x 0.42 -0.18 <- flagged (score < 0)
# payments-worker-2a1 0.03 0.09 <- normal
Note the ruler can only fire from LogQL metric queries — it cannot run inference itself. Inference has to live in a separate service or cron job, full stop; there’s no way around that with Loki’s current architecture, per the Loki alerting docs.
For drift, we retrain weekly on a rolling 7-day window. Daily retraining sounded more “real-time” on paper but just added noise from day-to-day variance without meaningfully improving detection — we tested both and dropped daily retraining after two weeks. We also track a KS test on the score distribution week over week to catch concept drift before it silently degrades precision.
For scanning and credential-stuffing patterns invisible to simple counters, we run a lightweight TF-IDF or hashing vectorizer on the path label (extracted via LogQL’s | pattern stage), then cluster with DBSCAN. This caught a low-and-slow endpoint enumeration attack that never spiked error rate enough to trip either the Isolation Forest or the rule-based alert alone.
Performance notes
Range queries over long windows with high-cardinality labels are the single biggest cost driver in this pipeline. Loki’s schema_config index period and chunk_target_size directly affect query latency for the feature extraction step — we run 24h index periods and saw noticeably better query times than the default shorter periods once cardinality grew past a few hundred entities.
Running inference every minute across thousands of entities means your feature-extraction query load can rival your normal dashboard load on Loki. We stopped re-querying Loki every cycle and started caching extracted features in VictoriaMetrics instead — the cost difference was not subtle, closer to an order of magnitude cheaper for the same query volume.
Gotcha worth flagging directly: clock skew between log ingestion timestamp and Loki’s processing timestamp can misalign your feature windows silently. Always key off the _msg timestamp, not scrape time — we lost a day debugging phantom anomalies that turned out to be windowing artifacts, not real signal.
Also resist the urge to retain raw logs longer “for better training data.” Storage cost multiplies linearly with retention, and it buys you almost nothing over just persisting the extracted feature time series, which is orders of magnitude smaller. We wrote more on cost-driven infra tradeoffs like this over at kuryzhev.cloud if you want the broader pattern beyond logging specifically.
Bottom line: ML anomaly detection on Loki logs works well as a statistical layer on top of LogQL aggregations, not as a replacement for them. Get the feature extraction and per-entity baselines right, keep inference out of the ingestion path, and treat the ML score as one input among several — not gospel.
