The scenario
Log anomaly detection became a priority for us after a week where our on-call rotation got paged 200+ times, and roughly 90% of those pages were noise. We had a set of Alertmanager rules built on grep-style regex matching against Loki: anything containing ERROR|CRITICAL fired a page. It worked fine when we had three services. At thirty services, it turned into background radiation that everyone learned to ignore, which is the worst possible outcome for an alerting system.
The deeper problem wasn’t just volume — it was that threshold-based alerting misses the interesting failures. A service logging perfectly normal INFO lines at 10x its usual rate right before a crash never trips an ERROR-based rule. Neither does a service that goes quiet when it should be chatty. Static regex rules only catch things you already knew to look for. What we actually needed was something that understood “this pattern doesn’t look like this service’s normal behavior,” regardless of log level.
So the goal became: build a lightweight, self-hosted pipeline that scores rolling windows of log data for statistical anomalies, and only escalates to a human when the pattern is genuinely novel. No SaaS log analytics bill, no GPU, and no black-box vendor model deciding what counts as weird for us.
Prerequisites
Before touching any code, make sure the stack lines up. We ran this on Python 3.11, scikit-learn 1.4.2, FastAPI 0.110, Vector 0.37.1 as the log shipper, and Docker 25.x with Compose v2 for orchestration. Pin these versions explicitly — scikit-learn’s IsolationForest default behavior for contamination has shifted across releases, and you don’t want your anomaly threshold drifting silently after a routine dependency bump.
You’ll also need a log source that’s already structured. We assumed JSON logs flowing into Loki, or at minimum a file path like /var/log/app/*.json, with consistent timestamp, level, service, and message fields. If your logs are still unstructured plaintext, fix that first — feature extraction downstream depends entirely on field consistency.
Finally, get write access to your Alertmanager config or, simpler, a Slack incoming webhook URL for the initial rollout. If you want the optional semantic triage layer described later, you’ll also need an OpenAI-compatible API key (we used gpt-4o-mini for cost reasons — more on that below).
Step 1 — Ship and normalize logs into a feature-ready format
The first move is getting raw logs into something numeric. We configured Vector with a remap transform (VRL syntax) to parse JSON, pull out level, service, message_length, and error_code, and drop noisy debug fields we didn’t need. Instead of streaming raw text downstream, we write parsed output into rolling 5-minute buckets, persisted to a local file that gets read by the training and scoring scripts.
Here’s the timestamp normalization pattern we standardized on inside the remap block:
[transforms.normalize]
type = "remap"
inputs = ["parse_json"]
source = '''
.timestamp = parse_timestamp!(.timestamp, format: "%+") |> to_unix_timestamp
.level = downcase(.level)
del(.debug_context)
'''Gotcha worth calling out explicitly: timezone mismatches between the application and the collector silently break 5-minute bucketing. If your app logs local time and Vector assumes UTC, your windows drift and features become inconsistent without throwing a single error. Normalize both sides to UTC in the remap step, every time, no exceptions.
Step 2 — Extract features and train the anomaly model
With normalized windows in hand, we build per-service, per-5-minute features: log count, ERROR ratio, unique message-template count (via drain3 for log template clustering), and average message length. This is where a lot of the real signal lives — a service suddenly producing dozens of never-before-seen message templates is a stronger anomaly indicator than a raw error count.
Below is the training script we run weekly to refresh the baseline:
# train_anomaly_model.py
# Trains an IsolationForest on normalized 5-min log windows produced by Vector.
# Run weekly via cron/CI to keep the baseline fresh.
import json
import joblib
import pandas as pd
from datetime import datetime
from sklearn.ensemble import IsolationForest
from drain3 import TemplateMiner
from drain3.template_miner_config import TemplateMinerConfig
FEATURE_COLUMNS = [
"log_count", "error_ratio", "unique_templates", "avg_msg_length"
]
def load_windows(path: str) -> pd.DataFrame:
# Vector writes newline-delimited JSON windows to this file
records = [json.loads(line) for line in open(path)]
df = pd.DataFrame(records)
df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True) # always UTC
return df
def extract_features(df: pd.DataFrame, miner: TemplateMiner) -> pd.DataFrame:
grouped = df.groupby([pd.Grouper(key="timestamp", freq="5min"), "service"])
rows = []
for (window, service), group in grouped:
templates = {miner.add_log_message(m)["cluster_id"] for m in group["message"]}
rows.append({
"window": window,
"service": service,
"log_count": len(group),
"error_ratio": (group["level"] == "ERROR").mean(),
"unique_templates": len(templates),
"avg_msg_length": group["message"].str.len().mean(),
})
return pd.DataFrame(rows)
def audit_training_window(df: pd.DataFrame, incident_dates: list[str]):
# Common mistake #1 guard: refuse to train if incident dates overlap the window
for d in incident_dates:
if pd.to_datetime(d, utc=True) in df["window"].values:
raise ValueError(f"Training window contains known incident date {d} — exclude it first")
if __name__ == "__main__":
config = TemplateMinerConfig()
config.load("drain3.ini")
miner = TemplateMiner(persistence_handler=None, config=config)
df = load_windows("/data/vector/windows.jsonl")
features = extract_features(df, miner)
audit_training_window(features, incident_dates=["2024-05-03"])
X = features[FEATURE_COLUMNS].fillna(0)
model = IsolationForest(n_estimators=200, contamination=0.02, random_state=42)
model.fit(X)
version = datetime.utcnow().strftime("%Y-%m-%d")
joblib.dump(model, f"/models/iforest_{version}.joblib")
print(f"Saved model iforest_{version}.joblib trained on {len(features)} windows")The mistake we made the first time: we trained on a window that included a past incident. The model learned that incident’s pattern as “normal” and stopped flagging near-identical events afterward. Always audit your training window before fitting — the audit_training_window function above exists specifically because we got burned by this once and didn’t want to repeat it. Also don’t skip persisting the drain3_state.json file across redeploys; losing it resets template IDs and corrupts feature continuity between runs.
Step 3 — Serve the model behind a scoring API
Once you have a trained model, wrap it in a small FastAPI service exposing a /score endpoint that accepts a feature vector and returns an anomaly score plus a boolean flag. A systemd timer running every 5 minutes (OnCalendar=*:0/5 — more restart-resilient than plain cron) pulls the latest Vector window, computes features using the same extraction module as training, and posts to /score.
Keep feature extraction logic in one shared module between the training script and the scoring API. If they drift, you’ll hit something like ValueError: X has 42 features, but IsolationForest is expecting 47 features as input — or worse, no error at all, just garbage scores because the feature order silently shifted. This is the single most common failure mode we’ve seen with this setup, and it’s completely avoidable with shared code.
On cost: IsolationForest scoring across ~50 features runs in sub-millisecond time and comfortably fits on a $5/mo VPS. No GPU, no managed inference endpoint. Compared to running every log window through an LLM continuously, this is the cheap half of the pipeline — which matters because the expensive half is next.
Here’s the full Compose stack tying it together, including a Redis cache used by the optional LLM triage layer:
# docker-compose.yml — minimal stack: log shipper, scoring API, and cache for LLM triage
version: "3.9"
services:
vector:
image: timberio/vector:0.37.1-alpine
volumes:
- ./vector.toml:/etc/vector/vector.toml:ro
- app_logs:/var/log/app:ro
- windows_data:/data/vector
command: ["--config", "/etc/vector/vector.toml"]
scoring-api:
build: ./scoring-api
environment:
- MODEL_PATH=/models/iforest_2024-05-14.joblib # pin explicit version, not "latest"
- REDIS_URL=redis://cache:6379/0
volumes:
- ./models:/models:ro
ports:
- "8080:8080"
depends_on:
cache:
condition: service_healthy
cache:
image: redis:7.2-alpine
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
retries: 5
volumes:
app_logs:
windows_data:Notice the model path is pinned to a specific date-versioned filename, not latest. Name your joblib files with the training window range baked in (e.g. iforest_2024-05-01_2024-05-14.joblib) so you never accidentally score against a stale model after a redeploy.
Step 4 — Add semantic triage before paging humans
This step is optional but it’s the piece that actually moved our false-positive rate down further, past what statistics alone could do. When /score returns is_anomaly: true, we send the top 10 raw log lines from that window to gpt-4o-mini with a simple prompt: is this a genuine infra anomaly, or benign noise like a deploy, cron job, or known batch process? The model’s verdict gets cached per service+message-template hash for 24 hours, so recurring benign patterns — nightly backups being the classic example — only get classified once, not every cycle.
Two things matter here. First, cost: statistical scoring is nearly free, but routing every flagged window through an LLM at $0.15/1M input tokens is the real line item in your budget. Cache aggressively or you’ll be surprised by the bill. Second, and this one is non-negotiable: strip tokens, IPs, and emails from log lines via regex before sending anything to an external LLM API. Never send raw, unfiltered logs to a third-party endpoint, and log the redaction pass itself so you have an audit trail if anyone asks what left your network.
Verify and test
Prove it works before trusting it. We injected a synthetic anomaly by spiking a test service’s log rate 20x:
for i in {1..500}; do curl -s localhost:8080/error > /dev/null; doneWithin one 5-minute cycle, /score should return something like this:
{"anomaly_score": -0.184, "is_anomaly": true, "service": "checkout-api", "window": "2024-05-14T09:35:00Z"}Beyond the synthetic test, track the false-positive rate over 48 hours against your old regex-based alerts. Our target was under 5 alerts/day, down from the original 200+, and we hit it within the first week of shadow mode. Also confirm the LLM triage layer is actually earning its cost — check cache hit logs and verify that recurring benign patterns like backups stop paging after their first classification. If they keep paging, your cache key (service + template hash) probably isn’t matching consistently, which usually traces back to drain3 template IDs shifting after a state file loss.
Closing
This pipeline isn’t a replacement for proper observability — it’s a noise filter sitting on top of what you already have, and it only works because the underlying log anomaly detection model is simple enough to trust and cheap enough to run everywhere. Roll it out incrementally: run in shadow mode, log-only, no paging, for at least a week to calibrate the contamination parameter against your real traffic before wiring it into Alertmanager with a dedicated receiver and a group_wait: 30s to batch near-simultaneous anomalies. Revisit and retrain monthly, because traffic patterns shift and a stale model is just a slower version of the regex problem you started with. For more on tightening alert pipelines and cutting through observability noise, check out our other posts on kuryzhev.cloud, and for the full API and options on the model side, the scikit-learn IsolationForest docs and the Vector VRL reference are worth bookmarking.
