Your on-call engineer spends the first 8 minutes of every incident manually grepping through 50,000 log lines — LLM log triage with Loki and CloudWatch Insights can cut that to under 30 seconds. I know because we timed it. The grep-and-guess workflow isn’t a skill gap; it’s a tooling gap. These seven tips close it without building a platform team project.
No intros needed. Here’s what actually works.
Tip 1: Write LogQL and Insights Queries That Feed Directly Into an LLM Prompt
Pull structured log chunks programmatically — never copy-paste from a browser tab at 2 AM.
The Loki HTTP API endpoint /loki/api/v1/query_range accepts a LogQL expression, a nanosecond-precision time range, and a limit parameter. You get back a JSON payload with stream labels and timestamped values. On the CloudWatch side, aws logs start-query kicks off an async job you poll with get-query-results — that poll can take anywhere from 2 to 30 seconds depending on log volume, so plan for it in your timeout budget.
The critical shape decision: do not send the raw API response to the LLM. The Loki response wraps every line in metadata — stream labels, nanosecond timestamps, result type — that bloats your token count by 3–5× versus extracting just the log line strings. Extract entry[1] from each value tuple and nothing else. For CloudWatch, stats count(*) by bin(1m) compresses a 10,000-line result down to roughly 60 data points for a one-hour window. Send the aggregate, not the raw lines, when you want trend context rather than stack traces.
Target 50–100 lines per request to stay predictably under the 8k-token range on mid-tier models. Loki 2.9+ supports logfmt and json parsers natively in LogQL, so you can filter on structured fields before the API even returns results — use it.
This script shows the full Loki fetch, PII scrubbing, deduplication, and LLM call wired together. Read it top to bottom once before using any individual piece:
#!/usr/bin/env python3
"""
loki_llm_triage.py — Pull errors from Loki, scrub PII, send to OpenAI for triage.
Requires: openai>=1.30.0, requests>=2.31.0, tenacity>=8.2.0
"""
import re
import json
import hashlib
import time
from collections import Counter
from datetime import datetime, timedelta, timezone
import requests
from openai import OpenAI
from tenacity import retry, wait_exponential, stop_after_attempt
# --- Config ---
LOKI_URL = "http://loki.internal:3100"
OPENAI_MODEL = "gpt-4o-2024-05-13" # pinned snapshot, never use floating alias
MAX_LOG_LINES = 75 # stay under 8k tokens for cost control
ERROR_COOLDOWN_SECS = 600 # 10-minute dedupe window per service+error_type
# In-memory dedupe store (replace with Redis in production)
_triage_cache: dict[str, float] = {}
# --- PII scrubber ---
PII_PATTERNS = [
(re.compile(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+"), "[EMAIL]"),
(re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"), "[IP]"),
(re.compile(r"eyJ[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+"), "[JWT]"),
(re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b"), "[UUID]"),
]
def scrub_pii(line: str) -> str:
for pattern, replacement in PII_PATTERNS:
line = pattern.sub(replacement, line)
return line
# --- Loki query ---
def fetch_loki_errors(app: str, minutes: int = 5) -> list[str]:
now = datetime.now(timezone.utc)
start = int((now - timedelta(minutes=minutes)).timestamp() * 1e9)
end = int(now.timestamp() * 1e9)
# logfmt parser + level filter drops INFO/DEBUG server-side before transfer
query = f'{{app="{app}", env="prod"}} |= "ERROR" | logfmt | level="error"'
resp = requests.get(
f"{LOKI_URL}/loki/api/v1/query_range",
params={"query": query, "start": start, "end": end, "limit": MAX_LOG_LINES},
timeout=10,
)
resp.raise_for_status()
results = resp.json().get("data", {}).get("result", [])
# Extract only the log line string (index 1); index 0 is nanosecond timestamp
lines = [entry[1] for stream in results for entry in stream.get("values", [])]
return lines
# --- Dedupe identical lines ---
def dedupe_lines(lines: list[str]) -> list[str]:
counts = Counter(lines)
deduped = []
for line, count in counts.most_common(MAX_LOG_LINES):
prefix = f"[x{count}] " if count > 1 else ""
deduped.append(prefix + line)
return deduped
# --- LLM triage call with retry on RateLimitError ---
@retry(wait=wait_exponential(min=1, max=10), stop=stop_after_attempt(3))
def call_llm_triage(log_chunk: str, app: str) -> dict:
client = OpenAI() # reads OPENAI_API_KEY from env
system_prompt = open("prompts/log-triage-v2.txt").read()
t0 = time.monotonic()
response = client.chat.completions.create(
model=OPENAI_MODEL,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"Service: {app}\n\nLogs:\n{log_chunk}"},
],
max_tokens=512,
temperature=0.2, # low temp = deterministic operational output
)
latency_ms = int((time.monotonic() - t0) * 1000)
return {
"summary": response.choices[0].message.content,
"model": response.model, # log the actual model name, not your config value
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"latency_ms": latency_ms,
}
# --- Main triage entry point ---
def triage_service(app: str):
lines = fetch_loki_errors(app)
if not lines:
return
# Dedupe key based on first 5 lines — fingerprints the error cluster
error_hash = hashlib.md5("\n".join(lines[:5]).encode()).hexdigest()[:8]
cache_key = f"{app}:{error_hash}"
if cache_key in _triage_cache and time.time() - _triage_cache[cache_key] < ERROR_COOLDOWN_SECS:
print(f"[{app}] Triage suppressed (cooldown active for {cache_key})")
return
_triage_cache[cache_key] = time.time()
clean_lines = [scrub_pii(l) for l in dedupe_lines(lines)]
log_chunk = "\n".join(clean_lines)
result = call_llm_triage(log_chunk, app)
print(json.dumps({"app": app, **result}, indent=2))
if __name__ == "__main__":
triage_service("payment-api")
Tip 2: Use a System Prompt Tuned for Log Noise, Not General Q&A
A generic “explain this” prompt returns generic answers. Production log triage needs a purpose-built system prompt.
The difference is dramatic. A vanilla prompt gets you a paragraph about what an OutOfMemoryError is. A tuned prompt gets you “OOM in heap space occurred 47 times in 5 minutes across 3 pods, consistent with a memory leak on the /checkout endpoint introduced after the 14:30 deploy — suggested action: rollback to previous image tag.” That’s the delta between a system prompt that says “you are a helpful assistant” and one that says “you are a senior SRE performing rapid log triage during an active incident.”
Pin the model role explicitly. Include instructions to ignore INFO-level noise, group repeated stack traces, and cap output at 200 words. Store the prompt in a versioned file so the team can iterate on it without touching Python code. Reference it by git SHA in your deployment config so you always know which prompt version produced which triage output. Here’s the prompt file we use:
# prompts/log-triage-v2.txt
# Version: 2.1 | Last updated: 2024-06-01 | Commit: a3f9c12
You are a senior SRE performing rapid log triage during an active incident.
Your job:
1. Identify distinct error patterns (group similar stack traces together).
2. Estimate the most probable root cause in 1-2 sentences.
3. Suggest one immediate mitigation action (restart, rollback, config change, etc.).
4. Flag any errors that look like security events (auth failures, injection attempts).
Rules:
- Ignore INFO and DEBUG lines entirely.
- Do NOT repeat log lines back verbatim.
- Output must be under 200 words.
- Use plain language — no markdown headers, no bullet symbols.
- If you cannot determine root cause, say "insufficient signal" and list
what additional logs would help.
Output format (plain text, no JSON):
PATTERN SUMMARY: <1-2 sentences>
PROBABLE CAUSE: <1-2 sentences>
SUGGESTED ACTION: <one concrete step>
SECURITY FLAG: <yes/no + reason, or "none">
Tip 3: Filter Before You Send — Cut Token Spend With Pre-LLM Query Logic
Sending raw, unfiltered log streams to an LLM is the fastest way to burn your API budget and get worse answers simultaneously.
In Loki, push the filter work into LogQL before the API call returns anything. The expression {app="api", env="prod"} |= "ERROR" | logfmt | level="error" drops INFO and DEBUG server-side. You never transfer those bytes, never pay tokens for them, and the model never has to ignore them. In CloudWatch Insights, filter @message like /ERROR|FATAL|Exception/ combined with stats count(*) by bin(1m) compresses a 10,000-line result to roughly 60 aggregated data points — that’s a 99.4% reduction in token cost when you need trend context rather than raw stack traces.
Client-side deduplication is equally important. One OOM error repeated 200 times shouldn’t consume 30 tokens × 200 occurrences. Use Python’s collections.Counter on raw lines, take the top N unique messages, and annotate repeat counts inline (e.g., [x47] java.lang.OutOfMemoryError: Java heap space). The model gets the frequency signal without the repetition cost. We routinely see 4–6× token reduction from this step alone on high-traffic services.
Tip 4: Stream LLM Responses Into a Slack or PagerDuty Webhook for Alert Context
The triage output is useless if the on-call engineer has to open a separate tool to read it. Close the loop in the alert itself.
Use the OpenAI streaming API (stream=True) or Anthropic’s SSE endpoint to start posting partial analysis to Slack before the full response completes. This matters at 3 AM — perceived latency drops significantly when the first sentence appears within 2–3 seconds instead of waiting 12 seconds for the complete response. Attach the raw query link alongside the summary: a Loki Grafana Explore URL or a CloudWatch Insights deep link gives engineers a one-click path to verify the model’s interpretation.
Watch out for this: always set a hard timeout of 15–20 seconds on the LLM call. If it exceeds that, fall back to posting the raw top-10 error lines with a [LLM triage unavailable] label. I’ve seen LLM API latency spike to 45+ seconds during provider incidents — you do not want that blocking your alerting critical path. Run the LLM call in an async worker (Python asyncio or a queue consumer) completely separate from the alert delivery pipeline. The alert fires immediately; the LLM enrichment follows as a reply thread.
Tip 5: Version-Pin Your Model and Log the Model Name With Every Triage Output
Silent model upgrades are a real production risk that most teams don’t think about until it bites them.
Always specify a dated model snapshot: model="gpt-4o-2024-05-13" instead of the floating alias gpt-4o. OpenAI deprecated gpt-3.5-turbo-0301 with three months’ notice, and teams using floating aliases experienced silent model swaps mid-incident — the triage output format changed, downstream Slack parsing broke, and no one immediately understood why the bot started behaving differently. That’s an entirely avoidable incident-within-an-incident.
Log everything: {model, prompt_tokens, completion_tokens, latency_ms, log_window_start, log_window_end} to your own observability stack. This gives you cost attribution per service, quality regression detection (sudden change in completion_tokens for the same service often means the prompt changed or log volume spiked), and an audit trail for post-incident reviews. Anthropic Claude 3 Haiku is roughly 10× cheaper than GPT-4o for high-frequency triage calls — about $0.00025 per 1k input tokens versus $0.005 — worth evaluating if you’re running this on more than a handful of services.
Tip 6: Gate LLM Calls Behind an Error-Rate Threshold to Avoid Alert Storms
A cascading failure generating 500 errors per minute will trigger your triage bot hundreds of times without a gate. That’s $5–$25 in unplanned API spend for a single 30-minute incident.
Only trigger LLM triage when the error rate crosses a meaningful threshold — more than 10 errors per minute sustained for at least 2 minutes is a reasonable starting point. Implement this as a simple cooldown timer or token bucket in the calling script. Add a per-service dedupe key built from a hash of service + error_type with a 10-minute TTL. In Redis that’s a single command: SETEX triage:{service}:{error_hash} 600 1. If the key exists, skip the LLM call and log the suppression. In the script above I used an in-memory dict for simplicity — replace it with Redis for any multi-instance deployment.
Watch out for this too: if you’re using tenacity for retry logic and you hit RateLimitError: Rate limit reached for gpt-4o, exponential backoff with wait=wait_exponential(min=1, max=10) is the correct pattern. But combine it with the cooldown gate — retrying 3 times across a 10-second window is fine; retrying 3 times for each of 200 concurrent alert triggers is not. The gate prevents the retry logic from multiplying your API calls during the worst possible moment.
Tip 7: Sanitize Logs Before Sending — PII and Secrets Leak Into LLM Providers
This is the tip most teams skip until their security team asks where customer emails are going. Don’t skip it.
Run a regex scrubber over every log line before building the prompt. The four patterns that catch the majority of sensitive data in application logs: email addresses, IPv4/IPv6 addresses, JWTs (the eyJ[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+ pattern matches the header.payload.signature structure), and UUIDs used as auth tokens or session identifiers. The scrub_pii() function in the script above handles all four and adds zero meaningful latency — regex substitution on 75 lines takes microseconds.
Understand the data retention reality: most LLM provider terms of service allow training on API inputs unless you explicitly opt out. OpenAI’s Zero Data Retention option requires a paid enterprise agreement — it is not available on standard API keys regardless of what parameters you pass. For environments under HIPAA, PCI-DSS, or SOC 2 controls, self-hosted models are the right answer. Ollama running llama3:8b or mistral:7b operates entirely within your VPC boundary on an AWS g4dn.xlarge instance at roughly $0.53/hr. The quality trade-off versus GPT-4o is real and worth accepting explicitly — document the decision in your runbook so the next engineer understands why you’re on a smaller model. For more on building secure, cost-aware automation pipelines, see the kuryzhev.cloud DevOps patterns archive.
LLM log triage with Loki and CloudWatch Insights is not a research project. The seven patterns above — structured extraction, tuned prompts, pre-filtering, streaming delivery, model pinning, rate gating, and PII scrubbing — are the complete production checklist. Wire them together once, and your on-call rotation gets 8 minutes back on every incident.
Related
