Overview
Loki retention and index tuning for high-volume log pipelines is one of those operational concerns that teams defer until storage bills spike or query latency becomes unacceptable. At low ingestion rates, Loki’s defaults are forgiving. Once you cross into multi-tenant environments processing dozens of gigabytes per day, the gap between “it works” and “it works correctly under pressure” becomes measurable in both cost and reliability.
Two failure modes appear repeatedly in production. The first is retention that appears configured but never actually deletes data — almost always because compactor.retention_enabled was left at its default of false. The second is index bloat caused by misconfigured schema periods or undersized caches, which degrades query performance progressively rather than catastrophically, making it harder to diagnose.
This guide addresses both. We will configure global and per-stream retention using limits_config and the compactor block, tune TSDB index parameters for throughput, and then validate that the compactor is actually enforcing the policies we set. The configuration targets Loki 2.8 and later, where TSDB has replaced BoltDB-shipper as the recommended index store.
If you are running a broader observability stack and want context on how Loki fits alongside alerting pipelines, the kuryzhev.cloud DevOps_DayS archive covers related topics including Grafana alerting rules and Promtail log shipping patterns.
Before You Start
Before making any changes to a running Loki instance, confirm the following prerequisites are in place.
- Loki version: 2.8 or later. TSDB index support and runtime config reload both require this minimum. Earlier versions should be upgraded before applying this configuration.
- Storage backend: The examples in this guide use the filesystem backend, which is appropriate for single-node deployments and staging environments. For production clusters on AWS or GCS, replace
filesystemreferences with your object storage configuration. Note that object storage bucket lifecycle policies must be set to a retention period equal to or longer than Loki’s own retention period — if the bucket deletes objects before the compactor processes them, you will see index inconsistencies. - Deployment method: Whether you are running Loki via Helm chart or as a standalone binary, the configuration blocks described here map directly to the same YAML structure. Helm users will need to nest these values under the appropriate
loki:key in theirvalues.yaml. - Access level: Write access to the Loki configuration file and the ability to send HTTP requests to the Loki API endpoint (port 3100 by default) for config reload and compactor inspection.
- Writable local path: The compactor requires a local working directory —
/data/loki/compactorin our examples. Ensure this path exists and is writable by the Loki process before enabling the compactor. - logcli: Installed and configured with your Loki address for post-change validation queries. See the official logcli documentation for installation instructions.
Step 1: Configure Retention Policies
Retention in Loki is not a simple time-to-live setting on stored objects. It is a two-part system: the limits_config block declares how long log streams should be kept, and the compactor block is responsible for actually enforcing those declarations by deleting expired chunks. Neither half works without the other.
The most important thing to understand before writing a single line of config is this: setting retention_period under limits_config without also setting compactor.retention_enabled: true results in a configuration that declares retention but never acts on it. Data accumulates indefinitely. This is the single most common production misconfiguration we see.
Below is the full configuration file we will be working with throughout this guide. It covers retention, index tuning, and ingester parameters in a single coherent block.
# loki-config.yaml — retention and index tuning for high-volume deployments
auth_enabled: false
server:
http_listen_port: 3100
enable_runtime_reload: true
common:
path_prefix: /data/loki
storage:
filesystem:
chunks_directory: /data/loki/chunks
rules_directory: /data/loki/rules
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
ingester:
chunk_idle_period: 30m
max_chunk_age: 2h
chunk_retain_period: 1m
limits_config:
retention_period: 168h # 7 days global default
ingestion_rate_mb: 64
ingestion_burst_size_mb: 128
max_streams_per_user: 100000
split_queries_by_interval: 24h
overrides:
team-infra:
retention_period: 720h # 30 days for infra team
team-debug:
retention_period: 48h # 2 days for short-lived debug logs
compactor:
working_directory: /data/loki/compactor
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
compaction_interval: 10m
delete_request_store: filesystem
chunk_store_config:
chunk_cache_config:
embedded_cache:
enabled: true
max_size_mb: 512
ttl: 1h
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 256
storage_config:
tsdb_shipper:
active_index_directory: /data/loki/tsdb-index
cache_location: /data/loki/tsdb-cache
cache_ttl: 24hWalk through the retention-specific sections deliberately.
Under limits_config, retention_period: 168h sets the global default — all tenants and all streams will have logs older than seven days eligible for deletion. The overrides block under limits_config provides per-tenant granularity without requiring separate Loki instances. team-infra gets 30 days because infrastructure logs are needed for post-incident review cycles. team-debug gets 48 hours because debug-level logs are high-volume and short-lived by nature. This per-tenant override pattern scales cleanly as the number of teams grows.
Under compactor, retention_enabled: true is the switch that activates enforcement. The retention_delete_delay: 2h setting introduces a grace period between when a chunk is marked for deletion and when it is actually removed — useful for catching accidental misconfiguration before data is permanently gone. retention_delete_worker_count: 150 controls parallelism; increase this if compaction is falling behind on high-ingestion deployments. compaction_interval: 10m determines how frequently the compactor wakes up to process pending work.
The ingester settings also matter for retention correctness. chunk_idle_period: 30m flushes a chunk to storage if no new logs arrive within 30 minutes. max_chunk_age: 2h forces a flush regardless of activity. Chunks that have not been flushed to the store cannot be processed by the compactor, so these values directly affect how quickly retention takes effect after logs are written.
Step 2: Tune Index Settings for Throughput
Index configuration has a disproportionate impact on both write throughput and query performance. Getting it wrong does not cause immediate failures — it causes gradual degradation that is difficult to attribute without understanding what is happening underneath.
The schema configuration block is where index behavior is defined. The key constraint is that index_period must always be 24h. This is not a recommendation — it is a requirement enforced by Loki. Attempting to set a different value will result in a startup error. If you need to change the period (which you should not need to), you must do so by adding a new schema config entry with a from date set in the future, never by modifying an existing entry. Modifying an existing schema entry after data has been written against it will corrupt your index.
The TSDB store, configured via store: tsdb in the schema block, is the correct choice for Loki 2.8 and later. It offers significantly better query performance at scale compared to BoltDB-shipper, particularly for high-cardinality label sets. The storage_config.tsdb_shipper block controls the active index directory and local cache behavior. Setting cache_ttl: 24h keeps recently accessed index pages in memory, which reduces object storage reads during query execution.
Index cache size is controlled separately from the TSDB shipper. The chunk_store_config.chunk_cache_config.embedded_cache.max_size_mb setting determines how much memory Loki allocates for caching chunk data. At 512 MB, this is a reasonable starting point for a mid-size deployment. For nodes with more available memory and high query rates, increasing this to 1024 MB or higher will measurably reduce tail latency on range queries.
The query_range.results_cache block caches the results of split sub-queries. Combined with split_queries_by_interval: 24h under limits_config, Loki splits long-range queries into 24-hour chunks and caches each chunk’s results independently. This means a repeated query over a 7-day window only fetches uncached portions from storage, which is a significant throughput improvement for dashboards that refresh frequently.
One tuning decision that affects both ingestion throughput and index size is the max_streams_per_user limit. The value of 100,000 in this configuration is appropriate for environments with well-controlled label cardinality. If your teams are generating high-cardinality labels — for example, including request IDs or user IDs as Loki labels rather than log line fields — you will see stream counts approach this limit quickly. The correct fix is to move high-cardinality values into the log line and use LogQL’s | json or | logfmt parsers to filter them at query time, not to raise the stream limit indefinitely.
Step 3: Validate and Apply Changes
Applying configuration changes to a running Loki instance without a restart is possible when enable_runtime_reload: true is set in the server block. Once your updated configuration file is in place, trigger a reload with a single POST request.
curl -X POST http://localhost:3100/-/reloadA successful reload returns HTTP 200 with no body. If the configuration contains syntax errors or invalid values, Loki will log the error and continue running with the previous configuration — it will not crash. Check the Loki logs immediately after the reload to confirm the new configuration was accepted.
To verify that the compactor is active and processing retention, tail the Loki logs and filter for compactor activity. A healthy compactor will emit lines similar to the following at each compaction interval.
level=info ts=2024-01-15T10:00:00Z caller=compactor.go msg="retention apply" table=index_19371
level=info ts=2024-01-15T10:00:00Z caller=compactor.go msg="compaction completed" duration=4.2sIf you see msg="retention apply" entries, the compactor is running and evaluating retention rules. The absence of these log lines after the compaction interval has elapsed is the primary indicator that something is wrong — covered in the Troubleshooting section below.
To confirm that retention is actually removing data and not just marking it, use logcli to query a stream that should have fallen outside the retention window. For a 7-day global retention period, any logs older than 168 hours should return no results.
# Query logs older than retention period — should return empty result
logcli query '{job="varlogs"}' \
--from="2024-01-01T00:00:00Z" \
--to="2024-01-02T00:00:00Z" \
--addr="http://localhost:3100"Adjust the --from and --to values to a range that predates your retention window. An empty result confirms that the compactor has successfully deleted the data. A non-empty result means retention is configured but not yet enforced — either the compactor has not run since the data aged out, or there is a configuration problem.
Also verify the compactor’s working directory is being used. After the first compaction cycle, /data/loki/compactor should contain state files. An empty directory after several compaction intervals indicates the compactor is not initializing correctly.
ls -lh /data/loki/compactor/Troubleshooting
The following issues appear with enough regularity in production environments that they are worth addressing directly.
Retention is configured but data is not being deleted. The most likely cause is compactor.retention_enabled being absent or set to false. Verify the running configuration by querying the Loki config endpoint and checking the compactor block.
curl -s http://localhost:3100/config | grep -A 10 "compactor:"If the response shows retention_enabled: false, the runtime reload either failed or the configuration file change was not saved correctly. A secondary cause is a mismatch between the object storage bucket lifecycle policy and Loki’s retention period. If the bucket is configured to delete objects after 30 days but Loki retention is set to 7 days, the compactor will attempt to delete chunks that the storage backend has not yet expired. On some backends this causes silent failures. Ensure the bucket lifecycle policy is set to a value greater than or equal to the longest retention period configured in Loki.
Index bloat despite compaction running. If the TSDB index directories are growing despite regular compaction, check whether multiple schema config entries are present with overlapping date ranges. Each active schema entry maintains its own index files. Entries with stale from dates that predate your data range can accumulate index files that are never compacted away. Review your schema_config.configs list and remove any entries whose date ranges contain no data.
Also confirm that tsdb_shipper.active_index_directory and cache_location are on separate paths. Placing them on the same directory can cause the shipper to re-read its own cache output as new index data, which inflates apparent index size without adding real content.
Compactor stuck in a loop. This typically manifests as the compactor log repeatedly emitting the same table name without progressing to subsequent tables. The most common cause is a permissions issue on the working_directory. The Loki process must have read and write permissions on /data/loki/compactor and all subdirectories. Verify ownership and permissions, then restart the compactor by restarting Loki — there is no hot-fix for a stuck compactor state without a process restart.
ls -la /data/loki/compactor/
# Expected: owned by the loki process user with rwx permissionsA secondary cause of compactor looping is clock skew between the Loki node and the storage backend. If the node’s system time is significantly ahead of or behind the storage timestamps, the compactor may repeatedly evaluate the same chunks as candidates for deletion without acting on them. Confirm NTP synchronization on the Loki host.
High ingestion rates causing chunk cache eviction. If you observe high cache miss rates in Loki metrics (loki_cache_fetched_keys_total vs loki_cache_hits_total), the embedded cache is undersized for your query pattern. Increase max_size_mb under chunk_cache_config.embedded_cache and monitor the ratio after the next compaction cycle. For deployments where memory is constrained, consider moving to an external Memcached or Redis cache instead of the embedded option.
Going Further
The configuration established in this guide handles the core retention and index concerns, but several adjacent areas are worth addressing as your deployment matures.
Alerting on ingestion rate. The ingestion_rate_mb and ingestion_burst_size_mb limits in limits_config will throttle tenants that exceed their allocation, but you will not know a tenant is being throttled unless you are monitoring for it. Add a Grafana alerting rule on loki_ingester_streams_created_total and loki_discarded_samples_total to catch throttling before it becomes a support issue. The kuryzhev.cloud post on Grafana alerting rules for container metrics covers the alerting rule structure if you need a starting point.
Per-tenant limits beyond retention. The overrides block under limits_config supports more than just retention period. You can set per-tenant ingestion rate limits, max label name length, and query timeout overrides. For multi-tenant environments where different teams have significantly different log volumes and query patterns, per-tenant limits are the primary tool for preventing noisy-neighbor problems. The Loki limits_config reference documents every available override key.
Object storage lifecycle policies. For deployments using S3, GCS, or Azure Blob as the storage backend, the bucket lifecycle policy is a second layer of retention enforcement. Set the lifecycle expiration to a value 20–30% longer than your longest Loki retention period. This gives the compactor time to process deletions gracefully before the storage backend removes objects independently, which prevents the index from referencing chunks that no longer exist.
Compactor metrics and dashboards. Loki exposes compactor-specific metrics including loki_compactor_runs_total, loki_compactor_blocks_cleaned_total, and loki_compactor_meta_sync_duration_seconds. Building a simple Grafana dashboard around these metrics gives you visibility into compaction health without having to parse log lines manually. At sustained high ingestion rates, a compactor that is falling behind will show a growing gap between loki_compactor_runs_total and the expected run count based on your compaction_interval.
