Stackbook Logo
← Back to Tools

Loki and Log Aggregation Guide

Centralize logs with Grafana Loki and Promtail or Alloy: label design, LogQL queries, retention, and how to avoid cardinality problems.

4 min read

What it is and when to use it

Grafana Loki is a horizontally scalable log aggregation system inspired by Prometheus. Instead of indexing the full text of every line, Loki indexes only a small set of labels per log stream and stores compressed log chunks in cheap object storage. The result is a much lower cost and operational footprint than full-text engines.

Use Loki when you already run Grafana and Prometheus, want to correlate logs with metrics, and mostly search by service, environment and time range followed by text filtering. If you need heavy full-text analytics or complex aggregations across arbitrary fields, a search engine such as Elasticsearch may suit you better.

Quick start

A single-node setup with Docker Compose, Loki and Grafana:

services:
  loki:
    image: grafana/loki:3.1.0
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
  grafana:
    image: grafana/grafana:11.1.0
    ports:
      - "3000:3000"

Then add Loki as a data source in Grafana with the URL http://loki:3100. Check readiness with curl localhost:3100/ready.

Shipping logs

An agent tails files or container output and pushes to Loki. Promtail is the classic agent, and Grafana Alloy is its successor. A Promtail configuration:

server:
  http_listen_port: 9080
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: app
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          env: prod
          __path__: /var/log/app/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
      - labels:
          level:

On Kubernetes, deploy the agent as a DaemonSet and use service discovery to attach namespace, pod and app labels automatically. OpenTelemetry Collectors can also push logs to Loki through its OTLP endpoint.

Core concepts

Streams and labels. A stream is a unique combination of label values. Every distinct combination creates a new stream, so labels must be few and bounded.

LogQL has two parts: a stream selector in braces, then optional filters and parsers.

{app="checkout", env="prod"} |= "timeout" != "healthcheck"

{app="checkout"} | json | level="error" | line_format "{{.msg}}"

sum by (app) (rate({env="prod"} |= "error" [5m]))

topk(5, sum by (path) (count_over_time({app="api"} | logfmt | status >= 500 [1h])))

Metric queries turn logs into time series, which you can chart or use in alert rules evaluated by the Loki ruler.

Storage and retention. Production deployments use object storage such as S3 or GCS with the TSDB index. Retention is enforced by the compactor:

limits_config:
  retention_period: 744h
compactor:
  retention_enabled: true
  delete_request_store: filesystem

Pitfalls

  • High-cardinality labels. Putting user ids, request ids, IP addresses or trace ids in labels creates millions of streams and wrecks performance. Keep those in the log line and filter with LogQL.
  • Unstructured logs. Emit JSON or logfmt so you can parse fields at query time.
  • Broad queries. Always narrow by labels and time range first. Line filters like |= are fast, regular expressions are slower.
  • Ingestion limits. Large bursts hit per-tenant rate limits. Tune ingestion_rate_mb and ingestion_burst_size_mb, and watch for HTTP 429 responses.
  • Out-of-order entries. Multiple writers on one stream can be rejected. Add a distinguishing label such as pod name.
  • No multi-tenancy awareness. With auth enabled you must send the X-Scope-OrgID header.
  • Skipping retention lets storage costs creep up.

How it fits an architecture

Logs are one pillar next to metrics and traces. A typical stack uses Alloy or Promtail on every node, Loki in simple scalable or microservices mode on object storage, Prometheus for metrics and Tempo or Jaeger for traces, all inside Grafana. Shared labels like app and env let you jump from a metric panel to the matching logs in one click, and derived fields turn a trace id in a log line into a link to the trace.

Apply log hygiene upstream too: structured output, consistent levels, correlation ids, and no secrets in messages.

Key takeaways

  • Loki indexes labels, not content, which keeps it cheap and simple.
  • Keep labels low-cardinality and bounded; put high-cardinality data in the line.
  • Learn LogQL stream selectors, line filters, parsers and metric queries.
  • Use object storage and compactor-based retention in production.
  • Emit structured logs and correlate with traces through ids.