Stackbook Logo
← Back to Tools

Grafana Dashboards

How to build useful Grafana dashboards: connect data sources, write panels, use variables, provision as code, and avoid cluttered, misleading views.

4 min read

Grafana is an open-source visualisation and alerting platform. It does not store metrics itself. Instead it queries data sources such as Prometheus, Loki, Tempo, Elasticsearch, or SQL databases and renders the results as panels on dashboards. Use Grafana when you want a shared, living view of system health that engineers, on-call responders, and product teams can all read.

Quick start

The fastest way to try Grafana is with Docker, alongside Prometheus.

docker run -d --name grafana -p 3000:3000 grafana/grafana-oss:latest

Open http://localhost:3000 and sign in with the default admin / admin, then change the password when prompted. Next, add a data source under Connections, Data sources, choose Prometheus, and set the URL, for example http://prometheus:9090 if both containers share a Docker network. Click Save and test.

Core concepts

  • Data source: a connection to a backend. Each panel query runs against one.
  • Dashboard: a collection of panels arranged in a grid, stored as JSON.
  • Panel: a single visualisation, such as a time series, stat, gauge, table, or heatmap.
  • Variable: a dropdown at the top of the dashboard that parameterises queries.
  • Folder and permissions: organise dashboards by team and control who can edit.
  • Alert rule: a query plus a condition that notifies a contact point.

Building a panel

Create a dashboard, add a visualisation, pick the data source, and enter a query. For a request-rate panel against Prometheus:

sum by (service) (rate(http_requests_total[5m]))

For an error ratio shown as a Stat panel with a percent unit:

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Set thresholds so the panel turns amber and red at meaningful values, set the unit explicitly, and give the panel a title that states what it measures, such as "5xx error ratio (5m)".

Variables and reuse

Variables let one dashboard serve many services. Define a query variable named service with the query below, then reference it as $service in panels.

label_values(http_requests_total, service)

sum(rate(http_requests_total{service="$service"}[5m]))

Enable multi-value and the All option for fleet-wide views, and use regex matching with =~ in that case.

Dashboards as code

Clicking together dashboards is fine for exploration, but production dashboards should live in version control. Grafana can load data sources and dashboards from files at startup.

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

# /etc/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: default
    folder: Platform
    type: file
    options:
      path: /var/lib/grafana/dashboards

Export a dashboard's JSON from the share menu, commit it to the folder above, and changes are reviewed like any other code. Tools such as Grafonnet or Terraform can generate dashboards when you manage many of them.

Common pitfalls

  • Wall-of-graphs dashboards. Fifty panels with no hierarchy hide the signal. Put the most important indicators at the top and detail in collapsed rows.
  • Averages that lie. Use percentiles from histograms, for example histogram_quantile(0.99, ...), for latency.
  • Missing units and axes. A graph with no unit invites misreading.
  • Heavy queries. Very wide ranges and high-cardinality labels slow dashboards and strain the data source. Use recording rules for expensive expressions.
  • Manual edits in production that are overwritten by provisioning, or the reverse. Pick one source of truth.
  • No link to action. Add data links to logs, traces, and runbooks.

Where Grafana fits in an architecture

Grafana sits at the top of an observability stack. Prometheus collects metrics, Loki aggregates logs, and Tempo or Jaeger stores traces. Grafana unifies them so an engineer can move from a latency spike to the relevant logs and then to a trace. Organise dashboards in layers: an overview per service using the RED method (rate, errors, duration), infrastructure dashboards using the USE method (utilisation, saturation, errors), and focused debugging dashboards.

Key takeaways

  • Grafana visualises data held elsewhere, so quality depends on your data sources.
  • Use variables to avoid copy-pasting dashboards per service.
  • Provision data sources and dashboards from files to keep them reviewable.
  • Design for the on-call reader: few panels, clear titles, units, and thresholds.
  • Pair dashboards with alert rules so problems find people, not the reverse.