Part 24: Building a Grafana Dashboard That Actually Works

We deployed the monitoring stack in Part 21 — VictoriaMetrics, Grafana, Loki, Alertmanager, node_exporter, cAdvisor — and it all came up healthy. But “healthy” and “actually showing data” are different things. This post is about closing that gap: diagnosing why the Grafana dashboard had no data, fixing the root causes, and building something that reflects our real environment.

The Problem: No Data, Broken Panels

After deploying the Homelab Overview dashboard via Ansible provisioning, we had three visible issues:

1. Container CPU and Memory panels showed “No data”
2. The Host Status panel showed every host as DOWN (including unmanaged appliances)
3. The dashboard was provisioned but datasource references were wrong

Digging into each one surfaced a layered set of issues.

Datasource UID Mismatch

The first thing to check was whether VictoriaMetrics was receiving any data at all:

ssh docker-host 'curl -s "http://localhost:8428/api/v1/query?query=up{job=\"node_exporter\"}"'

It was — all managed hosts were reporting up=1. So the data was there; the dashboard just couldn’t find it.

The issue was in the provisioned dashboard JSON. When we generated the JSON, we used human-readable datasource UIDs like "VictoriaMetrics" and "Loki". But Grafana auto-generates its own UIDs when datasources are created from provisioning files:

ssh docker-host 'curl -s -u admin:$PASSWORD http://localhost:3030/api/datasources' | jq '.[].uid'
# "P4169E866C3094E38"  ← VictoriaMetrics
# "P8E80F9AEF21F6940"  ← Loki

Any dashboard JSON that references "uid": "VictoriaMetrics" will silently get no data — Grafana finds no matching datasource and renders empty panels. The fix is to use the actual UIDs in the provisioned dashboard, which meant regenerating the JSON with the real values.

The underlying lesson: datasource UIDs in provisioned dashboards must match exactly what Grafana generates from the datasource provisioning config. The cleanest long-term fix is to pin the UID in the datasource provisioning YAML:

# ansible/roles/monitoring/files/datasources/victoriametrics.yml
datasources:
  - name: VictoriaMetrics
    uid: victoriametrics   # pin this — then use "victoriametrics" in all dashboards
    type: prometheus
    url: http://victoriametrics:8428

With a pinned UID, the dashboard JSON becomes portable and survives redeployment without UID drift.

cAdvisor Label Problem

The container panels were broken for a different reason. cAdvisor by default scrapes all cgroups on the host — Docker containers, systemd slices, the init scope, everything. And it exposes them all using the cgroup id label, not a name label:

ssh docker-host 'curl -s "http://localhost:8428/api/v1/query?query=container_cpu_usage_seconds_total" | \
  python3 -c "import sys,json; d=json.load(sys.stdin); print(d[\"data\"][\"result\"][0][\"metric\"])"'

# {'id': '/system.slice/docker-26282f99c048ab38ab5ef7bf238c9772d9e877e72bb5485090bd63d0800e94ee.scope',
#  'instance': 'docker-host', 'job': 'cadvisor'}

Docker containers appear as /system.slice/docker-<sha>.scope. No container name. No image name. 51 series total, all indistinguishable from each other in a Grafana legend.

The fix is the --docker_only flag. When cAdvisor runs with --docker_only=true, it filters to Docker containers only and includes the name label (the container name):

# stacks/monitoring/docker-compose.yml
cadvisor:
  command:
    - --docker_only=true

This cuts the series count dramatically and gives us usable labels. We’d actually already added this flag in a previous fix — but the dashboard queries were still written expecting name!="" which wouldn’t match the cgroup-path format. Fixing both together (flag + correct queries) is what makes container panels work.

Host Status Panel: Unmanaged Hosts

The Host Status panel was showing every scrape target including net-appliance, nvr, hk-bridge, and workstation — all DOWN, and all cluttered in the panel because the labels were overlapping at the default panel height.

The right fix was twofold:

1. Remove unmanaged hosts from the Ansible inventory’s managed groups — we moved net-appliance, nvr, and hk-bridge to the unmanaged group in hosts.yml. This stops Ansible from trying to configure them and generates the correct sudo password and unreachable errors.

2. Filter the dashboard query to exclude hosts we don’t expect to be up:

up{job="node_exporter", instance!~"net-appliance|nvr|hk-bridge|workstation"}

The monitoring stack still scrapes whatever the VictoriaMetrics scrape config says — but the dashboard is now opinionated about what it displays.

Instance Label Format

One more gotcha: node_exporter metrics came in with bare hostname labels (caddy, docker-host) not the host:port format you see in many public dashboards. Any dashboard JSON copied from Grafana’s dashboard library will use queries like:

node_cpu_seconds_total{instance="$instance"}

where $instance is a template variable populated from the label values. If the variable shows caddy but the query was written expecting caddy:9100, the variable won’t match.

Our scrape config uses relabeling to set instance to just the hostname:

# stacks/monitoring/docker-compose.yml (victoria metrics scrape config)
relabel_configs:
  - source_labels: [__address__]
    target_label: instance
    regex: '([^:]+):.*'
    replacement: '$1'

This is actually the cleaner approach for a homelab — short hostnames are more readable than 10.0.0.12:9100 — but it means any query using instance must be written with bare hostnames in mind.

The Result

After fixing datasource UIDs, cAdvisor flags, query labels, and panel filters, the dashboard now shows:

• docker-host host health: CPU %, RAM %, load avg, uptime, disk usage (two mount points)
• All managed hosts reachability: up status for every node_exporter target, green/red
• Container resource usage: CPU % and memory for top containers on docker-host (by name, not cgroup ID)
• Network I/O: receive and transmit rates per managed host
• Log volume: Loki log rate by container over time

The dashboard is provisioned via Ansible — no manual clicking in the Grafana UI. The JSON lives in stacks/monitoring/dashboards/homelab-overview.json and is deployed to /opt/stacks/stacks/monitoring/dashboards/ on docker-host, where Grafana’s dashboard provisioning config picks it up.

What’s Next

The monitoring stack is solid enough to build on. The natural next step is self-healing: wiring Alertmanager to n8n so that when a service goes down, an automated workflow attempts remediation before we get paged. We also want to add Proxmox hypervisor monitoring via pve_exporter — right now we can see what’s running inside VMs and containers, but not VM-level CPU/RAM/disk from Proxmox’s perspective.

The gap in coverage is deliberate for now: get the dashboard right before adding more signal.