Phase 2: Monitoring and Observability with VictoriaMetrics, Grafana, and Loki

The Problem

We had 14 VMs and LXCs with no idea what was happening inside any of them. No metrics, no dashboards, no alerts. If a disk was filling up or a service was quietly consuming memory, we would not know until something broke. Phase 2 of the homelab IaC project set out to fix that with full observability: every host emitting metrics, a central stack collecting and visualising them, and alerts firing within minutes of something going wrong.

Choosing the Right Tools

VictoriaMetrics, Not Prometheus

Prometheus is the default choice for metrics in this space, but for a homelab with ~10 hosts VictoriaMetrics is strictly better. It uses 3–5× less RAM and disk for identical PromQL workloads, deploys as a single binary or single container (no operator required), and is a drop-in replacement for Prometheus — same scrape config format, same /metrics endpoint, same PromQL query language. There is no reason to run Prometheus at homelab scale unless you have an existing investment in the Prometheus ecosystem.

We set a 90-day retention period. With ~10 hosts scraping at 30-second intervals, storage comes to around 1–2 GB/month. Comfortable on docker-host.

node_exporter for Host Metrics

Every Ansible-managed host gets the Prometheus node_exporter binary installed as a systemd service. It exposes CPU, memory, disk, network, and filesystem metrics at :9100/metrics in Prometheus format. The binary is small (~20 MB), resource usage is negligible, and it is the universal standard for Linux host metrics.

A UFW rule restricts port 9100 to ingress from docker-host (10.0.0.12) only. Nothing else on the LAN can scrape it.

cAdvisor for Container Metrics

cAdvisor (Container Advisor) runs as a privileged container on docker-host alongside the monitoring stack. It exposes Docker container metrics — CPU, memory, network I/O per container — at :8080/metrics. This gives us a second layer of visibility: host-level metrics from node_exporter, and workload-level metrics from cAdvisor.

We bind both to 127.0.0.1 on docker-host. They are only accessible through Caddy.

Grafana for Dashboards

No real decision here. Grafana is the universal dashboard tool and has first-class support for VictoriaMetrics (via the Prometheus datasource type). We run it on docker-host at :3030 (internal) and proxy it through Caddy at grafana.example.com with the local_only snippet — LAN access only.

Loki + Promtail for Log Aggregation

Loki is Grafana Labs’ log aggregation system. Unlike Elasticsearch, it indexes only log labels (not the full text), which makes storage far more efficient. Promtail is the agent that ships logs to Loki. We run Promtail on docker-host, where it scrapes Docker container logs via the Docker socket and ships them to Loki. Thirty-day retention keeps storage manageable.

The payoff: from a single Grafana screen you can pivot from a metrics spike directly into the container’s log stream at that timestamp.

Alertmanager for Routing

VictoriaMetrics evaluates alert rules and pushes firing alerts to Alertmanager. Alertmanager handles grouping, deduplication, silencing, and routing. We configured two receivers:

• default — email for all warning-severity alerts
• critical — email plus an n8n webhook for HostDown and DiskCritical

The n8n webhook is the hook for Phase 4 self-healing: when a host goes down, Alertmanager calls n8n, which will try to SSH in and restart the offending service before escalating to the operator.

Architecture: File-Based Service Discovery

The most important design decision in the scrape configuration was avoiding hardcoded targets. If we list every host IP directly in scrape.yml, adding a new host means editing the scrape config and restarting VictoriaMetrics.

Instead, we use VictoriaMetrics’s file_sd (file-based service discovery). The scrape config references a directory of YAML target files:

scrape_configs:
  - job_name: node_exporter
    file_sd_configs:
      - files:
          - /etc/victoriametrics/file_sd/node_exporter.yml
        refresh_interval: 30s

Those YAML files are generated by Ansible from the inventory and written to /etc/victoriametrics/file_sd/ on docker-host. VictoriaMetrics hot-reloads them every 30 seconds — no restart required when a host is added or removed.

The flow for adding a new host to monitoring:

1. Add the host to ansible/inventory/hosts.yml
2. Run ansible-playbook playbooks/monitoring.yml
3. Ansible installs node_exporter on the new host and regenerates the file_sd target files on docker-host
4. VictoriaMetrics picks up the new target within 30 seconds

ansible/inventory/hosts.yml
        │
        ▼ (ansible-playbook monitoring.yml)
/etc/victoriametrics/file_sd/node_exporter.yml   ← bind-mounted into container
        │
        ▼ (hot-reload every 30s)
VictoriaMetrics scrape targets

This means stacks/monitoring/scrape.yml never needs to be edited for new hosts. The inventory is the single source of truth.

The Stack

stacks/monitoring/docker-compose.yml defines five services, all bound to 127.0.0.1 on docker-host:

All services have restart: unless-stopped and Docker healthchecks. Grafana depends on VictoriaMetrics being healthy before starting; Promtail depends on Loki. Startup ordering is handled by depends_on with condition: service_healthy.

Secrets are injected via Portainer stack environment variables — admin password, SMTP credentials, and the n8n webhook URL never touch the repo.

Alert Rules

We defined seven alert rules in stacks/monitoring/alerts.yml:

- alert: HostDown          # node_exporter unreachable for 2m
- alert: DiskWarning       # >85% full for 5m
- alert: DiskCritical      # >95% full for 2m
- alert: MemoryHigh        # >90% sustained for 5m
- alert: ContainerDown     # container_last_seen absent for 2m
- alert: BackupJobFailed   # restic exit code != 0 or no success in 36h
- alert: OffsiteSyncFailed # rclone gdrive exit code != 0
- alert: Heartbeat         # always fires — dead man's switch

The Heartbeat rule is a dead man’s switch. It uses vector(1) — a VictoriaMetrics expression that always returns 1. Configure Alertmanager to send an alert if this stops firing and you have monitoring of your monitoring: if the stack goes down silently, the absence of the heartbeat will notify you.

An inhibit rule suppresses MemoryHigh if HostDown is already firing for the same instance — a host that is unreachable will naturally appear to have high memory usage, and we do not want duplicate noise.

A Lesson in Portainer GitOps

We hit a non-obvious snag during deployment: the monitoring stack files were committed and pushed, but Portainer never picked them up. The assumption was that Portainer scans the repository for new stack directories and auto-registers them.

This is wrong. Portainer’s GitOps polling only updates stacks it already knows about. It does not discover new stacks from the repository. A new stack must be registered once — either via the Portainer UI or the API — before auto-polling takes over.

The API command to register a new stack is:

source ~/.portainer.env
curl -sk -X POST \
  "$PORTAINER_URL/api/stacks/create/standalone/repository?endpointId=3" \
  -H "X-API-Key: $PORTAINER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "monitoring",
    "repositoryURL": "$GITHUB_REPO_URL",
    "repositoryReferenceName": "refs/heads/main",
    "filePathInRepository": "stacks/monitoring/docker-compose.yml",
    "repositoryAuthentication": true,
    "repositoryUsername": "$GITHUB_USER",
    "repositoryPassword": "$GITHUB_PAT",
    "autoUpdate": { "interval": "5m" }
  }'

After this one-time registration, future pushes to main trigger automatic redeployment within 5 minutes. We documented this in CLAUDE.md and the SRE agent definition so it is not forgotten next time.

Deployment Steps

With the stack committed, the full deployment sequence is:

1. Register the stack in Portainer (one time) — API call above
2. Set env vars in Portainer UI — GF_SECURITY_ADMIN_PASSWORD, ALERTMANAGER_EMAIL_FROM, ALERTMANAGER_EMAIL_TO, SMTP credentials, ALERTMANAGER_N8N_WEBHOOK
3. Deploy node_exporter to all hosts:

   cd ~/homelab/ansible
   ansible-playbook playbooks/monitoring.yml

4. Verify — curl http://10.0.0.12:9100/metrics from docker-host, then check grafana.example.com for green datasources

The monitoring playbook does two things: installs node_exporter on every managed host (excluding unmanaged and proxmox groups), then generates the file_sd target files on docker-host. VictoriaMetrics picks up the targets within 30 seconds of the playbook completing.

What’s Next

Phase 4 is self-healing: the Alertmanager → n8n webhook we configured here becomes the trigger for automatic service restarts. When HostDown fires, n8n SSHes into the host, restarts the service, and silences the alert if recovery succeeds. If three attempts fail, it escalates to the operator.

The monitoring stack we built in Phase 2 is the prerequisite: you cannot automate remediation of things you cannot observe.