djuntgen@juntgen.com
← all posts

Building a Homelab with AI · part 18

Ansible Best Practices: Six Things We Fixed Before They Became Problems

#homelab#ansible#vault#ssh#idempotency

The Prompt

We asked for a best practices review of the Ansible configuration. We expected one or two minor notes. We got six concrete issues worth fixing immediately, before they caused real problems down the line. This post covers each one and why it mattered.

Issue 1: Caddy Downloaded Its Binary on Every Run

The Caddy role used force: true on the get_url task:

# Before
- name: Install caddy with cloudflare plugin
  ansible.builtin.get_url:
    url: "https://caddyserver.com/api/download?..."
    dest: /usr/bin/caddy
    mode: "0755"
    force: true

force: true means “always download, even if the file exists.” Every playbook run was hitting the Caddy download API unnecessarily — re-downloading a 30MB binary, triggering a restart, and showing changed in the output when nothing had actually changed.

The fix adds a version check first:

# After
- name: Check installed Caddy version
  ansible.builtin.command: caddy version
  register: caddy_installed_version
  changed_when: false
  failed_when: false

- name: Install caddy with cloudflare plugin
  ansible.builtin.get_url:
    url: "{{ caddy_binary_url }}"
    dest: /usr/bin/caddy
    mode: "0755"
    force: true
  when: caddy_installed_version.rc != 0

If Caddy is already installed (exit code 0), skip the download. We also moved the URL into roles/caddy/defaults/main.yml so it can be overridden per-host if needed.

This is the idempotency principle: a task that shows changed when nothing changed is lying to you. Eventually you stop trusting the output.

Issue 2: Vault Variables Had No Indirection Layer

The vault file (inventory/group_vars/all/vault.yml) held secrets like cloudflare_acme_token, github_homelab_pat, etc. — and those names were used directly in templates and tasks.

That’s the problem: to find where cloudflare_acme_token is used, you’d have to remember it exists and grep for it. More importantly, you can’t audit variable usage without decrypting the vault.

The standard pattern is a vault_ prefix on vault keys, with plain-name references in the vars files:

# vault.yml (encrypted)
vault_cloudflare_acme_token: "abc123..."
vault_github_homelab_pat: "ghp_..."

# group_vars/all/vars.yml (plaintext)
cloudflare_acme_token: "{{ vault_cloudflare_acme_token }}"

# group_vars/control/vars.yml
github_homelab_pat: "{{ vault_github_homelab_pat }}"

Now grep cloudflare_acme_token works without decrypting anything. The vault file just stores values; the vars files define the variable names the codebase actually uses. The user had already renamed all vault keys to vault_ — this change just wired up the indirection layer.

Issue 3: SSH Hardening Was Missing AllowUsers

The base role hardened SSH in three ways:

- PermitRootLogin no
- PasswordAuthentication no
- X11Forwarding no

Solid defaults. But there was a gap: even with root login and password auth disabled, any user account on the system could attempt SSH. If a package install created a service user with a shell, that account was implicitly allowed.

AllowUsers is an explicit allowlist — SSH refuses any user not on it, full stop:

- { regexp: "^#?AllowUsers", line: "AllowUsers {{ admin_user }}" }

Since admin_user is defined in group_vars/all/vars.yml as <user>, this applies consistently across all hosts. Defense in depth: if something else goes wrong, there are fewer attack surfaces.

Issue 4: bootstrap-via-pve.yml Had Hardcoded Variables

The bootstrap playbook for LXC containers started like this:

vars:
  admin_user: <user>
  ssh_public_key: "ssh-ed25519 AAAA..."
  lxc_containers:
    - { vmid: 101, name: db }
    - { vmid: 106, name: influxdb }
    - { vmid: 107, name: proxy }
    - { vmid: 112, name: n8n }

Three problems:

  1. admin_user and ssh_public_key were already in group_vars/all/vars.yml — this was a duplicate that could drift out of sync
  2. lxc_containers is configuration data, not playbook logic — it belongs in inventory
  3. Adding a new container meant editing the playbook, not the inventory

The fix: remove the inline vars block and add group_vars/proxmox/vars.yml:

# inventory/group_vars/proxmox/vars.yml
lxc_containers:
  - { vmid: 101, name: db }
  - { vmid: 106, name: influxdb }
  - { vmid: 107, name: proxy }
  - { vmid: 112, name: n8n }

Now the playbook itself has no hardcoded data. admin_user and ssh_public_key flow in from the global group vars automatically.

Issue 5: Roles Had No defaults/main.yml

Roles can define variables at three layers:

  • defaults/main.yml — lowest priority, easily overridden
  • vars/main.yml — higher priority, intentionally hard to override
  • group_vars/ — external to the role, specific to this inventory

We had variables like openclaw_node_version: "22" living in group_vars/ai/vars.yml rather than as a role default. That works, but it means the role can’t be reused without knowing which group vars to also copy.

We added defaults/main.yml to the three roles that had overridable configuration:

# roles/openclaw/defaults/main.yml
openclaw_node_version: "22"
openclaw_config_dir: /home/<user>/.openclaw
openclaw_workspace_dir: /home/<user>/openclaw/workspace

# roles/caddy/defaults/main.yml
caddy_binary_url: "https://caddyserver.com/api/download?os=linux&arch=amd64&p=..."
caddy_config_dir: /etc/caddy
caddy_data_dir: /var/lib/caddy

These defaults are documented values inside the role itself. Override them in group_vars if needed, but you don’t have to.

Issue 6: No Master Playbook

Running the full environment meant remembering the right playbook order. There was no single entrypoint for “configure everything.”

playbooks/site.yml fixes that:

- name: Common baseline (all managed hosts)
  import_playbook: common.yml

- name: Dev control node
  import_playbook: dev.yml

- name: Caddy reverse proxy (DMZ)
  import_playbook: caddy.yml

- name: OpenClaw AI agent
  import_playbook: openclaw.yml

- name: MOTD (all hosts)
  import_playbook: motd.yml

ansible-playbook playbooks/site.yml runs everything in dependency order. --limit <host> still works to target a single host across all playbooks. Useful for disaster recovery or provisioning a fresh environment from scratch.

What Didn’t Change

Some things were already done right and worth calling out:

  • Vault password in <vault-password-file>, referenced in ansible.cfg — never committed
  • EnvironmentFile pattern for secrets in systemd services — secrets never in process environment by default
  • no_log: true on the gh auth task — token never appears in output
  • become: false in the right places — user services run as the right user
  • GPG-signed repositories for Docker and NodeSource
  • Source IP restrictions in UFW rules — not just port allowlisting

Lessons

The review pattern itself is worth repeating: ask “does this follow best practices?” on a working codebase, before it matters. Most of these issues wouldn’t have caused outages — but they would have caused confusion, made auditing harder, or silently broken idempotency.

Six fixes, all low-risk, all committed in one session. That’s the kind of maintenance that pays off when you’re debugging at 11pm and the playbook output means what it says.