djuntgen@juntgen.com
← all posts

Building a Homelab with AI · part 3

Ansible from Scratch: Bootstrapping a Fleet of LXC Containers

#homelab#ansible#lxc#proxmox#bootstrap#ssh

This is Part 3 of the Building a Homelab with AI series. Previously: Why IaC? | Next: GitOps with Portainer

The Starting Point

With the repo structure in place, the first real task was getting Ansible talking to every machine in the homelab. My dev machine (dev, 10.0.0.13) is an Ubuntu 24.04 LXC container on Proxmox, and it would serve as the Ansible control node.

The inventory covers everything: the Proxmox hypervisor, the Docker host, network appliances, services, even the DMZ:

all:
  children:
    proxmox:
      hosts:
        pve:
          ansible_host: 10.0.0.4
    docker:
      hosts:
        docker-host:
          ansible_host: 10.0.0.12
    services:
      hosts:
        n8n:
          ansible_host: 10.0.0.5
        influxdb:
          ansible_host: 10.0.0.9
        db:
          ansible_host: 10.0.0.20

Seven hosts across five groups, all reachable via SSH. Easy, right?

The Bootstrap Playbook (Take One… and Two… and Three)

The bootstrap playbook’s job is simple: create the <user> user with SSH key auth, set up passwordless sudo, and harden SSH. Standard stuff. But getting it to actually run against fresh LXC containers was an adventure.

Attempt 1: -u root. Fresh containers only have root. So I ran ansible-playbook bootstrap.yml -u root. Didn’t work. Ansible’s variable precedence means ansible_user set in group_vars overrides the -u CLI flag. This is a well-documented Ansible behavior, but it’s the kind of thing you don’t internalize until it bites you.

Attempt 2: -e ansible_user=root. Extra vars have the highest precedence in Ansible, so this should work. And it did — for connecting. But then the playbook tried to install sudo as the first step, and several containers didn’t have sudo installed. Catch-22: can’t become root without sudo, can’t install sudo without being root.

Attempt 3: become: false with raw module. Tried running raw commands as root without privilege escalation. Got closer, but the playbook was turning into a mess of conditional logic.

The Solution: Bootstrap via Proxmox

The breakthrough came from stepping back and asking: what access do I actually have? I have SSH access to the Proxmox hypervisor, and Proxmox can reach inside any LXC container using pct exec. No SSH required, no sudo required — pct exec runs as root inside the container by definition.

So I wrote bootstrap-via-pve.yml:

- name: Bootstrap LXC containers via Proxmox host
  hosts: pve
  gather_facts: false
  vars:
    containers:
      - { vmid: 101, name: n8n }
      - { vmid: 102, name: influxdb }
      - { vmid: 103, name: db }
  tasks:
    - name: Install sudo in container
      command: "pct exec {{ item.vmid }} -- apt-get install -y sudo"
      loop: "{{ containers }}"

    - name: Create admin user
      command: "pct exec {{ item.vmid }} -- useradd -m -s /bin/bash {{ admin_user }}"
      loop: "{{ containers }}"

    - name: Add to sudo group
      command: "pct exec {{ item.vmid }} -- usermod -aG sudo {{ admin_user }}"
      loop: "{{ containers }}"

It runs on the Proxmox host and uses pct exec to reach into each container by VMID. No SSH, no sudo, no chicken-and-egg problem. Once the user is created and SSH keys are deployed, the regular playbooks take over.

The group_vars Gotcha

Another landmine: I had group_vars/all.yml at the Ansible project root (ansible/group_vars/all.yml), but Ansible kept ignoring it. Turns out, group_vars must live adjacent to the inventory file. Since my inventory was at ansible/inventory/hosts.yml, Ansible expected ansible/inventory/group_vars/all.yml.

This is one of those things that’s technically in the docs but easy to get wrong when you’re setting up a project from scratch. Moving the directory fixed it immediately.

The Common Playbook

With bootstrap complete, the common.yml playbook handles ongoing configuration:

  • Essential packages (curl, wget, htop, vim, tmux, etc.)
  • Timezone set to America/Chicago
  • SSH hardening (disable root login, disable password auth)
  • Ensuring the SSH service is running and enabled

Nothing fancy, but now it’s codified. If I spin up a new LXC container tomorrow, it’s two playbook runs to go from bare Proxmox template to production-ready.

The Payoff

$ ansible all -m ping
pve | SUCCESS
docker-host | SUCCESS
n8n | SUCCESS
influxdb | SUCCESS
db | SUCCESS
net-appliance | SUCCESS
dev | SUCCESS

Seven hosts. All responding. All configured consistently. All from code that lives in Git.

The bootstrap struggle was humbling — three failed approaches before finding one that worked. But that’s the value of working through it with an AI pair programmer. Every failed attempt was fast to try and fast to learn from.

Next up: making Docker stacks deploy themselves with Portainer GitOps.