What This Is
This is the story of how I took a homelab held together by tribal knowledge and muscle memory and turned it into a fully codified, version-controlled, reproducible infrastructure — pair-programmed with an AI.
I have been a software engineer and Linux administrator for 26 years. I run a homelab on Proxmox with a fleet of LXC containers and VMs: a Docker host, a reverse proxy, a database server, home automation, network monitoring, and more. For years, every service was configured by hand. If a disk failed or a container needed rebuilding, I was piecing things together from memory and terminal scrollback.
This series documents the journey from that state to Infrastructure as Code: Ansible managing every host, Docker Compose stacks auto-deploying via GitOps, a reverse proxy isolated in a DMZ, and security hardening across the board. Every line of code, every playbook, every configuration was built in conversation with Claude Code, Anthropic’s AI CLI tool.
Who This Is For
- Homelab operators who want to level up from manual configuration to Infrastructure as Code
- Linux administrators who want a practical security hardening reference
- Engineers curious about AI-assisted workflows who want to see real examples, not toy demos
- Anyone running self-hosted services who has ever asked “what happens when this disk fails?”
You should be comfortable with the Linux command line, SSH, and basic networking. I do not explain what a container is or how to use apt. But I do explain Ansible variable precedence, sudoers file naming rules, and why your reverse proxy should not live on the same subnet as your Docker host.
The Series
The AI Toolkit
| # | Post | Description |
|---|---|---|
| 1 | Claude Code: How an AI Pair Programmer Made Me a Super Full Stack Engineer | What becomes possible when you pair deep domain expertise with the latest LLMs. Prompt engineering, real workflow examples, best practices, and anti-patterns for AI-assisted infrastructure work. |
Infrastructure as Code
| # | Post | Description |
|---|---|---|
| 2 | Why Infrastructure as Code? From Manual Configs to Git-Driven Homelab | The motivation for codifying a homelab. Repo structure design, tooling choices (Ansible, Docker Compose, Portainer GitOps), and why IaC is about documentation as much as automation. |
| 3 | Ansible from Scratch: Bootstrapping a Fleet of LXC Containers | Getting Ansible talking to every host. The chicken-and-egg bootstrap problem, three failed approaches, and the pct exec breakthrough. |
| 4 | GitOps for Docker Stacks with Portainer | Auto-deploying Docker Compose stacks from Git. Portainer configuration, the self-management paradox, and the Portainer API. |
| 5 | Moving Caddy to the DMZ: Isolating Your Reverse Proxy | The single biggest security improvement in the project. VM vs LXC, Alpine vs Ubuntu, Docker vs bare binary, and Ansible Vault for secrets. |
| 6 | Security Review: Hardening a Homelab IaC Repo | A full security audit of everything we built. Real findings, false positives, defense-in-depth analysis, and the architectural security win. |
| 7 | Lessons Learned: AI-Assisted Infrastructure as Code | What worked, what required course correction, and broader lessons about AI-assisted infrastructure work. |
Linux Security Hardening
| # | Post | Description |
|---|---|---|
| 8 | Sudoers Demystified: Understanding Privilege Escalation | How sudo actually works, anatomy of sudoers rules, the NOPASSWD debate, file hierarchy, permissions, and the visudo imperative. |
| 9 | Linux File Permissions Deep Dive: The Bits That Protect Your Homelab | The permission model, octal notation, ownership, directory permissions, special bits (setuid/setgid/sticky), Docker permission gotchas, and umask. |
| 10 | Automating Sudoers with Ansible: Never Manually Edit Sudoers Again | Building a reusable Ansible role for sudoers management. Bootstrap via Proxmox, visudo validation, permission enforcement, and safe testing workflows. |
| 11 | Anti-Patterns and War Stories: Sudoers Mistakes That Break Things at 3 AM | Ten anti-patterns collected from 26 years of Linux administration. The hardening checklist. Why your homelab security matters. |
Developer Experience & Tooling
| # | Post | Description |
|---|---|---|
| 12 | Sites, DNS, and MOTDs: A Productive Homelab Session | Planning a multi-site web architecture, registering a domain, cleaning up DNS, rotating Cloudflare tokens to least privilege, and deploying a dynamic MOTD across the fleet. |
| 13 | Persistent Terminal Sessions on the Control Node with Zellij | Installing Zellij for persistent SSH sessions. Session serialization, SSH auto-attach via .bashrc, the login vs non-login shell PATH trap, and Nerd Fonts for iTerm2. |
AI Agent Infrastructure
| # | Post | Description |
|---|---|---|
| 14 | OpenClaw on Dedicated VM: IaC-First AI Agent Deployment | Deploying an AI agent framework on a dedicated Proxmox VM. Why not Docker-on-docker-host, the Ansible role design, sandbox containers, Day-0 UFW hardening, and SSH tunnel access. |
| 15 | OpenClaw Day Two: Chasing Ghosts, Trusting the Tunnel, and Closing Security Debt | Debugging a Cloudflare ACME 403 and finding the wrong Caddy. The cost of stale documentation in AI-assisted infrastructure work. OpenClaw daemon instability, a deliberate revert, and closing out plaintext secrets in Ansible Vault. |
| 16 | Cleanup Day: Revoking Dead Tokens, Fixing the Vault, and Learning to Ask Before You Build | Removing a ghost Caddy container with a hardcoded token, auditing and rotating Cloudflare credentials, simplifying Ansible Vault from double to single encryption, and learning that rsync + docker compose beats GitHub Actions + GHCR + Watchtower for a homelab deploy. |
| 17 | OpenClaw Round Two: LAN Deployment, Linger Lessons, and Caddy Integration | Promoting OpenClaw from SSH-tunnel-only to a proper LAN reverse proxy behind Caddy. Four bugs hit and fixed: systemd linger, daemon status triggering restarts, gateway bind mode, and allowedOrigins WebSocket rejection. |
| 18 | Ansible Best Practices: Six Things We Fixed Before They Became Problems | A best practices review of the full Ansible codebase. Vault variable indirection, idempotency fixes, SSH AllowUsers, role defaults, removing hardcoded vars, and adding a site.yml master playbook. |
Backup and Resilience
| # | Post | Description |
|---|---|---|
| 19 | Phase 3: Backups with Proxmox Backup Server and Google Drive | Building a 3-2-1 backup chain: PBS LXC provisioned via Terraform, configured via Ansible, nightly PVE backup jobs, and weekly rclone offsite sync to Google Drive. Five gotchas documented: enterprise repo trap, NTS in LXC, OAuth token format, keep-last minimum, and ACL prerequisites. |
| 20 | Consistent Time Sync Across All Proxmox Hosts with Chrony | Diagnosing broken time sync on fresh LXC containers — root cause: Ubuntu/Debian defaulting to NTS (TCP 4460), which fails in Proxmox LXC network namespaces. Fixing it fleet-wide with a two-task Ansible approach and a plain NTP pool template. |
Monitoring and Observability
| # | Post | Description |
|---|---|---|
| 21 | Phase 2: Monitoring and Observability with VictoriaMetrics, Grafana, and Loki | Full observability for every host: node_exporter via Ansible, VictoriaMetrics + Grafana + Loki + cAdvisor + Alertmanager as a Docker stack, file_sd dynamic target discovery from the Ansible inventory, seven alert rules including a dead man’s switch, and a lesson in how Portainer GitOps actually handles new stacks. |
| 22 | Security Audit: Scrubbing Usernames and Secrets from a Public Git Repository | A systematic audit of what was leaking from a public homelab repo: usernames hardcoded in 18 blog posts, Ansible task files, inventory, and API examples. How we moved admin_user to Ansible Vault, parameterised the openclaw role, used sed for bulk doc cleanup, and renamed the agent file. |
Reading Order
The series is designed to be read in order, but each post stands on its own. If you are here for a specific topic:
- Just want to learn about AI-assisted engineering? Start with Post 1.
- Want to set up IaC for your homelab? Start with Post 2 and read through Post 7.
- Here for Linux security hardening? Start with Post 8 and read through Post 11.
- Want the full journey? Start at Post 1 and read straight through.
The Tech Stack
For reference, here is what the homelab looks like:
- Hypervisor: Proxmox VE on a dedicated host
- Docker Host: Ubuntu 24.04 LXC (
docker-host), running stacks via Portainer - Reverse Proxy: Caddy on a dedicated Ubuntu VM in the DMZ, with Cloudflare DNS-01 ACME
- Configuration Management: Ansible, running from an Ubuntu 24.04 LXC control node (
dev) - GitOps: Portainer watching a GitHub repo, polling every 5 minutes
- Secrets: Ansible Vault for IaC secrets, Portainer env vars for runtime secrets
- DNS: Cloudflare (external), the edge firewall (internal, migration planned)
- AI Tooling: Claude Code with Claude Opus 4.6
Every Commit Tells a Story
Every AI-assisted commit in the homelab repo includes:
Co-Authored-By: Claude Opus 4.6 <[email protected]>This is not about giving credit to a machine. It is about transparency. When future-me reads the Git log, I want to know how each change was made. The AI is a tool. The decisions, the architecture, and the accountability are mine.
Let us get started. Post 1: Claude Code awaits.