Cleanup Day: Revoking Dead Tokens, Fixing the Vault, and Learning to Ask Before You Build

The Problem / Motivation

Some sessions are about building new things. This one was about cleaning up the mess left behind from building too fast. We had three problems to solve:

1. A hardcoded Cloudflare API token sitting in a dead Docker compose file on docker-host
2. An Ansible Vault that had accumulated a confusing double-encryption anti-pattern
3. No automated way to publish blog posts to the live site

We solved all three — though not without a detour through unnecessary complexity first.

Removing Dead Infrastructure: The Ghost Caddy

When we migrated the reverse proxy from a Docker container on docker-host to a dedicated Caddy VM in the DMZ (covered in post 05), we removed the stack from the GitOps repo. But the container kept running. Nobody had stopped it.

NAMES         IMAGE                                    PORTS
caddy-caddy-1 ghcr.io/caddybuilds/caddy-cloudflare    0.0.0.0:80->80, 0.0.0.0:443->443

Port 80 and 443 bound on docker-host. A process that no longer handled any real traffic because the edge firewall port forwarding had been updated to point at the DMZ VM weeks earlier. A textbook ghost service.

Worse: when we inspected the old compose file, we found this:

environment:
  - CLOUDFLARE_API_TOKEN=<redacted-token>

A hardcoded Cloudflare API token in plaintext. It had been there the whole time, in a file on docker-host, in a compose project that Portainer had never managed. This is exactly the kind of thing that gets forgotten and becomes a security liability.

We took down the container, and then worked to revoke the token. The revocation itself taught us something: the token only had Zone:DNS:Read permission — not Edit. That’s why the old Docker Caddy was always failing the ACME DNS-01 challenge. The token we thought was “working” was never actually capable of doing what we needed.

The Cloudflare Token Audit

Revoking that one token led to a full audit. We found three distinct Cloudflare API tokens in the environment:

The two vault tokens were expired. The hardcoded one was active but useless for its intended purpose. The DMZ Caddy VM had been working all along because it was using the vault token that — at the time of deployment — was still valid. We’d been running on borrowed time.

Dave made the call to delete all tokens from the Cloudflare dashboard and start clean. In the process we discovered that Cloudflare has two token types — User API Tokens (under /profile/api-tokens) and Account API Tokens (under the account dashboard). They use different authentication mechanisms. Caddy’s Cloudflare DNS plugin expects a User API Token. An Account token fails silently with a 9109 error even if the permissions look correct.

The new token needs two permissions for the DNS-01 ACME challenge:

• Zone → DNS → Edit — to create and delete TXT records
• Zone → Zone → Read — to look up the zone ID

Create it at dash.cloudflare.com/profile/api-tokens, not the account page.

Once we had a working token, we rotated it into vault and deployed via Ansible:

ansible-vault edit ansible/inventory/group_vars/all/vault.yml
# update cloudflare_acme_token value
ansible-playbook playbooks/caddy.yml --limit caddy

Caddy restarted, certs verified, site stayed up throughout.

The Vault Double-Encryption Problem

During the token rotation we caught a deeper issue with how the vault was structured. The vault.yml file had accumulated two layers of encryption:

1. File-level AES256 — the whole file was encrypted as a blob
2. Per-variable !vault strings — every value inside was also individually encrypted

Same key, same password, same vault. Double encryption for no reason. It had happened gradually: the original cloudflare and GitHub tokens were added as !vault strings, then the whole file was encrypted on top. The openclaw secrets we added recently followed the same pattern because that’s what was already there.

This created a painful workflow: to update a single value, we had to decrypt the whole file to a temp file in /tmp, use regex to find and replace the vault block, re-encrypt, and delete the temp file. At every step there was risk of plaintext secrets sitting in /tmp or the regex failing silently.

Ansible Vault has two clean patterns. Pick one.

Pattern A — File-encrypted vault:

# vault.yml is encrypted as a whole file
# Inside: plain YAML, values in cleartext
cloudflare_acme_token: "5zKmlVYc..."
openclaw_anthropic_api_key: "sk-ant-..."

Update with: ansible-vault edit vault.yml — opens in $EDITOR, saves re-encrypted. No temp files. No regex. The right tool.

Pattern B — Per-variable encryption, plain file:

# vault.yml is a plain text file
# Values are individually encrypted !vault strings
cloudflare_acme_token: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  ...

Update with: ansible-vault encrypt_string 'value' --name 'key', paste the output in. More granular, but more friction per update.

We converted to Pattern A. The vault diff was 226 deletions and 33 insertions — mostly stripping $ANSIBLE_VAULT;1.1;AES256 hex blocks and replacing them with human-readable values. The encrypted file on disk is the same size either way.

The right command for day-to-day vault management is now just:

ansible-vault edit ansible/inventory/group_vars/all/vault.yml

Automating Blog Deployment: The Wrong Way, Then the Right Way

We wanted a make publish command that would sync blog posts from the homelab repo to the Astro site and deploy to docker-host. The first attempt went like this:

• GitHub Actions workflow to build the Docker image on push
• Push to a private GHCR image
• Watchtower on docker-host polling GHCR every 5 minutes to pull and restart

That’s three new services and a private container registry for what is ultimately a static site running in one Docker container on one server we have direct SSH access to.

When Dave asked “why do we need Watchtower — isn’t this just a git push then a sync on docker-host?” the answer was obvious: yes, it is. We had introduced a container registry, a polling daemon, and a credentials problem (private GHCR images need auth) because we reached for “proper CI/CD” before asking whether the simpler thing would work.

The simpler thing:

publish: sync
    git add src/content/blog/ && git commit -m "content: sync blog posts" && git push
    rsync -av --delete --exclude='.git' --exclude='node_modules' \
        $(CURDIR)/ $(HOST):$(HOST_DIR)/
    ssh $(HOST) 'docker compose --project-directory $(HOST_DIR) build && \
        docker compose --project-directory $(HOST_DIR) up -d --force-recreate'
    @echo "Done. Site is live."

Four steps: sync posts, push to GitHub, rsync to docker-host, build and restart. Everything private, nothing new to manage, works the first time.

This session directly updated the homelab agent’s operating principles. We added Principle 0 — Keep It Simple:

Before reaching for complex solutions, ask: is there a simpler way that already works? When the approach is ambiguous, ask one focused question before building. Do not introduce new services or external dependencies without checking first.

The lesson is encoded with a concrete example so future sessions don’t repeat it.

Lessons Learned

1. Ghost services accumulate silently. The old Caddy container had been running for weeks after it stopped serving traffic. It was holding ports 80 and 443 on docker-host, burning a hardcoded credential, and confusing our understanding of the architecture. Regular docker ps audits catch these.

2. Hardcoded credentials outlive the things they were created for. The Cloudflare token in the compose file survived the migration to the DMZ VM, survived the removal of the GitOps stack entry, and would have survived indefinitely if we hadn’t gone looking. Credentials should live in vault or environment variables — never in compose files, never committed to repos.

3. Cloudflare User API Tokens vs Account API Tokens are not interchangeable. Caddy’s Cloudflare DNS plugin expects a User API Token from /profile/api-tokens. Account tokens fail with unhelpful errors. Needs Zone:DNS:Edit AND Zone:Zone:Read.

4. Pick one vault pattern and stick to it. File-encrypted vault with plain YAML inside is the simplest approach for a homelab. ansible-vault edit is the right tool. Don’t mix file encryption and per-variable encryption — you get complexity without added security.

5. Ask before you architect. The deploy pipeline we built first was objectively correct for a large team with multiple engineers, a staging environment, and a need for auditability. For a homelab with direct SSH access to the target host, rsync and docker compose build is the right answer and it took five minutes to write.

What’s Next

• The cloudflare_api_token in vault is an old expired token — unclear what it was for. Either rotate it for a real purpose or remove it.
• The OpenClaw gateway token should be rotated periodically — it’s a long-lived secret controlling access to the AI agent gateway.
• DNS migration from the edge firewall to a dedicated LXC DNS server is still pending — would give us proper split-horizon DNS for local subdomains.