OpenClaw Day Two: Chasing Ghosts, Trusting the Tunnel, and Closing Security Debt

The Problem / Motivation

Post 14 ended on a high note: OpenClaw deployed to a dedicated Proxmox VM, sandboxed, firewalled, accessible via SSH tunnel. Infrastructure as Code, the way we like it.

But we left two loose threads dangling:

1. Plaintext secrets — openclaw_anthropic_api_key and openclaw_gateway_token were sitting in vault.yml as bare strings, only protected by file-level encryption. Fine in practice, but inconsistent with how we handle every other secret in the vault.

2. Caddy ACME failures — the Cloudflare DNS-01 challenge was throwing HTTP 403s, and we’d deferred investigating why.

Day two was supposed to be a quick cleanup session. It turned into something more interesting: a debugging rabbit hole, a hard revert, and a reminder about the value of accurate documentation.

What We Found

Three Cloudflare Tokens and a Ghost

The 403 on ACME was the first thing we pulled on. Diagnosing it meant auditing how Caddy actually gets its Cloudflare credentials — and we discovered we had three distinct tokens in play:

The root cause: the active token had Zone:DNS:Read but not Zone:DNS:Edit, which is what DNS-01 requires. The vault tokens — which had the right permissions — had expired and never been rotated.

Here’s the twist: the Caddy we were testing against wasn’t the one having problems.

During the investigation we did a deeper audit of the infrastructure — reading through the full Ansible repo, checking all the roles — and confirmed that the Caddy DMZ VM (10.0.10.10) had been fully operational for weeks, using a separate working token deployed via Ansible. The Docker-based Caddy on docker-host was the interim setup we’d never fully cleaned up.

The 403s were on the old stack. The production reverse proxy was fine. We were chasing a ghost.

Documentation Drift Is a Real Cost

This discovery exposed something uncomfortable: the agent system prompt we use to give Claude Code context about the homelab was significantly out of date. It still described Caddy as a Docker container on docker-host. It didn’t mention the openclaw or caddy VM hosts. The IaC directory structure it showed bore no resemblance to what was actually in the repo.

For AI-assisted infrastructure work, stale context is dangerous. The agent was reasoning about the wrong architecture — which directly contributed to the confusion about the ACME errors. We spent time investigating the wrong Caddy instance.

The fix was a systematic pass through every section of the system prompt: hosts table, Caddy configuration details, directory structure, key operational notes, Caddyfile update workflow. Everything cross-checked against the actual repo and live infrastructure.

We also added the agent definition file itself to the repo (homelab-agent.md), so it gets the same version-controlled treatment as everything else. Documentation that lives outside git is documentation that drifts.

The OpenClaw Stability Problem

With the environment fully mapped, we decided to try exposing the OpenClaw Control UI via Caddy — adding openclaw.example.com as a local_only subdomain pointing to 10.0.0.14:18789. That meant also updating the UFW rules on the openclaw VM to allow inbound on 18789 from the LAN.

It failed. Not because of the Caddy config or the firewall rules — those were fine. The gateway kept not binding, or binding and then going away. We’d see ss -tlnp show port 18789 for a few seconds, then nothing.

The culprit was subtle: running openclaw daemon status to check health was itself triggering a daemon restart. When the RPC probe fails — which it does within the 3-second startup window — the status command interprets that as a crashed gateway and attempts a restart. Run status while the gateway is starting, and you reset the clock.

This is a bug in OpenClaw’s daemon management. The distinction between “is the gateway healthy” and “start the gateway if it’s not running” should not live in the same command. But that’s what we were dealing with.

We also found a stale lock file at /tmp/openclaw-1000/gateway.*.lock that prevented clean restarts even after clearing the symptom.

The Revert Decision

At this point we had two options:

1. Keep debugging — trace the full startup sequence, find a reliable health check that doesn’t trigger restarts, potentially patch around the lock file issue.

2. Revert to the original design — gateway on loopback, SSH tunnel access only — and wait for a more stable OpenClaw release.

We chose option 2. The reasoning:

• The SSH tunnel works. ssh -L 18789:localhost:18789 <user>@openclaw costs five seconds and provides a stable, encrypted channel to the gateway.
• Exposing an unstable service through Caddy adds a layer of complexity that masks the underlying problem rather than solving it.
• The openclaw-gateway.service systemd unit is more reliable when nothing is probing it via commands that have restart side effects.
• OpenClaw is research infrastructure. It doesn’t need to be always-on or conveniently accessible from every browser on the LAN — it needs to be stable enough to run experiments.

The Caddyfile change was reverted, the UFW rule additions were removed, and the gateway bind was confirmed back to 127.0.0.1. The right time to expose OpenClaw via Caddy is after the upstream daemon management story stabilizes.

Closing the Vault Security Debt

With the infrastructure investigation settled, we returned to the original goal: encrypting the plaintext openclaw secrets.

Understanding the Vault Structure

Our vault.yml uses two layers of protection:

• File-level encryption: The entire file is AES256-encrypted with ansible-vault. Anyone who gets the file without the vault password sees an opaque blob.
• Per-variable !vault encryption: Individual secrets within the file are also encrypted as !vault inline strings.

The cloudflare and GitHub tokens already had per-variable encryption. The openclaw secrets, added during the initial deployment sprint, were plaintext inside the encrypted file — protected by the outer layer, but inconsistent with how everything else was handled.

The Encryption Process

ansible-vault encrypt_string is the right tool, but it has a quirk in our setup: the ansible.cfg references <vault-password-file> via vault_password_file, which causes the vault-ids to be registered twice (default,default). The fix is to always pass --encrypt-vault-id default explicitly:

ansible-vault encrypt_string 'your-secret-value' \
  --name 'your_variable_name' \
  --encrypt-vault-id default \
  --vault-password-file <vault-password-file>

We ran this for both secrets, then did the replacement non-interactively:

# Decrypt to temp file
ansible-vault decrypt inventory/group_vars/all/vault.yml \
  --output=/tmp/vault_decrypted.yml \
  --vault-password-file <vault-password-file>

# Python substitution (regex replace the plaintext section)
python3 << 'EOF'
import re
# ... replace plaintext openclaw lines with !vault blocks
EOF

# Re-encrypt
ansible-vault encrypt /tmp/vault_new.yml \
  --vault-password-file <vault-password-file> \
  --encrypt-vault-id default

# Replace original
cp /tmp/vault_new.yml inventory/group_vars/all/vault.yml
rm /tmp/vault_decrypted.yml /tmp/vault_new.yml

Important: delete the temp decrypted file immediately after use. A plaintext secrets file sitting in /tmp is exactly the kind of footgun that defeats the purpose of vault encryption.

Verification

Before committing, we confirmed Ansible could decrypt both secrets correctly:

ansible all -m debug -a "var=openclaw_anthropic_api_key" --limit openclaw
ansible all -m debug -a "var=openclaw_gateway_token" --limit openclaw

Both returned the expected values. The vault is clean.

Lessons Learned

1. Three tokens is two too many. When you have multiple Cloudflare API tokens in different places serving the same purpose, confusion is inevitable. The fix isn’t to debug the 403 — it’s to consolidate to a single authoritative token in vault and make sure everything uses it. That’s a cleanup item for a future session.

2. Documentation that drifts costs real time. Stale context in the agent system prompt caused us to investigate the wrong Caddy instance for longer than necessary. For AI-assisted infrastructure work specifically, accurate context is not optional — the AI reasons about what you tell it, and if what you tell it is wrong, the reasoning is wrong too.

3. Know when to stop and revert. The instinct to push through and make a feature work is strong, especially when the fix should be straightforward. But OpenClaw’s daemon management quirks are upstream problems we can’t control. The SSH tunnel is a perfectly good solution for research infrastructure. Recognizing that and reverting was the right call — and the revert itself took about five minutes.

4. File-level and per-variable vault encryption serve different purposes. File-level encryption protects the whole file from unauthorized access. Per-variable !vault encryption means individual secrets can live in non-encrypted files and still be protected, and it makes the encryption granularity explicit in code review. Using both is belt-and-suspenders, but it’s also consistent: every secret in the vault looks the same, which makes auditing easier.

5. Temp files with decrypted secrets are a hazard. The decrypt-edit-encrypt workflow is necessary when you can’t use ansible-vault edit non-interactively, but it creates a window where secrets exist in plaintext on disk. Delete them immediately, don’t leave decrypted files sitting around between steps.

What’s Next

• Cloudflare token consolidation: One token, in vault, with correct Zone:DNS:Edit permissions. Rotate and verify. Remove the hardcoded token from the old compose file.
• OpenClaw stability monitoring: Wait for a more stable release, then reconsider whether LAN-accessible Control UI via Caddy makes sense.
• OpenClaw vault token rotation: The openclaw_gateway_token was auto-generated during initial setup. It should be rotated periodically since it controls gateway access.
• DNS server migration: The planned move from the edge firewall as DHCP/DNS to a dedicated LXC DNS server is still pending. That would give us proper split-horizon DNS and make local-only subdomains available without going through Cloudflare.