For most of this series, Portainer GitOps has been our Layer 3 — the piece that watched our GitHub repo and automatically deployed Docker Compose stacks to docker-host whenever we pushed a change. On paper it was exactly what we wanted: commit, push, done. In practice, it stopped working, and the troubleshooting journey led us somewhere better.
What Went Wrong
The symptom was straightforward: stacks weren’t updating on push. We’d commit a change to stacks/monitoring/docker-compose.yml, wait five minutes, check Portainer, and find the old version still running.
Our first assumption was a configuration problem on our end. We’d recently been doing a lot of work on port bindings and secrets, so it seemed plausible we’d broken something. We worked through the obvious candidates:
- Port bindings were wrong — we’d been binding to
127.0.0.1instead of0.0.0.0, which meant Caddy couldn’t reach the backends. Fixed. But that was a proxy routing issue, not a GitOps issue. - The GitHub PAT might have expired or been scoped incorrectly. We rotated it, confirmed the scopes, updated the Portainer stack configuration. Still no auto-deploy.
- We tested the Portainer API directly — manually triggering a redeploy via
POST /api/stacks/<id>/git/redeployworked fine. So Portainer could pull from GitHub when explicitly told to. It just wasn’t doing it automatically.
That distinction was the tell. The polling mechanism itself was broken, not the authentication or the repository access.
Confirmed Upstream Bugs
A search through the Portainer GitHub issues confirmed what we were seeing wasn’t isolated. Three separate issues describe the same behavior in Portainer CE:
- #12974 — GitOps auto-update stops polling after certain stack operations
- #12906 — PAT authentication fails silently for auto-update even when manual redeploy works
- #10235 — GitOps polling state gets corrupted and requires stack re-registration to recover
All three are open against Portainer CE. The common thread in the comments: this works more reliably in Portainer Business Edition. The CE fix timeline is unclear.
We were hitting a confirmed upstream defect with no available workaround beyond re-registering stacks periodically. That’s not a foundation we wanted to build on.
The Architectural Decision
The question became: fix Portainer, or replace it?
Fixing Portainer meant either waiting for upstream or switching to Business Edition — neither of which addressed the underlying issue. And stepping back, the GitOps model had a structural problem we’d glossed over: it was implicit. A push to main would silently trigger a deploy on docker-host. We’d added a comment in a compose file and triggered a container restart without meaning to. The “automatic” part was a liability as much as a convenience.
Explicit is better than implicit, especially for infrastructure. We already had Ansible. We already had the compose files in stacks/. The missing piece was a role that would docker compose up -d them on docker-host — something we could run deliberately, inspect the output of, and commit as a deploy event in git history.
We built the ansible/roles/docker-stacks role.
The docker-stacks Role
The role handles two categories of stacks:
Image-based stacks (docker_image_stacks) pull from a registry — Docker Hub, GHCR, or a private registry. The role ensures the compose file is present on docker-host, pulls the latest images, and runs docker compose up -d. This covers monitoring, openwebui, and the juntgencom site.
Build-based stacks (docker_build_stacks) have a local Dockerfile. The role clones or syncs the source, runs docker compose build, and then docker compose up -d. This covers clearbenefit, which builds from our own source.
Both categories are defined in ansible/roles/docker-stacks/defaults/main.yml. Adding a new stack means adding an entry there and committing — the role is idempotent, so running it again on a stack that’s already current is a no-op.
Secrets are handled cleanly: Ansible Vault values are written to EnvironmentFiles on docker-host at deploy time, and compose files reference them via env_file. Nothing sensitive touches the repo.
The deploy command is:
cd ~/homelab/ansible
ansible-playbook playbooks/docker-stacks.ymlThat’s it. Every container update, every config change, every new stack — one command, full output, clear history.
Removing Portainer
With all four stacks (monitoring, openwebui, juntgencom, clearbenefit) running cleanly under Ansible management, we removed Portainer:
ssh docker-host 'docker stop portainer && docker rm portainer'
ssh docker-host 'docker volume rm portainer_data'The stacks/portainer/docker-compose.yml file moved to stacks/retired/portainer/. We removed the portainer.example.com block from the Caddyfile and ran ansible-playbook playbooks/caddy.yml. The subdomain is gone.
Change Control Model
The new model is review → commit → deploy:
- Edit
stacks/<name>/docker-compose.ymllocally - Review the diff — what’s actually changing
- Commit and push to GitHub
- Run
ansible-playbook playbooks/docker-stacks.yml - Verify with
ssh docker-host 'docker ps'
Step 4 is intentional. We decide when docker-host changes. There’s no background process that might apply an accidental commit at 3 AM. The git log is the deploy log.
We also added docker-stacks.yml to site.yml, so a full environment redeploy — the disaster recovery scenario — now includes Docker stacks in the correct dependency order, after Caddy is configured.
Cleaning Up
Two vault secrets became dead weight: vault_portainer_api_key and vault_github_portainer_pat. We removed them from ansible/inventory/group_vars/all/vault.yml directly via ansible-vault edit. They were the only credentials scoped exclusively to Portainer, so their removal also reduces the blast radius of any future credential exposure.
Lessons Learned
Trust upstream issue trackers. We spent time troubleshooting what turned out to be a known, open defect. Searching the issue tracker earlier would have saved an hour.
Automatic is not always better. The GitOps polling model felt like progress — less to do manually. But implicit deploys create invisible coupling between git history and infrastructure state. Explicit Ansible runs make that coupling visible and controllable.
Portainer was solving a problem we didn’t have. The UI was useful early on when we were learning what containers were doing. Now that stacks are stable and well-understood, docker ps and docker logs are sufficient. The orchestration layer was adding complexity without adding value.
Idempotency is the right primitive. Whether we’re running docker-stacks.yml for the first time or the fiftieth, the result is the same. That’s the property we want from infrastructure tooling — not “deploy if changed” magic, but “ensure this state is true.”
The homelab is simpler now. Three layers of IaC, all explicit, all auditable, all in git.