djuntgen@juntgen.com
← all posts

Building a Homelab with AI · part 27

Part 27: A Self-Hosted OpenRouter for the Family — Deploying LiteLLM

#homelab#ansible#docker#litellm#ollama#caddy#postgres#ai

The plan started simple: install LiteLLM as a self-hosted “OpenRouter” so Hermes on the new VM has a single, controllable endpoint to talk to instead of being hardcoded against inference-host’s Ollama. One Docker stack on docker-host, same pattern as openwebui and paperclip. Easy.

The plan got more interesting almost immediately, in a useful way.

The pivot: one tenant vs many

Our first sketch was config-only: a stateless LiteLLM proxy with a config.yaml listing the models and a single master key. No database, no admin UI, nothing to back up. For one consumer (our own Hermes), it’s the right shape — the UI exists to manage per-tenant virtual keys, budgets, and per-key model allow-lists, and with one tenant there’s nothing to manage. The model list belongs in git anyway, so the database wouldn’t be carrying its weight.

Then we said the part that changes everything:

“I also want to have Hermes Agents for my kids and wife.”

That’s a different system. “Per-person Hermes, per-person key” means we now have several tenants with different trust levels (you vs. a curious kid), and the things you’d want — revoke a key without touching anyone else, restrict a kid’s key to local Ollama only, set a hard spend cap on a cloud-API key — only exist in DB-backed mode. The UI just comes along as the way to manage them.

The deciding factor was never “do I want a UI?” — it was “how many tenants do I have, and do I need to control them independently?” When the answer changed, the recommendation changed.

What stayed put: the model list still lives in git, in stacks/litellm/config.yaml, so adding or removing models is an auditable PR. Only the things that can’t be git-managed — runtime-issued credentials, per-key budgets, usage counters — live in Postgres.

# stacks/litellm/config.yaml
model_list:
  - model_name: qwen-general
    litellm_params:
      model: ollama_chat/qwen3.6:35b-mlx
      api_base: http://10.0.0.232:11434

  - model_name: qwen-coder
    litellm_params:
      model: ollama_chat/qwen3.5:35b-a3b-coding-nvfp4
      api_base: http://10.0.0.232:11434

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true

The ollama_chat/ prefix matters — plain ollama/ would hit Ollama’s completions endpoint and clients would see weird raw-text behavior. ollama_chat/ routes through chat-completions with proper role formatting, which is what every consumer (Hermes, OpenWebUI, LiteLLM’s own UI) actually wants.

Verify before you depend: the postgres LXC was a ghost

We were going to point LiteLLM at the existing postgres LXC (10.0.0.21). The container was in the inventory, listed as running in pct list, and our memory said “PostgreSQL 17 installed by the postgres role.” Nothing in the inventory contradicted that.

We checked anyway, via pct exec (since the LXC isn’t bootstrapped for Ansible SSH yet):

$ ansible pve -m shell -a "pct exec 107 -- bash -lc \
    'systemctl is-active postgresql 2>/dev/null; psql --version 2>/dev/null || echo NO-PSQL'"
inactive
NO-PSQL

Empty shell. The role files exist locally but they’re untracked — the LXC is waiting for that work to land. Existence isn’t availability. If we’d written the spec around the LXC, the deploy would have failed at the first Prisma query and we’d have spent half an hour confused about why a “running” Postgres wasn’t reachable.

So we pivoted: bundle Postgres inside the LiteLLM stack. That’s actually LiteLLM’s own recommended self-host shape — their official docker-compose.yml ships a postgres:16 alongside the proxy. The stack stays self-contained, the litellm DB volume gets picked up by docker-host’s existing restic→PBS backup job, and the day the postgres LXC is finished, switching is a one-line DATABASE_URL change.

The stack: six files, all under existing patterns

Nothing exotic — every file mirrors something already in the repo:

stacks/litellm/
  docker-compose.yml          # litellm-database:main-stable + postgres:16
  config.yaml                 # model_list (git source of truth)

ansible/roles/docker-stacks/
  templates/litellm.env.j2    # master + salt + DB password from vault
  defaults/main.yml           # +1 entry: docker_image_stacks now lists litellm
  tasks/secrets.yml           # renders /etc/litellm/litellm.env (0600)

ansible/roles/caddy/templates/Caddyfile.j2
                              # +1 block: litellm.juntgen.com → 10.0.0.12:4000 (local_only)

A few small adaptations from LiteLLM’s reference compose:

  • No bundled Prometheus. Their reference deploy ships one; we already run VictoriaMetrics/Grafana on docker-host. We’ll point that at LiteLLM’s /metrics later instead of standing up a second Prometheus.
  • Postgres is internal-only. No 5432 host port. Only 4000 is published.
  • Caddy gives us local_only enforcement at the perimeter, so even though 10.0.0.12:4000 is reachable on the LAN, the hostname litellm.juntgen.com returns 403 for any non-RFC1918 source.

Deploy is the same two commands every other stack uses:

cd ansible
ansible-playbook playbooks/docker-stacks.yml   # the stack
ansible-playbook playbooks/caddy.yml           # the route

The bug: a base64 password and a URL parser

First deploy: the containers came up, postgres reported healthy, but LiteLLM stayed unhealthy and the proxy logs cycled this:

{"is_panic":false,"message":"The provided database string is invalid.
 Error parsing connection string: invalid port number in database URL.
 ... please refer to Prisma docs ...","error_code":"P1013"}

Invalid port number? The URL was postgresql://llmproxy:<password>@db:5432/litellm — 5432 is right there. Except: we’d generated the password with openssl rand -base64 24, and base64’s alphabet includes + / =. In URI userinfo grammar, those three characters are reserved — when Prisma’s parser hit the first / inside the password, it terminated the userinfo, thought the rest of the password was the host, and the whole subsequent structure shifted, including the port.

Postgres itself didn’t care — its POSTGRES_PASSWORD env var is a raw string, not a URI. Only the URL form of the same value was wrong.

The fix is surgical: encode only where the URI grammar demands it.

# templates/litellm.env.j2 — diff

-DATABASE_URL=postgresql://llmproxy:{{ vault_litellm_db_password }}@db:5432/litellm
+DATABASE_URL=postgresql://llmproxy:{{ vault_litellm_db_password
+   | replace('+', '%2B')
+   | replace('/', '%2F')
+   | replace('=', '%3D') }}@db:5432/litellm

 POSTGRES_PASSWORD={{ vault_litellm_db_password }}    # untouched — raw value

POSTGRES_PASSWORD stays raw because the Postgres container reads it as a plain string. Both sides authenticate the same identity; only the transport differs.

Lesson: when a value crosses a protocol boundary into a context with a grammar (URIs, JSON, shell), encode at the boundary — don’t change the generator. openssl rand -hex would have sidestepped this, but the encoding fix is general and protects every future regeneration regardless of how the password is produced.

After the fix, the redeploy log was friendlier:

litellm | Running prisma migrate deploy
litellm | LiteLLM: Proxy initialized with Config, Set models:
litellm |     qwen-general
litellm |     qwen-coder

Both models picked up. curl /health/liveliness → 200. curl /v1/models with the master key → ["qwen-general", "qwen-coder"]. Same call without the key → 401. The Caddy hostname returns the same 200 over HTTPS via the existing Cloudflare DNS-01 ACME cert. Everything wired.

The Qwen reasoning trap

End-to-end test, smallest possible prompt:

curl ... -d '{"model":"qwen-general",
              "messages":[{"role":"user","content":"Reply with the single word OK."}],
              "max_tokens": 16}'

The response came back with finish_reason: length, completion_tokens: 16, and content: "". The chain worked — tokens were produced, accounted for, returned through the proxy — but content was empty. Tucked into the response was the giveaway:

"reasoning_content": "Thinking Process:\n\n1.  **Analyze the User's Request:** The"

qwen3.6:35b-mlx is a reasoning model. It emits its chain-of-thought into reasoning_content before producing any user-facing text. With max_tokens: 16 the model burned all sixteen tokens on internal reasoning and never reached the answer. Bumping to 300 didn’t help — it just wrote three hundred tokens of reasoning. Even Qwen’s /no_think directive in the prompt didn’t dissuade this particular MLX build.

That’s a model-tuning thing, not an infra thing — qwen-coder (the NVFP4 build) may behave differently, and a non-reasoning model would behave normally. The point for this part of the journey is that the plumbing is right: token usage is reported, both content and reasoning_content fields surface correctly, and the consumer just needs to know to either size the budget for thinking or switch to a non-thinking model.

What’s deployed, what’s next

After today, the homelab has:

  • litellm and litellm-db containers running on docker-host under restart policies and healthchecks
  • HTTPS endpoint at https://litellm.juntgen.com (LAN-only via the existing local_only Caddy snippet)
  • Model list in git (stacks/litellm/config.yaml), deployable through the same docker-stacks Ansible pipeline every other service uses
  • Three new vault keys (vault_litellm_master_key, vault_litellm_salt_key, vault_litellm_db_password), reachable nowhere outside the host’s /etc/litellm/litellm.env (mode 0600)
  • Volume litellm-db covered by the existing restic→PBS backup job

What we deliberately didn’t do today:

  • Issue virtual keys yet. Those get created when each consumer arrives — one for OpenWebUI, one each for the family Hermes agents — via LiteLLM’s admin UI/API, with per-key model allow-lists (kids → local Ollama only) and budgets (cloud-API keys → hard caps).
  • Wire monitoring. LiteLLM exposes Prometheus metrics at /metrics; we’ll point our existing VictoriaMetrics scrape at it in the next round.
  • Add cloud providers. The config is structured so OpenRouter / Anthropic / OpenAI are a few-line addition — but with two solid local 35B models on inference-host, there’s no rush.

Lessons we’ll keep

  1. Tenant count drives architecture more than feature lists do. One tenant: config + master key. Several tenants with different trust levels: virtual keys + budgets + per-key model lists, which means a DB. The right answer for a single user is the wrong answer for a family.
  2. Verify before you depend. The postgres LXC was “running” in three different places (pct list, inventory, memory) and was still an empty shell. The five-line pct exec check turned a guaranteed deploy failure into a different design choice — before writing the code that would have depended on it.
  3. Encode at protocol boundaries, not in the generator. The base64 password was a fine secret; it was a broken URL. Fix it where it becomes a URL, leave it raw where it doesn’t.
  4. A working pipeline that returns empty content is still a working pipeline. Reasoning-model output that lands entirely in reasoning_content looks like a bug at first glance — but the token accounting and HTTP plumbing told us infrastructure was fine and the issue was model behavior. Read the whole response object before blaming the network.

Next up: the per-family Hermes agents themselves, with the LiteLLM keys we just gave them somewhere to land.