We set out to give everyone in the house their own AI agent — one for me, one for my wife, one for each of the kids — self-hosted, private, and running on our own hardware on top of Hermes (Nous Research’s self-hosted personal-agent framework). What we actually did first was spend the better part of a day building an elaborate machine to manage agents that didn’t exist yet, get stuck, and then discover that the tool already did 90% of it for free.
This is the story of that detour, because the lesson is more useful than the architecture: don’t out-engineer the tool. And there’s a wrinkle that makes it a fitting first post for this series — the thing doing most of the over-engineering was the AI assistant driving the work. The human kept saying “keep it simple.” The human was right.
The vision
The goal hasn’t changed. Each family member gets a distinct agent with its own personality, its own chat bot, and — critically — its own memory, so the kids’ agents can’t read mine and mine can’t read my wife’s. Adults get access to a strong frontier model; kids stay on a free local model with tight budgets. Everything runs on a dedicated VM, behind our own reverse proxy, with no agent ever holding the keys to change anything dangerous (locks, the firewall, money) — those go through a separate, human-approved broker we’ll build later in the series.
Sensible. The trouble started with how we tried to build it.
How we over-built it
Hermes has one hard security truth, stated plainly in its own SECURITY.md: there is no
in-process containment against a prompt-injected agent. The operating-system user is the only
real boundary. So we reasoned: each agent must run as its own locked-down OS user, so a
compromised agent can’t read another’s secrets off disk.
From that — correct — premise we built a tower:
- a custom Ansible role that created a system user per agent (
hermes-dave,hermes-jordan, …), - a shared Hermes install in
/opt, made world-readable so each user could execute it, - hand-written
systemdunits per agent, - hand-crafted
config.yamland.envtemplates for each profile, - and a whole 1Password → Ansible Vault secrets pipeline feeding it all.
Then we deployed it. And it didn’t work. The gateway came up healthy… and connected to nothing.
No Telegram. We went spelunking into Hermes’ source to find out why — the gateway builds its
messaging adapters from a config.platforms structure that our hand-written config never
populated correctly. We reverse-engineered the config loader, the JWT auth model, the platform
registry. Hours in, with the human telling us twice that this “seems too complicated,” the
realization finally landed:
We were rebuilding a feature Hermes ships out of the box.
The obvious answer
Hermes already does multi-agent. It’s called profiles, and the docs are blunt about it:
All profiles run as the same OS user, distinguished by the
-pflag and separate home directories under~/.hermes/profiles/<name>. No separate OS users needed.
Each profile gets its own config.yaml, .env, persona, memory, state database, bot token,
and systemd service — created with one command:
hermes profile create dave # makes the profile + a `dave` command alias
dave setup # interactive wizard: model + messaging platform
dave gateway install && dave gateway startEvery piece of machinery we’d hand-built — the per-OS-user isolation, the shared venv, the
systemd templates, the config files — Hermes either provides or makes unnecessary. The
profiles are isolated by home directory; the gateway is installed by gateway install; the
config is written by the setup wizard, correctly, the first time.
What did we “lose” by dropping our design? Cross-profile filesystem isolation — a prompt-injected agent could, in theory, read another profile’s token off disk. For read-only agents that hold nothing but their own rotatable bot token and model key, that’s a marginal risk, and the genuinely dangerous capabilities were always going to sit behind an external broker anyway. Not worth a tower of custom Ansible. We deleted the role.
Getting the first agent live — the simple way
With the over-engineering torn down, the first agent (mine) came up in a handful of commands on
the dedicated VM, as a single hermes user:
# install Hermes the supported way (their own installer)
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash
# create my profile and run the wizard
hermes profile create dave
dave setupTwo decisions in the wizard were worth getting right:
Model: subscription OAuth, not metered API. Hermes can log into xAI’s Grok via a browser OAuth flow against your SuperGrok subscription — no API key, flat monthly cost instead of per-token billing. For an agent you’ll hit all day, that’s the right primary. We set a local model as the fallback (our own LiteLLM proxy in front of an Ollama box), so if the cloud provider is down or rate-limited, the agent keeps working for free:
dave fallback # custom OpenAI-compatible endpoint -> local qwen-generalThe split we’ll carry across the family: adults use their own OAuth subscriptions; kids stay on the local model, where budgets and governance actually matter.
Memory: self-hosted Honcho, isolated per person. Honcho gives the
agent persistent, cross-session memory — it builds a model of you over time. We point each
profile at our self-hosted Honcho with a key scoped to that person’s workspace, so the
isolation we’d originally wanted from OS users we get for free at the memory layer instead: my
agent’s key only reaches the dave workspace; it cannot read my wife’s.
A short while later, a “hey” to the Telegram bot came back with a friendly reply — over the local model, with memory accumulating in the background. First agent: live.
Lessons
- Don’t out-engineer the tool. If a framework ships an installer, a
setupwizard, and a multi-agent feature, use them. Hand-crafting their config and reverse-engineering their internals is a smell, not diligence. - Test the real thing early. We analyzed logs and source for hours before doing the one cheap test that mattered: send the bot a message. Lead with the end-to-end check.
- Match the control model to the risk. The OS-user isolation we agonized over wasn’t worth it for read-only agents. Save the heavy controls for the parts that can actually do damage — which, for us, is a separate human-approved broker, not the agents themselves.
- “Keep it simple” is a hard constraint, not a vibe. When the person you’re working with says it twice, stop optimizing and step back.
What’s next
The agents can read and talk; they can’t yet do anything dangerous, by design. Next in the series: the Executor — an approval-gated broker that lets agents propose risky actions (home-automation, spending, network changes) while a human signs off out-of-band — followed by bringing the kids online through a private household chat server. One agent down; a household to go.