djuntgen@juntgen.com
← all posts

Building a Homelab with AI · part 13

Persistent Terminal Sessions on the Control Node with Zellij

#homelab#zellij#terminal#ssh#bash#dotfiles#nerd-fonts

This is Part 13 of the Building a Homelab with AI series. Previously: Sites, DNS, and MOTDs

The problem is familiar to anyone who manages a server over SSH: you are halfway through something, your connection drops, and whatever you were doing is gone. Or you close your laptop, come back to dev the next day, and have to reconstruct your context from scratch. For a laptop or a desktop, that is tolerable. For a dedicated control node that runs Ansible, manages stacks, and hosts Claude Code sessions, it is friction that compounds every single day.

The fix is a terminal multiplexer. We installed Zellij.

Why Zellij and Not tmux

tmux is the default answer in this space. It is installed everywhere, the documentation is deep, and it works. We have used it for years. But Zellij has earned its own consideration:

  • Session serialization built-in — Zellij can serialize the full state of a session to disk, including pane layout, working directories, and scrollback history. After a reboot, you get your session back. tmux has plugins that approximate this (tmux-resurrect, tmux-continuum), but they are add-ons with their own failure modes.
  • Modern configuration format — Zellij uses KDL (a structured document language) rather than the tmux config syntax, which has accumulated decades of quirks.
  • Better defaults — Zellij’s default keybindings and UI work well without extensive configuration. The status bar shows available modes and keybindings, which matters when you are learning.
  • Actively developed — The release cadence is fast and the developer community is responsive.

The trade-off is that Zellij is less ubiquitous. It will not be installed on every machine you touch. For a dedicated control node you own and manage, that is not a concern.

Installation

The apt version of Zellij lags behind current releases. We install directly from the official GitHub release:

ZELLIJ_VERSION="v0.43.1"
curl -fsSL "https://github.com/zellij-org/zellij/releases/download/${ZELLIJ_VERSION}/zellij-x86_64-unknown-linux-musl.tar.gz" \
  -o /tmp/zellij.tar.gz
tar -xzf /tmp/zellij.tar.gz -C /tmp
sudo install -m 755 /tmp/zellij /usr/local/bin/zellij
rm /tmp/zellij.tar.gz /tmp/zellij

The musl build is statically linked — no shared library dependencies. It runs cleanly on Ubuntu 24.04 without anything extra.

Configuration

Generate the default config as a starting point:

mkdir -p ~/.config/zellij
zellij setup --dump-config > ~/.config/zellij/config.kdl

The defaults are reasonable, but a few settings matter a lot for the persistent session use case. Here is what we set and why:

// Detach instead of quit when terminal window is closed.
// Without this, killing your SSH connection kills the session.
on_force_close "detach"

// Serialize session state to disk on a regular interval.
// Tabs, panes, working directories, and running commands are all captured.
session_serialization true

// Also serialize scrollback so pane history survives a reboot.
serialize_pane_viewport true
scrollback_lines_to_serialize 10000

// Compact bar uses one line instead of two. Saves vertical space.
default_layout "compact"

// No need for release notes on every upgrade.
show_release_notes false

The key one is on_force_close "detach". The default is already detach, but it is worth making it explicit. When your SSH client closes — intentionally or because your network dropped — the Zellij session keeps running on the server. The next time you connect, you attach to exactly where you left off.

session_serialization goes further. Even if the server reboots, Zellij will attempt to resurrect the session state when it next starts. Your pane layout and working directories come back. Running processes do not — there is no way to resume a process across a reboot — but the structural context does.

SSH Auto-Attach

Installing Zellij is not enough on its own. We want the persistent session to be automatic: SSH into dev and land directly in the main session, creating it if it does not exist.

This belongs in ~/.bashrc, at the very end:

# --- Zellij auto-attach ---
# On SSH login: attach to (or create) the persistent 'main' session.
# Guards:
#   $SSH_TTY        — only on interactive SSH with a real TTY (not scp/sftp/rsync)
#   $ZELLIJ         — don't nest if already inside a zellij session
#   $ZELLIJ_SKIP    — escape hatch: `ZELLIJ_SKIP=1 ssh dev` to get a raw shell
if [[ -n "$SSH_TTY" ]] && [[ -z "$ZELLIJ" ]] && [[ -z "$ZELLIJ_SKIP" ]]; then
    exec zellij attach --create main
fi

Each guard is deliberate:

$SSH_TTY is only set when SSH allocates a real pseudo-terminal — an interactive session where a human is on the other end. Ansible tasks, scp, sftp, and rsync connections do not allocate a TTY, so this variable is empty for all of them. The auto-attach does not fire for automated operations.

$ZELLIJ is set by Zellij for all shells running inside a session. Zellij panes spawn new bash instances, and those instances source ~/.bashrc. Without this guard, every new pane would try to attach to Zellij again, recursively. The guard prevents that.

$ZELLIJ_SKIP is an escape hatch. If you ever need a raw shell — to debug a ~/.bashrc issue, for instance — you can bypass the auto-attach without modifying any file: ZELLIJ_SKIP=1 ssh dev.

The exec keyword replaces the bash process with Zellij rather than running Zellij as a child process. This means when you detach from the session, the SSH connection closes cleanly. There is no bash prompt waiting underneath. The intended experience is: connect → session → detach → disconnect. Reconnect → same session.

The PATH Bug, and Why It Bit Us

The first attach worked. The second thing we tried — running claude from a Zellij pane — did not:

$ claude
claude: command not found

This is a bash shell initialization subtlety that trips people up constantly.

On Ubuntu, ~/.bashrc is sourced for interactive non-login shells. ~/.profile is sourced for login shells. When you SSH into a machine, bash starts as a login shell, sources ~/.profile, and ~/.profile sources ~/.bashrc. So far so good.

But Zellij spawns new pane shells differently. It execs $SHELL directly — not as a login shell. The new pane gets ~/.bashrc but never ~/.profile. And in the default Ubuntu setup, ~/.local/bin (where Claude Code installs) is added to $PATH in ~/.profile, not ~/.bashrc:

# In ~/.profile:
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

Zellij panes never see this. So ~/.local/bin is not in $PATH, and any binary installed there — including claude — is invisible.

The fix is to ensure ~/.local/bin is in $PATH for all interactive shells, regardless of login or non-login origin. We add this near the top of ~/.bashrc, right after the interactivity check:

# Ensure ~/.local/bin is in PATH for all interactive shells (login and non-login).
# ~/.profile handles login shells, but zellij panes are non-login — this covers both.
[[ -d "$HOME/.local/bin" ]] && [[ ":$PATH:" != *":$HOME/.local/bin:"* ]] && export PATH="$HOME/.local/bin:$PATH"

The double-colon pattern (":$PATH:" != *":$HOME/.local/bin:"*) is the idiomatic bash way to check whether a value is already present in a colon-separated list. It prevents the directory from being prepended twice on login shells that got it from ~/.profile first. Idempotent, safe, correct.

After adding this and sourcing ~/.bashrc in the current shell, claude was immediately available everywhere.

The lesson is worth internalizing: when you introduce a process supervisor that spawns new shells (Zellij, tmux, screen, Docker), those child shells get a different initialization path than your SSH login shell. Anything critical to your workflow needs to live in ~/.bashrc, not ~/.profile.

Nerd Fonts for the Tab Bar

Out of the box, Zellij’s tab bar showed ? Tab #1 ? instead of the styled separators. This is a font rendering issue, not a Zellij bug. Zellij uses special Unicode glyphs from the Nerd Fonts set for its UI elements. If your terminal emulator does not have a patched font configured, those glyphs render as question marks.

The fix is entirely on the client side — nothing on the server needs to change.

Install a Nerd Font on your Mac:

brew install --cask font-jetbrains-mono-nerd-font

Set it in iTerm2:

Settings → Profiles → Text → Font → select JetBrainsMono Nerd Font

A full iTerm2 restart may be needed to flush the font cache. After that, the Zellij tab bar renders correctly and the status bar mode indicators look as intended.

What We Have Now

The workflow is now:

  1. ssh dev — land directly in the main Zellij session
  2. Open panes and tabs freely — claude, Ansible, SSH to other hosts, log tailing
  3. Close the laptop — session keeps running on dev
  4. Come back — ssh dev — pick up exactly where you left off

The configuration and ~/.bashrc changes are committed to the homelab repo under dotfiles/, alongside the Zellij config. If we ever bootstrap a new control node, reproducing this setup is a single file copy and a binary install.

Part 13 of the Building a Homelab with AI series.