djuntgen@juntgen.com
← all posts

Building a Homelab with AI · part 20

Consistent Time Sync Across All Proxmox Hosts with Chrony

#homelab#ansible#ntp#chrony#proxmox#lxc

The Discovery

During the PBS deployment, we ran a quick sanity check on the fresh Debian 12 LXC:

$ timedatectl
               Local time: Tue 2026-03-10 21:14:03 CST
           Universal time: Wed 2026-03-11 03:14:03 UTC
                 RTC time: Wed 2026-03-11 03:14:03
                Time zone: America/Chicago (CST, -0600)
 System clock synchronized: no
               NTP service: active
           RTC in local time: no

System clock synchronized: no. The NTP service was running, but chrony was not actually synchronizing with anything. We dug in:

$ chronyc tracking
Reference ID    : 00000000 ()
Stratum         : 0
Ref time (UTC)  : Thu Jan 01 00:00:00 1970
System time     : 0.000000000 seconds slow of NTP time
Last offset     : +0.000000000 seconds
RMS offset      : 0.000000000 seconds
Frequency       : 0.000 ppm
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 1.000000000 seconds
Root dispersion : 1.000000000 seconds
Leap status     : Not synchronised

Reference ID 00000000. Ref time: January 1, 1970. Chrony was up, not synchronized to anything, and had never been.

$ chronyc sources
MS Name/IP address    Stratum Poll Reach LastRx Last sample
====================================================================
^? nts.netnod.se            0   6     0     -     +0ns[   +0ns] +/-    0ns
^? time.cloudflare.com      0   6     0     -     +0ns[   +0ns] +/-    0ns

Reach: 0 on every source. Not a single NTP packet had been received.

Root Cause: NTS Inside LXC Containers

The default chrony configuration on Ubuntu and Debian (since approximately 2022) uses NTS — Network Time Security. NTS is authenticated NTP: it uses TLS over TCP port 4460 to bootstrap trust, then exchanges time over UDP 123.

The problem is the TCP 4460 component. Proxmox LXC containers run in a constrained network namespace. The default Proxmox LXC configuration does not allow TCP 4460 through the container network stack in a way that works with NTS’s TLS handshake requirements. The containers can route UDP 123 fine. The NTS bootstrap over TCP 4460 silently fails. Chrony logs no errors — it just never receives any responses from the NTS servers, and Reach stays at 0 forever.

The config responsible for this is the Ubuntu/Debian default:

# /etc/chrony/sources.d/ubuntu-ntp-pools.sources (the problem)
pool ntp.ubuntu.com        iburst maxsources 4 nts
pool 0.ubuntu.pool.ntp.org iburst maxsources 1 nts
pool 1.ubuntu.pool.ntp.org iburst maxsources 1 nts
pool 2.ubuntu.pool.ntp.org iburst maxsources 2 nts

The nts keyword at the end of each pool line tells chrony to use NTS authentication. Remove it, and chrony falls back to standard UDP NTP, which works fine.

Why Chrony Over ntpd

We use chrony across the fleet rather than the classic ntpd daemon for two reasons:

Suspend/resume handling. Proxmox VMs and LXCs can be paused, migrated, or have their clocks jump when the host system is under load. ntpd panics on large clock offsets — it has a hardcoded limit (typically 1000 seconds) beyond which it refuses to step the clock and exits. Chrony handles arbitrarily large offsets gracefully by stepping the clock on startup and slewing from there.

Fast initial convergence. With iburst, chrony sends a burst of 8 packets immediately on start, converging to synchronized time in 2–4 seconds instead of ntpd’s typical 5-minute wait through the initial polling interval ladder.

The Fix

Two tasks in the base role, ordered carefully:

- name: Disable default NTS sources if present
  ansible.builtin.file:
    path: /etc/chrony/sources.d/ubuntu-ntp-pools.sources
    state: absent
  notify: Restart chrony
  failed_when: false

- name: Deploy chrony NTP sources (standard pools, no NTS)
  ansible.builtin.template:
    src: chrony-ntp-pools.sources.j2
    dest: /etc/chrony/sources.d/ntp-pools.sources
    owner: root
    group: root
    mode: "0644"
  notify: Restart chrony
  failed_when: false

The template is deliberately minimal:

# /etc/chrony/sources.d/ntp-pools.sources — managed by Ansible
# Uses standard NTP (UDP 123) — NTS (TCP 4460) avoided for LXC compatibility

pool 0.pool.ntp.org iburst maxsources 2
pool 1.pool.ntp.org iburst maxsources 2
pool 2.pool.ntp.org iburst maxsources 2
pool 3.pool.ntp.org iburst maxsources 2

Four pools, two sources each, iburst on all of them. No nts keyword. This gives us up to 8 simultaneous NTP sources, 2 from each pool subdomain, which provides clock quality comparable to a well-configured enterprise NTP setup.

We also added chrony to common_packages in group_vars/all/vars.yml to ensure it is installed on every managed host:

common_packages:
  - curl
  - wget
  - htop
  - vim
  - tmux
  - git
  - unzip
  - sudo
  - chrony

Task Ordering Matters

The remove-then-deploy ordering is deliberate. If we deployed our config first and removed the NTS config second, there would be a window where both config files existed simultaneously in sources.d/. Chrony processes all .sources files in that directory. During that window, chrony would have NTS sources and our plain sources active, and might attempt NTS connections that fail before falling back to our pools.

Removing the NTS config first means there is never a moment with conflicting sources. The handler restarts chrony once, after both file operations complete, so the restart happens with only our clean config in place.

The failed_when: false on both tasks handles two edge cases:

  • The NTS sources file may not exist on hosts where chrony was freshly installed from our common_packages list (since chrony gets installed before the distro default sources are configured)
  • The sources.d/ directory may not exist on hosts without chrony installed yet

Deployment

We ran common.yml against the full fleet:

ansible-playbook playbooks/common.yml

Nine hosts reached: dev, pve, docker-host, caddy, n8n, db, influxdb, openclaw, pbs.

Three of them — caddy, docker-host, and pve — had the Ubuntu NTS sources file present and showed changed on the removal task. All three restarted chrony. The remaining six either had chrony freshly installed (and no NTS sources) or were already on our config from a previous run.

After the play completed, we spot-checked three hosts:

# caddy (DMZ VM — was NTS)
$ chronyc tracking | grep 'Reference ID\|Stratum\|synchronized'
Reference ID    : 83EF7380 (ntp1.snet.net)
Stratum         : 2
System clock synchronized: yes

# pbs (fresh LXC — was broken)
$ chronyc tracking | grep 'Reference ID\|Stratum\|synchronized'
Reference ID    : C0A87B01 (10.0.0.1)
Stratum         : 3
System clock synchronized: yes

# openclaw (VM — was working)
$ chronyc tracking | grep 'Reference ID\|Stratum\|synchronized'
Reference ID    : D8EF2300 (time.cloudflare.com)
Stratum         : 3
System clock synchronized: yes

All three synchronized. Reference IDs resolving to real NTP servers. Stratum 2 or 3, which is correct for a homelab without a local stratum 1 source.

Why This Matters

Time synchronization is invisible infrastructure. Until it breaks. The things that go wrong when clocks drift:

  • TLS certificate validation fails. Certificates have notBefore and notAfter fields. A clock 5 minutes ahead or behind can cause cert validation errors that look like cert issuance problems.
  • Log correlation becomes impossible. When correlating events across caddy, docker-host, and pbs, timestamps need to agree. A 30-second drift makes distributed log analysis useless.
  • PBS backup jobs show wrong timestamps. PBS records backup times in the datastore index. A drifted clock means the “most recent backup” is not actually the most recent.
  • OAuth and TOTP tokens become invalid. Time-based tokens have a tolerance window of typically 30 seconds. Drift beyond that and tokens start being rejected.

None of these failures have obvious error messages. “Your clock is wrong” is rarely the first thing you check.

Lessons Learned

1. NTS is a good idea that breaks in containers. Ubuntu and Debian defaulting to NTS is correct behavior for physical servers and VMs with full network access. For LXC containers on Proxmox, it silently fails. The failure mode — Reach: 0, System clock synchronized: no — does not mention NTS or TCP 4460 anywhere.

2. sources.d/ is a directory, not a single file. Chrony reads all .sources files in the directory. When deploying a replacement config, you must remove the old file — putting a new one next to it does not replace it.

3. failed_when: false is not sloppiness. We use it here specifically because the tasks need to be idempotent across hosts that may or may not have the file being removed. The alternative — checking if the file exists before attempting removal — is more lines of YAML for the same result.

4. Check time sync on every fresh container. Add it to the commissioning checklist. timedatectl takes two seconds and catches this class of problem before it causes something subtle.