djuntgen@juntgen.com
← all posts

Building a Homelab with AI · part 11

Anti-Patterns and War Stories: Sudoers Mistakes That Break Things at 3 AM

#homelab#linux#security#sudoers#anti-patterns#best-practices#hardening

This is Part 11 (final) of the Building a Homelab with AI series. Previously: Automating Sudoers with Ansible

Why Anti-Patterns Matter

The previous three posts told you what to do. This one tells you what not to do — and more importantly, why. Every anti-pattern here is something I’ve either done myself, fixed on someone else’s system, or narrowly avoided. They’re the kind of mistakes that work fine for months and then fail spectacularly when you least expect it.

Anti-Pattern 1: Editing /etc/sudoers Directly

The mistake: Opening /etc/sudoers in vim and adding your rules inline.

Why it’s wrong:

  1. Package updates can overwrite /etc/sudoers. When sudo is upgraded, the package manager may replace the main file with the upstream default. Your custom rules vanish. Drop-in files in /etc/sudoers.d/ survive package upgrades.

  2. No visudo safety net. Editing with vim (or worse, nano, or worst, echo >>) skips syntax validation. A missing space, a typo, a stray character — and sudo stops working for everyone. On a headless server with no console access, this is game over.

  3. Merge conflicts. If a package manager detects the file has been modified, you’ll get prompted about keeping your version vs. the package’s version during upgrades. With drop-in files, there’s nothing to conflict.

The fix: Always use /etc/sudoers.d/ for custom rules. One file per user or role. Use visudo -f /etc/sudoers.d/username to edit with syntax validation.

Anti-Pattern 2: chmod 777

The mistake: “The permissions are wrong? Just chmod 777 it.”

Why it’s wrong: 777 means every user on the system can read, write, and execute the file. For configuration files, this means any compromised service can modify your configs. For credential files, it means any process can read your secrets. For directories, it means anyone can create or delete files.

I’ve seen chmod 777 on:

  • Docker socket files (root-equivalent access to every user)
  • Web server document roots (allows any process to inject malicious pages)
  • Cron directories (allows any user to schedule jobs as root)
  • SSL certificate directories (allows any process to steal TLS private keys)

The fix: Figure out the actual required permissions. Ask: “Who needs to read this? Who needs to write it? Does anyone need to execute it?” Set the minimum permissions that answer those questions. If you’re stuck, 0644 for files and 0755 for directories is almost always a safe starting point for non-sensitive content.

# Instead of:
chmod 777 /opt/myapp/config.yml      # NO

# Think about what's needed:
chmod 644 /opt/myapp/config.yml      # Owner read/write, everyone else read-only
chown root:root /opt/myapp/config.yml # Root owns it

Anti-Pattern 3: Sudoers Files Without Trailing Newlines

The mistake: Writing a sudoers rule without a newline at the end.

# Missing newline -- subtle but dangerous
echo -n "<user> ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/<user>

Why it’s wrong: Some versions of sudoers parsers require a trailing newline. Without it, the rule might be silently ignored, or worse, it might work on one OS version and not another. When you upgrade the OS and sudo stops working, you’ll be debugging for hours.

The fix: Always include a trailing newline. In Ansible:

content: "{{ admin_user }} ALL=(ALL) NOPASSWD: ALL\n"
#                                                 ^^
#                                            This matters

In shell:

echo "<user> ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/<user>
# echo adds a newline by default. printf does not unless you include \n.

Anti-Pattern 4: Dots in Drop-in Filenames

The mistake: Naming a sudoers drop-in file with a dot or tilde.

# These are ALL silently ignored:
/etc/sudoers.d/<user>.conf      # dot in filename
/etc/sudoers.d/<user>.sudoers   # dot in filename
/etc/sudoers.d/<user>~          # backup file suffix
/etc/sudoers.d/.<user>          # hidden file (starts with dot)

Why it’s wrong: The @includedir directive skips files containing . or ending in ~. This is by design — it prevents editor backup files (sudoers~, sudoers.bak) from being parsed. But it also means well-intentioned filenames like <user>.conf are silently ignored. No error, no warning. Your user just doesn’t have sudo access and you have no idea why.

The fix: Name drop-in files with no extension and no dots. Match the username or role name:

/etc/sudoers.d/<user>           # correct
/etc/sudoers.d/monitoring      # correct
/etc/sudoers.d/ansible-deploy  # correct (hyphens are fine)

Anti-Pattern 5: Running a Root Shell Instead of Using sudo

The mistake: Instead of prefixing commands with sudo, running sudo -s or sudo su - and working in a root shell for an extended session.

# The lazy approach
$ sudo -s
# root@docker-host:~# apt update
# root@docker-host:~# systemctl restart caddy
# root@docker-host:~# vim /etc/ssh/sshd_config
# root@docker-host:~# exit

Why it’s wrong:

  1. No audit trail. When you use sudo apt update, the auth log records exactly which command was elevated. When you use sudo -s, the log shows ” started a root shell” and nothing else. If something breaks, you can’t trace which command caused it.

  2. Blast radius. A typo in a root shell affects the whole system. rm -rf /opt /data (note the space) in a root shell deletes both /opt and /data. With sudo rm -rf /opt/data, you at least have the sudo prompt as a speed bump, and the command is logged.

  3. Habit formation. Once you’re comfortable in a root shell, you start doing everything there. Configuration edits, package management, service restarts — all without the discipline of sudo forcing you to think “do I really need root for this?”

The fix: Use sudo per-command. With NOPASSWD configured, there’s no friction difference. You’re just adding sudo to the front of the command, and you gain a full audit log in return.

# The right approach -- each command is logged individually
$ sudo apt update
$ sudo systemctl restart caddy
$ sudo vim /etc/ssh/sshd_config  # Actually, use sudoedit (see below)

The sudoedit Exception

For editing files, sudoedit (or sudo -e) is better than sudo vim:

$ sudoedit /etc/ssh/sshd_config

sudoedit copies the file to a temp location, opens the editor as your user (not root), and copies the modified file back. This means:

  • Your editor plugins and configuration are used (not root’s)
  • Editor vulnerabilities don’t run as root
  • The file is still owned by root after editing

Anti-Pattern 6: NOPASSWD for Multi-User Systems

The mistake: Applying NOPASSWD: ALL on a shared system where multiple people have sudo access.

Why it’s wrong: In a single-operator homelab, NOPASSWD is fine — you’re the only person with access. On a shared system (a work server, a family NAS with multiple admin accounts, a dev environment used by a team), NOPASSWD means anyone who compromises any sudo-capable account has instant, silent root access. The password prompt isn’t security theater on multi-user systems — it’s a real gate.

The fix: On multi-user systems, use password-requiring sudo with a reasonable timeout:

Defaults    timestamp_timeout=15
<user>       ALL=(ALL) ALL

This caches the password for 15 minutes. You enter it once per session, not every command. On single-user homelabs? NOPASSWD is fine. Know your threat model.

Anti-Pattern 7: Ignoring sudo Logging

The mistake: Having sudo configured but never looking at the logs.

Why it’s wrong: Sudo logs every invocation to syslog. On Debian/Ubuntu, this goes to /var/log/auth.log. If you’re not monitoring these logs, you’re missing:

  • Failed sudo attempts (someone guessing passwords, or a misconfigured script)
  • Unexpected sudo usage (a compromised process trying to escalate)
  • Successful commands (useful for forensics after an incident)
# What sudo logging looks like:
$ sudo grep sudo /var/log/auth.log | tail -5
Feb 15 18:43:22 dev sudo:    <user> : TTY=pts/0 ; PWD=/home/<user> ; USER=root ; COMMAND=/usr/bin/apt update
Feb 15 18:43:45 dev sudo:    <user> : TTY=pts/0 ; PWD=/home/<user> ; USER=root ; COMMAND=/usr/bin/systemctl status caddy

The fix: At minimum, set up log rotation so auth.log doesn’t fill your disk. Ideally, ship logs to a central aggregator (Loki, Elasticsearch, even a simple syslog server). When you build out monitoring (it’s on my homelab backlog), add alerts for failed sudo attempts.

For now, even a simple cron job helps:

# /etc/cron.daily/sudo-audit
#!/bin/bash
# Report failed sudo attempts from the last 24 hours
grep "sudo.*FAILED" /var/log/auth.log | tail -20

Anti-Pattern 8: Not Having a Break-Glass Procedure

The mistake: Your only path to root is sudo, and you haven’t tested what happens when sudo breaks.

Why it’s wrong: Sudoers syntax errors, corrupted files, or accidental permission changes can lock you out of root entirely. If your only access is SSH with key auth and sudo, and sudo is broken, you’re locked out. On a cloud instance, this might mean rebuilding from scratch. On a homelab, it means walking to the server room (or Proxmox console).

The fix: Document and test your recovery path. In my homelab:

  1. Proxmox pct exec: From the Proxmox host, run pct exec <vmid> -- bash to get a root shell inside any LXC container. No SSH, no sudo. This is my primary break-glass path.

  2. Proxmox console: For the Proxmox host itself, physical console access or IPMI/iDRAC if available.

  3. Recovery boot: For VMs and bare metal, boot from a rescue ISO, mount the root filesystem, and fix /etc/sudoers.d/.

Test this before you need it. Right now, while everything works, try:

# From the Proxmox host -- verify pct exec works
$ pct exec 105 -- whoami
root

# Good. Now you know your escape hatch is functional.

Anti-Pattern 9: Granting sudo to Service Accounts Without Restrictions

The mistake: A monitoring agent, backup script, or CI/CD runner needs one specific root command, so you give it ALL=(ALL) NOPASSWD: ALL.

Why it’s wrong: Principle of least privilege. If that service account is compromised (and services get compromised more often than interactive accounts), the attacker now has unrestricted root. If the account only had NOPASSWD: /usr/bin/systemctl status *, the blast radius is limited to reading service status — not installing packages, modifying configs, or pivoting to other hosts.

The fix: Enumerate exactly what commands the service needs and grant only those:

# /etc/sudoers.d/monitoring
monitoring ALL=(ALL) NOPASSWD: /usr/bin/systemctl status *, \
                               /usr/bin/journalctl --no-pager -u *, \
                               /usr/bin/docker ps

# /etc/sudoers.d/backup
backup ALL=(ALL) NOPASSWD: /usr/bin/rsync --server *, \
                           /usr/sbin/zfs list, \
                           /usr/sbin/zfs snapshot *

Yes, this is more work upfront. It’s dramatically less work than recovering from a compromised service account with full root access.

Anti-Pattern 10: Storing sudo Passwords in Scripts

The mistake: Embedding passwords in automation scripts.

# DO NOT DO THIS
echo "mypassword" | sudo -S apt update

Why it’s wrong: The password is visible in the process table (ps aux), in shell history, in the script file itself, and in any logs that capture command output. It’s plaintext secrets scattered across the system.

The fix: Use NOPASSWD for automated tasks. This is actually more secure than embedding passwords, because:

  • No secrets to leak
  • The sudoers rule explicitly documents what the automation can do
  • The access control is in one place (/etc/sudoers.d/) instead of scattered across scripts

For interactive use where you want a password, just type it when prompted. For automation, NOPASSWD with a restricted command set.

The Hardening Checklist

Here’s the consolidated checklist from this entire series. Run through it for every host in your homelab:

Sudoers

  • Custom rules in /etc/sudoers.d/, not in /etc/sudoers
  • Drop-in files have mode 0440, owned by root:root
  • Drop-in filenames have no dots, no tildes
  • All rules end with a newline
  • visudo -c passes with no errors
  • Service accounts have restricted command sets, not ALL

SSH

  • ~/.ssh/ is mode 0700, owned by the user
  • ~/.ssh/authorized_keys is mode 0600, owned by the user
  • PermitRootLogin no in sshd_config
  • PasswordAuthentication no in sshd_config
  • Only ed25519 or RSA 4096+ keys in use

File Permissions

  • No world-writable files in /etc/
  • Credential files (.env, vault passwords, API keys) are mode 0600
  • No unexpected setuid binaries (find / -perm -4000)
  • Docker socket is root:docker with tight group membership

Automation

  • Sudoers deployed via Ansible with visudo -cf validation
  • Vault password file is in .gitignore
  • No plaintext secrets in the repository
  • Break-glass recovery procedure documented and tested

Monitoring

  • Auth logs are being written and rotated
  • Failed sudo attempts are detectable
  • SSH login events are logged

Closing Thoughts

Security hardening is never done. It’s a practice, not a destination. But the difference between a hardened homelab and an “it works, don’t touch it” homelab is usually a few hours of deliberate configuration — and the discipline to codify those configurations so they’re consistent, auditable, and reproducible.

The tools are all free. Ansible, visudo, drop-in sudoers files, SSH key authentication, file permissions — none of this requires buying anything or running complex software. It requires attention and intention.

If you’ve read through this series and found even one thing to fix on your systems, it was worth writing. And if you’re just starting your homelab journey, starting with these fundamentals will save you from the 3 AM debugging sessions that taught me these lessons the hard way.

Your homelab is your infrastructure. Treat it like it matters. Because it does.

This is the final post in the Building a Homelab with AI series. The entire series — from IaC setup through security hardening — was pair-programmed with Claude Code. Every commit tells the story.