Erase your darlings: immutable infrastructure for mutable systems

The article outlines a method to create stateless NixOS systems by rendering the root filesystem ephemeral, ensuring each reboot starts with a pristine environment. This approach eliminates configuration drift and runtime cruft while retaining critical persistent data through strategic redirects. Key steps include:

1. Ephemeral Root Setup: The root filesystem is reset on every boot using ZFS snapshots (e.g., rolling back to an initial “blank” state). This ensures no residual runtime changes persist.

2. Persistent Storage: Critical state data (SSH keys, network configs, Bluetooth pairing info, etc.) is redirected to a dedicated persistent storage area (/persist). NixOS modules and systemd tmpfiles create symlinks from standard paths (e.g., /etc/ssh, /var/lib/bluetooth) to /persist.

3. Service Configuration: Services are configured to store data in the persistent directory. Examples include:

   • SSH host keys pointing to /persist/etc/ssh.

   • NetworkManager connections stored under /persist/etc/NetworkManager/system-connections.

   • ACME certificates persisted via symlinks to /persist/var/lib/acme.

4. Validation: Tools like zfs diff or filesystem listings (tree) help audit what data is temporarily written, ensuring nothing important is lost on reboot.

5. Benefits: Systems remain clean and consistent, akin to a fresh installation every boot. This model works across diverse hardwarefrom servers to Raspberry Pisenhancing reliability and reducing legacy baggage. Regular reboots confirm the systems resilience and proper state management.

The author emphasizes proactive consideration of each services data needs during setup, ensuring all critical state is preserved while embracing ephemeral infrastructure. The approach leverages NixOSs declarative configuration to maintain consistency, making it a robust reference architecture for modern deployments.