microvm-ram-snapshot-clones¶
Modules
Per-project throwaway dev VMs that spawn in under a second. A NixOS module
that gives you a vm command: cd into any project, run vm, and you're
dropped into a fresh, isolated, already-booted microvm whose workspace is that
directory.
The problem¶
You want a real, isolated Linux environment per project — full systemd, real services, a place to run untrusted build steps or agents — but:
- A container isn't a real machine (shared kernel, awkward systemd).
- A normal VM takes tens of seconds to boot and a fat disk image per instance.
- Cloud dev environments are slow to reach and cost money to idle.
The key insight¶
The "snapshot" is a frozen-RAM image, not a disk image.
- Boot the guest microvm exactly once.
stopthe CPU over QMP, thenmigratetoexec:zstd— a compressed dump of the guest's live RAM.- A clone is that image piped back into a fresh QEMU via
-incoming "exec:zstd -d -c snapshot.zst". The guest wakes up already booted — no init, no service startup, no waiting.
Because the guest is a microvm booted straight from -kernel/-initrd/init=
with a shared memfd memory backend, restore is essentially the cost of
zstd -d plus SSH coming up.
Storage: no block device at all¶
There is no disk image. The guest's only storage is two virtiofs mounts, and the cache modes are load-bearing:
| Mount | Host source | Cache | Why |
|---|---|---|---|
/nix/store |
host /nix/store |
cache=always |
Store is immutable; sharing it read-only means clones cost zero store duplication — every clone reuses the host closure. |
| workspace | the project dir | cache=auto |
Bound read-write, so edits inside the clone are edits on the host tree. This is the whole point — you work on your real files. |
Nothing else survives a stop. The clone is disposable; your files are not,
because they were never in the clone.
Clone identity is derived, not assigned¶
The same directory always maps to the same clone and port, so vm from a
project is idempotent — re-enter if running, spawn if not. Trap: the port
space is only 800 wide, so two different project dirs can collide on a port.
Fine for a personal workstation; think twice for many-user hosts.
The base snapshot is cache-keyed on the guest closure¶
vm update records nix hash path <guest-toplevel> next to the snapshot. If
the hash is unchanged, it's a no-op. The base image only rebuilds when you
actually change the guest system definition, so the boot-once cost is paid
rarely.
Usage¶
# flake.nix / configuration.nix
{
imports = [ ./modules/microvm-ram-snapshot-clones ];
modules.microvmClone = {
enable = true;
guestUser = "dev";
guestSystem = self.nixosConfigurations.devvm-guest; # see below
};
}
Then, on the host:
cd ~/projects/foo
vm # spawn (or re-enter) the clone for this dir, drop into a shell
vm status # name / port / snapshot size
vm list # all clones: running / saved / dead
vm save "wip: refactoring the parser, mid-migration" # freeze to disk (>20 char desc)
vm restore # thaw a saved clone (spawn does this automatically too)
vm pause / vm resume # stop and restart the clone's CPU over QMP
vm root-enter # SSH in as root
vm logs # journalctl -f from inside the clone
vm stop # kill and forget the clone
sudo vm update # (re)build the base snapshot; auto-runs at boot
vm completions bash|zsh|fish # shell completions (also installed into /etc)
vm help # full command list
spawn is the default when no subcommand is given, and enter / init /
start are aliases for it. Every directory-taking command accepts an optional
[dir], defaulting to $PWD; a bare vm <existing-dir> spawns for that dir.
Options¶
| Option | Default | Meaning |
|---|---|---|
enable |
false |
Turn the module on. |
guestSystem |
— (required) | An evaluated nixosSystem for the guest, including microvm.nixosModules.microvm. |
guestUser |
"dev" |
Unprivileged user to SSH into inside the guest. |
guestWorkspace |
/home/<guestUser>/workspace |
Where the project dir is mounted in the guest; interactive sessions cd here. |
snapshotDir |
/var/lib/microvm-clone-snapshot |
Persistent base snapshot. |
clonesDir |
/run/microvm-clones |
Per-clone state. On tmpfs by default, so clones vanish on reboot. |
snapshotAtBoot |
true |
Build the base snapshot from a boot-time systemd oneshot. |
direnv |
false |
Install a layout_devvm that auto-spawns a clone on cd and drops vm/vm-run/vm-stop shims into .direnv/bin. |
sshExtraOpts |
"" |
Extra SSH opts on every connection (e.g. -i <key>). |
sshKeySetup |
"" |
Shell run before SSH (e.g. materialize a key from the store into a 0600 tempfile). |
sshEnterOpts |
"" |
Extra SSH opts for interactive sessions only (e.g. GPG forwarding). |
forwardAgent |
false |
Forward your ssh-agent (ssh -A) into interactive sessions. Off by default — see the security note below. |
The guest system¶
guestSystem is an externally-evaluated microvm nixosSystem. The module
reaches into it for microvm.kernel, microvm.initrdPath,
system.build.toplevel, microvm.mem, microvm.vcpu and boot.kernelParams,
and boots those directly with QEMU (no bootloader, init=<toplevel>/init).
Your guest must satisfy three contracts:
- Be a microvm — import
microvm.nixosModules.microvm(from astro/microvm.nix) and setmem/vcpu. - Mount the two virtiofs shares by the tags the host uses:
ro-storeat/nix/storeandworkspaceatguestWorkspace. - Signal boot readiness by touching
.boot-readyin the workspace mount, and run an sshd the host can reach on the forwarded port (root andguestUser, key-based).
A minimal sketch (adapt to your own keys and packages):
# devvm-guest.nix — evaluated with nixpkgs.lib.nixosSystem and passed as guestSystem
{ modulesPath, lib, ... }:
{
imports = [ /* microvm.nixosModules.microvm */ ];
microvm.mem = 4096;
microvm.vcpu = 4;
# Host mounts /nix/store RO and the project dir RW; match the tags.
microvm.shares = [
{ source = "/nix/store"; mountPoint = "/nix/.ro-store"; tag = "ro-store";
proto = "virtiofs"; }
{ source = "/dev/null"; mountPoint = "/home/dev/workspace"; tag = "workspace";
proto = "virtiofs"; }
];
users.users.dev = {
isNormalUser = true;
openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAA... you@example.com" ];
};
users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAA... you@example.com" ];
services.openssh = {
enable = true;
settings.PermitRootLogin = "prohibit-password";
};
# Tell the host we've finished booting.
systemd.services.boot-ready = {
wantedBy = [ "multi-user.target" ];
after = [ "sshd.service" ];
serviceConfig.Type = "oneshot";
script = "touch /home/dev/workspace/.boot-ready";
};
system.stateVersion = "24.05";
}
The exact
microvm.sharesschema depends on your microvm.nix version — the point is: exposero-store/workspacetags and mount them. The host passes-device vhost-user-fs-pci,...,tag=ro-storeandtag=workspace.
Caveats¶
- KVM required for usable speed (
accel=kvm:tcgfalls back to slow TCG). - x86_64 only as written (
qemu-system-x86_64,-cpu max). Adjust the QEMU binary and machine type for other arches. - Port collisions — 800-wide derived port space (see above).
vm updateis privileged — it boots the base VM with virtiofsd and writes the snapshot intosnapshotDir, which is created0755 root root; so it runs as root (systemd oneshot at boot, orsudo vm update).clonesDiron tmpfs means running clones and their RAM state die on reboot. That's intentional (disposable), butvm saveis the escape hatch: it re-freezes a running clone tosave.zst, which a latervm/vm restorethaws. PointclonesDirat persistent storage if you want clones to survive reboots.- The guest closure is shared, not copied — a clone sees the host's live
/nix/store. Don't GC the store out from under a running clone. - SSH host keys are ignored (
StrictHostKeyChecking=no,UserKnownHostsFile=/dev/null) because clones are ephemeral and share a base image. Fine for localhost-forwarded ports; don't reuse this SSH posture for anything routable.
Security notes¶
- Agent forwarding is opt-in.
ssh -Ais not used unless you setforwardAgent = true. Because the whole point of a clone is to run possibly-untrusted build steps or agents, forwarding your ssh-agent would let anything inside the guest authenticate to any host your keys reach (GitHub, production, …) without ever seeing the key material. Leave it off unless the guest workload is trusted. clonesDiris shared and world-writable, but sticky (1777). ThevmCLI writes pidfiles, sockets and saved state underclonesDir, and clone identity is a predictablemd5(realpath project-dir). The sticky bit stops one local user from renaming or deleting another user's clone state (which could otherwise plant a pidfile sovm stop/vm listkills an arbitrary PID, or redirect writes via a symlinkedCLONE_DIR). On a multi-user host, give each user a privateclonesDir(e.g. under/run/user/$UID) for full isolation; on a single-user workstation the default is fine.
How it fits together¶
vm update vm (spawn)
| |
boot guest once virtiofsd: /nix/store (RO, cache=always)
QMP stop + migrate virtiofsd: project dir (RW, cache=auto)
-> zstd -> snapshot.zst qemu -incoming "exec:zstd -d -c snapshot.zst"
QMP cont -> guest already booted
ssh -p <derived port> -> your shell in the workspace
Source¶
modules/microvm-ram-snapshot-clones/default.nix
# microvm-ram-snapshot-clones
#
# Sub-second, per-project throwaway dev VMs. Boot a microvm guest once, freeze
# its live RAM to a compressed image (QMP `stop` + `migrate` to `exec:zstd`),
# then spawn clones by streaming that image back into a fresh QEMU via
# `-incoming exec:zstd -d`. Clones wake up already-booted.
#
# There is no block device at all: /nix/store is shared read-only over
# virtiofs (cache=always, so clones cost zero store duplication), and the
# project directory is bound read-write as the workspace (cache=auto), so
# edits inside the clone are edits on the host tree.
#
# Clone identity is *derived*, not assigned: md5(realpath project-dir) yields a
# stable VM name and SSH port, so the same directory always maps to the same
# clone. The base snapshot only rebuilds when the guest `toplevel` nix hash
# changes, so refresh is a cheap no-op most of the time.
#
# This is a generic NixOS module. Import it, point `guestSystem` at an
# evaluated microvm nixosSystem, set `guestUser`, and enable. See README.md.
{
config,
lib,
pkgs,
...
}:
let
cfg = config.modules.microvmClone;
# The guest is an evaluated microvm nixosSystem. We reach into it for the
# kernel, initrd and toplevel that QEMU boots directly (no bootloader).
vmCfg = cfg.guestSystem.config;
kernel = "${vmCfg.microvm.kernel.out}/${pkgs.linux.target}";
initrd = vmCfg.microvm.initrdPath;
toplevel = vmCfg.system.build.toplevel;
mem = toString vmCfg.microvm.mem;
vcpu = toString vmCfg.microvm.vcpu;
kernelParams = lib.concatStringsSep " " vmCfg.boot.kernelParams;
virtiofsd = "${pkgs.virtiofsd}/bin/virtiofsd";
qemu = "${pkgs.qemu_kvm}/bin/qemu-system-x86_64";
zstd = "${pkgs.zstd}/bin/zstd";
ssh = "${pkgs.openssh}/bin/ssh";
guestUser = cfg.guestUser;
workspace = cfg.guestWorkspace;
sshOpts = "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR ${cfg.sshExtraOpts}";
# Agent forwarding is off by default: anything running in the guest (including
# the untrusted build steps/agents this VM is meant to host) can use a
# forwarded agent socket to authenticate as the caller. Opt in via
# forwardAgent only when the guest is trusted.
sshAgentOpt = lib.optionalString cfg.forwardAgent "-A";
# Boot the microvm straight from kernel+initrd+toplevel. The memory backend is
# a shared memfd, which is what makes the live-RAM migrate/restore cheap.
commonQemuFlags = ''
-M q35,accel=kvm:tcg \
-m ${mem} \
-smp ${vcpu} \
-nodefaults -no-user-config -no-reboot -nographic \
-cpu max \
-kernel ${kernel} \
-initrd ${initrd} \
-append "earlyprintk=ttyS0 console=ttyS0 reboot=t panic=-1 ${kernelParams} init=${toplevel}/init" \
-numa node,memdev=mem \
-object memory-backend-memfd,id=mem,size=${mem}M,share=on'';
# Minimal QMP client. Speaks the JSON line protocol to drive
# stop/migrate/cont, which is how we freeze and thaw a running guest.
qmpScript =
pkgs.writers.writePython3 "microvm-clone-qmp"
{
flakeIgnore = [
"E401"
"E302"
"E305"
"E501"
"W391"
];
}
''
import socket, json, sys, time
class Conn:
def __init__(self, sock):
self.sock = sock
self.buf = b""
def recv_json(self):
while b"\n" not in self.buf:
data = self.sock.recv(4096)
if not data:
raise ConnectionError("closed")
self.buf += data
while True:
line, _, self.buf = self.buf.partition(b"\n")
line = line.strip()
if line:
return json.loads(line)
if b"\n" not in self.buf:
while b"\n" not in self.buf:
data = self.sock.recv(4096)
if not data:
raise ConnectionError("closed")
self.buf += data
def send(self, cmd, **args):
msg = {"execute": cmd}
if args:
msg["arguments"] = args
self.sock.sendall(json.dumps(msg).encode() + b"\n")
while True:
r = self.recv_json()
if "event" not in r:
return r
path = sys.argv[1]
action = sys.argv[2]
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.settimeout(30)
sock.connect(path)
conn = Conn(sock)
conn.recv_json()
conn.send("qmp_capabilities")
if action == "snapshot":
uri = sys.argv[3]
conn.send("stop")
for _ in range(60):
r = conn.send("query-status")
if r.get("return", {}).get("status") == "paused":
break
time.sleep(0.1)
conn.send("migrate", uri=uri)
for _ in range(600):
r = conn.send("query-migrate")
st = r.get("return", {}).get("status", "")
if st == "completed":
print("OK", file=sys.stderr)
break
if st == "failed":
print(f"FAILED: {r}", file=sys.stderr)
sys.exit(1)
time.sleep(1)
else:
print("TIMEOUT", file=sys.stderr)
sys.exit(1)
conn.send("quit")
elif action == "resume":
for i in range(120):
r = conn.send("query-status")
st = r.get("return", {}).get("status", "")
print(f"status: {st}", file=sys.stderr)
if st not in ("inmigrate", "none"):
break
time.sleep(0.5)
r = conn.send("cont")
print(f"cont: {r}", file=sys.stderr)
for _ in range(60):
r = conn.send("query-status")
st = r.get("return", {}).get("status", "")
if st == "running":
print(f"after cont: {st}", file=sys.stderr)
break
time.sleep(0.1)
elif action == "pause":
conn.send("stop")
for _ in range(60):
r = conn.send("query-status")
if r.get("return", {}).get("status") == "paused":
print("paused", file=sys.stderr)
break
time.sleep(0.1)
elif action == "quit":
conn.send("quit")
sock.close()
'';
# Derive clone identity from the project path: same dir -> same name+port.
# Note the port space is 800 wide, so distinct dirs *can* collide on a port.
resolve = ''
resolve_clone() {
local project_dir="$(realpath "''${1:-$PWD}")"
VM_HASH=$(echo "$project_dir" | md5sum | cut -c1-8)
VM_NAME="devvm-''${VM_HASH}"
CLONE_DIR="${cfg.clonesDir}/''${VM_NAME}"
SSH_PORT=$((2200 + 16#''${VM_HASH:0:3} % 800))
PROJECT_DIR="$project_dir"
}
clone_running() {
[ -f "''${CLONE_DIR}/qemu.pid" ] && \
kill -0 "$(cat "''${CLONE_DIR}/qemu.pid")" 2>/dev/null
}
'';
vmCli = pkgs.writeShellScriptBin "vm" ''
set -euo pipefail
${resolve}
die() { echo "error: $*" >&2; exit 1; }
cmd_snapshot() {
SNAPSHOT_DIR="${cfg.snapshotDir}"
SNAPSHOT_FILE="$SNAPSHOT_DIR/snapshot.zst"
BUILD_HASH="$SNAPSHOT_DIR/build-hash"
CURRENT_HASH=$(${pkgs.nix}/bin/nix hash path ${toplevel} 2>/dev/null || echo "unknown")
# The snapshot only rebuilds when the guest toplevel hash changes, so
# this is a cheap no-op most of the time.
if [ -f "$BUILD_HASH" ] && [ -f "$SNAPSHOT_FILE" ]; then
if [ "$(cat "$BUILD_HASH")" = "$CURRENT_HASH" ]; then
echo "Snapshot up to date"
exit 0
fi
fi
echo "Creating base RAM snapshot..." >&2
mkdir -p "$SNAPSHOT_DIR"
EMPTY_DIR=$(mktemp -d)
trap 'rm -rf "$EMPTY_DIR"; kill $(jobs -p) 2>/dev/null || true' EXIT
# RO /nix/store share: cache=always, since the store is immutable.
${virtiofsd} \
--socket-path="$SNAPSHOT_DIR/virtiofs-ro-store.sock" \
--shared-dir=/nix/store \
--cache=always >> "$SNAPSHOT_DIR/virtiofsd.log" 2>&1 &
VFSD_STORE_PID=$!
# Base image is captured with an empty workspace mount; clones bind the
# real project dir over the same tag at spawn time.
${virtiofsd} \
--socket-path="$SNAPSHOT_DIR/virtiofs-workspace.sock" \
--shared-dir="$EMPTY_DIR" \
--cache=never >> "$SNAPSHOT_DIR/virtiofsd.log" 2>&1 &
VFSD_WORK_PID=$!
SOCK_OK=0
for _i in $(seq 1 100); do
if ! kill -0 $VFSD_STORE_PID 2>/dev/null || ! kill -0 $VFSD_WORK_PID 2>/dev/null; then
echo "virtiofsd died. Log:" >&2
cat "$SNAPSHOT_DIR/virtiofsd.log" >&2
exit 1
fi
if [ -S "$SNAPSHOT_DIR/virtiofs-ro-store.sock" ] && \
[ -S "$SNAPSHOT_DIR/virtiofs-workspace.sock" ]; then
SOCK_OK=1
break
fi
sleep 0.2
done
[ "$SOCK_OK" -eq 1 ] || { echo "virtiofsd sockets timeout" >&2; exit 1; }
echo "Booting base VM..." >&2
nohup ${qemu} \
-name devvm-snapshot \
${commonQemuFlags} \
-serial file:"$SNAPSHOT_DIR/console.log" \
-chardev socket,id=fs-ro-store,path="$SNAPSHOT_DIR/virtiofs-ro-store.sock" \
-device vhost-user-fs-pci,chardev=fs-ro-store,tag=ro-store \
-chardev socket,id=fs-workspace,path="$SNAPSHOT_DIR/virtiofs-workspace.sock" \
-device vhost-user-fs-pci,chardev=fs-workspace,tag=workspace \
-netdev user,id=net0,hostfwd=tcp::2299-:22 \
-device virtio-net-pci,netdev=net0,mac=02:00:00:ff:ff:ff \
-qmp unix:"$SNAPSHOT_DIR/qmp.sock",server,nowait \
> /dev/null 2>&1 &
echo $! > "$SNAPSHOT_DIR/qemu.pid"
for _i in $(seq 1 60); do
[ -S "$SNAPSHOT_DIR/qmp.sock" ] && break
if ! kill -0 "$(cat "$SNAPSHOT_DIR/qemu.pid")" 2>/dev/null; then
echo "QEMU failed to start" >&2; exit 1
fi
sleep 0.2
done
# The guest must signal readiness by touching .boot-ready in the
# workspace mount (see README for the guest-side one-liner).
echo "Waiting for boot..." >&2
BOOT_OK=0
for _i in $(seq 1 900); do
if [ -f "$SNAPSHOT_DIR/qemu.pid" ] && ! kill -0 "$(cat "$SNAPSHOT_DIR/qemu.pid")" 2>/dev/null; then
echo "QEMU died during boot" >&2
tail -n 20 "$SNAPSHOT_DIR/console.log" >&2 2>/dev/null || true
exit 1
fi
if [ -f "$EMPTY_DIR/.boot-ready" ]; then
BOOT_OK=1
break
fi
sleep 1
done
[ "$BOOT_OK" -eq 1 ] || { echo "Boot timeout" >&2; tail -n 40 "$SNAPSHOT_DIR/console.log" >&2 2>/dev/null; exit 1; }
# Freeze the CPU, then dump live RAM through zstd to the snapshot file.
echo "Saving RAM snapshot..." >&2
rm -f "$SNAPSHOT_FILE"
${qmpScript} "$SNAPSHOT_DIR/qmp.sock" snapshot "exec:${zstd} -T0 -3 -o $SNAPSHOT_FILE"
if [ -f "$SNAPSHOT_DIR/qemu.pid" ]; then
for _i in $(seq 1 30); do
kill -0 "$(cat "$SNAPSHOT_DIR/qemu.pid")" 2>/dev/null || break
sleep 0.2
done
fi
rm -f "$SNAPSHOT_DIR"/*.sock "$SNAPSHOT_DIR"/qmp.sock "$SNAPSHOT_DIR"/qemu.pid
echo "$CURRENT_HASH" > "$BUILD_HASH"
echo "Snapshot ready: $(du -h "$SNAPSHOT_FILE" | cut -f1)" >&2
}
cmd_spawn() {
${cfg.sshKeySetup}
NO_ENTER=0
ARGS=()
for arg in "$@"; do
case "$arg" in
--no-enter) NO_ENTER=1 ;;
*) ARGS+=("$arg") ;;
esac
done
resolve_clone "''${ARGS[0]:-}"
if clone_running; then
if [ "$NO_ENTER" -eq 1 ]; then
echo "''${SSH_PORT}"
else
exec ${ssh} -t ${sshAgentOpt} ${sshOpts} ${cfg.sshEnterOpts} -p "''${SSH_PORT}" ${guestUser}@localhost "cd ${workspace}; exec \$SHELL -l"
fi
exit 0
fi
local RESTORING=0
local SNAPSHOT_FILE
if [ -f "''${CLONE_DIR}/save.zst" ]; then
SNAPSHOT_FILE="''${CLONE_DIR}/save.zst"
RESTORING=1
echo "Restoring: $(cat "''${CLONE_DIR}/save.desc" 2>/dev/null)" >&2
else
SNAPSHOT_FILE="${cfg.snapshotDir}/snapshot.zst"
[ -f "$SNAPSHOT_FILE" ] || die "No snapshot found. Run: sudo vm update"
echo "Spawning clone for $(basename "$PROJECT_DIR")..." >&2
fi
mkdir -p "''${CLONE_DIR}"
rm -f "''${CLONE_DIR}"/*.sock "''${CLONE_DIR}"/*.pid
# Same RO store share as the base image (cache=always, zero duplication).
${virtiofsd} \
--socket-path="''${CLONE_DIR}/virtiofs-ro-store.sock" \
--shared-dir=/nix/store \
--cache=always > "''${CLONE_DIR}/virtiofsd.log" 2>&1 &
echo $! > "''${CLONE_DIR}/virtiofsd-ro-store.pid"
# Bind the real project dir RW as the workspace (cache=auto): edits in
# the clone land on the host tree.
${virtiofsd} \
--socket-path="''${CLONE_DIR}/virtiofs-workspace.sock" \
--shared-dir="''${PROJECT_DIR}" \
--cache=auto >> "''${CLONE_DIR}/virtiofsd.log" 2>&1 &
echo $! > "''${CLONE_DIR}/virtiofsd-workspace.pid"
for _i in $(seq 1 100); do
[ -S "''${CLONE_DIR}/virtiofs-ro-store.sock" ] && \
[ -S "''${CLONE_DIR}/virtiofs-workspace.sock" ] && break
for pidfile in "''${CLONE_DIR}"/virtiofsd-*.pid; do
[ -f "$pidfile" ] && ! kill -0 "$(cat "$pidfile")" 2>/dev/null && \
{ echo "virtiofsd failed to start (see ''${CLONE_DIR}/virtiofsd.log)" >&2; exit 1; }
done
sleep 0.2
done
# Restore: stream the frozen RAM back in. The guest wakes already-booted.
printf " Restoring from snapshot..." >&2
nohup ${qemu} \
-name "''${VM_NAME}" \
${commonQemuFlags} \
-serial file:"''${CLONE_DIR}/console.log" \
-chardev socket,id=fs-ro-store,path="''${CLONE_DIR}/virtiofs-ro-store.sock" \
-device vhost-user-fs-pci,chardev=fs-ro-store,tag=ro-store \
-chardev socket,id=fs-workspace,path="''${CLONE_DIR}/virtiofs-workspace.sock" \
-device vhost-user-fs-pci,chardev=fs-workspace,tag=workspace \
-netdev user,id=net0,hostfwd=tcp::''${SSH_PORT}-:22 \
-device virtio-net-pci,netdev=net0,mac=02:00:00:00:00:10 \
-qmp unix:"''${CLONE_DIR}/qmp.sock",server,nowait \
-incoming "exec:${zstd} -d -c $SNAPSHOT_FILE" \
> /dev/null 2>&1 &
echo $! > "''${CLONE_DIR}/qemu.pid"
for _i in $(seq 1 120); do
[ -S "''${CLONE_DIR}/qmp.sock" ] && break
sleep 0.5
done
${qmpScript} "''${CLONE_DIR}/qmp.sock" resume 2>/dev/null
printf " done\n" >&2
printf " Waiting for SSH..." >&2
SSH_OK=0
for _i in $(seq 1 60); do
if ${ssh} -q ${sshOpts} -o ConnectTimeout=1 -p "''${SSH_PORT}" ${guestUser}@localhost true 2>/dev/null; then
SSH_OK=1
break
fi
if ! kill -0 "$(cat "''${CLONE_DIR}/qemu.pid" 2>/dev/null)" 2>/dev/null; then
printf " failed\n" >&2
echo "QEMU died during restore (see ''${CLONE_DIR}/console.log)" >&2
exit 1
fi
sleep 0.5
done
if [ "$SSH_OK" -ne 1 ]; then
printf " failed\n" >&2
echo "SSH timeout on port ''${SSH_PORT}" >&2
exit 1
fi
printf " done\n" >&2
if [ "$RESTORING" -eq 1 ]; then
rm -f "''${CLONE_DIR}/save.zst" "''${CLONE_DIR}/save.desc" "''${CLONE_DIR}/save.date"
fi
if [ "$NO_ENTER" -eq 1 ]; then
echo "''${SSH_PORT}"
else
exec ${ssh} -t ${sshAgentOpt} ${sshOpts} ${cfg.sshEnterOpts} -p "''${SSH_PORT}" ${guestUser}@localhost "cd ${workspace}; exec \$SHELL -l"
fi
}
cmd_save() {
local desc="$*"
if [ ''${#desc} -le 20 ]; then
die "Description required (>20 chars): vm save <what you're working on>"
fi
resolve_clone
clone_running || die "No running clone for $PROJECT_DIR"
echo "$PROJECT_DIR" > "''${CLONE_DIR}/project_dir"
echo "$desc" > "''${CLONE_DIR}/save.desc"
echo "$(date -Iseconds)" > "''${CLONE_DIR}/save.date"
printf "Saving VM state..." >&2
${qmpScript} "''${CLONE_DIR}/qmp.sock" snapshot "exec:${zstd} -T0 -3 -o ''${CLONE_DIR}/save.zst"
if [ -f "''${CLONE_DIR}/qemu.pid" ]; then
for _i in $(seq 1 30); do
kill -0 "$(cat "''${CLONE_DIR}/qemu.pid")" 2>/dev/null || break
sleep 0.2
done
fi
for pidfile in "''${CLONE_DIR}"/*.pid; do
[ -f "$pidfile" ] && kill "$(cat "$pidfile")" 2>/dev/null || true
done
rm -f "''${CLONE_DIR}"/*.sock "''${CLONE_DIR}"/qmp.sock "''${CLONE_DIR}"/qemu.pid "''${CLONE_DIR}"/*.pid
printf " done\n" >&2
echo "Saved: $desc" >&2
}
cmd_stop() {
resolve_clone "''${1:-}"
[ -d "''${CLONE_DIR}" ] || exit 0
if [ -S "''${CLONE_DIR}/qmp.sock" ]; then
${qmpScript} "''${CLONE_DIR}/qmp.sock" quit 2>/dev/null || true
if [ -f "''${CLONE_DIR}/qemu.pid" ]; then
for _i in $(seq 1 30); do
kill -0 "$(cat "''${CLONE_DIR}/qemu.pid")" 2>/dev/null || break
sleep 0.2
done
fi
fi
for pidfile in "''${CLONE_DIR}"/*.pid; do
[ -f "$pidfile" ] && kill "$(cat "$pidfile")" 2>/dev/null || true
done
rm -rf "''${CLONE_DIR}"
echo "Stopped ''${VM_NAME}" >&2
}
cmd_list() {
printf "%-12s %-8s %-6s %s\n" "CLONE" "STATUS" "PORT" "INFO"
for dir in ${cfg.clonesDir}/devvm-*/; do
[ -d "$dir" ] || continue
name=$(basename "$dir")
hash=''${name#devvm-}
port=$((2200 + 16#''${hash:0:3} % 800))
pid_file="$dir/qemu.pid"
if [ -f "$pid_file" ] && kill -0 "$(cat "$pid_file")" 2>/dev/null; then
printf "%-12s %-8s %-6s %s\n" "$name" "running" "$port" "pid $(cat "$pid_file")"
elif [ -f "$dir/save.zst" ]; then
printf "%-12s %-8s %-6s %s\n" "$name" "saved" "$port" "$(cat "$dir/save.desc" 2>/dev/null)"
else
printf "%-12s %-8s %-6s %s\n" "$name" "dead" "$port" "-"
fi
done
}
cmd_status() {
resolve_clone "''${1:-}"
echo "Project: $PROJECT_DIR"
echo "Clone: $VM_NAME"
echo "Port: $SSH_PORT"
if clone_running; then
echo "Status: running (pid $(cat "''${CLONE_DIR}/qemu.pid"))"
elif [ -f "''${CLONE_DIR}/save.zst" ]; then
echo "Status: saved"
echo "Desc: $(cat "''${CLONE_DIR}/save.desc" 2>/dev/null)"
echo "Date: $(cat "''${CLONE_DIR}/save.date" 2>/dev/null)"
elif [ -d "''${CLONE_DIR}" ]; then
echo "Status: dead"
else
echo "Status: not created"
fi
local snap="${cfg.snapshotDir}/snapshot.zst"
if [ -f "$snap" ]; then
echo "Snapshot: $(du -h "$snap" | cut -f1)"
else
echo "Snapshot: none"
fi
}
cmd_root_enter() {
${cfg.sshKeySetup}
resolve_clone "''${1:-}"
clone_running || die "Clone not running for $PROJECT_DIR"
exec ${ssh} -t ${sshAgentOpt} ${sshOpts} ${cfg.sshEnterOpts} -p "''${SSH_PORT}" root@localhost
}
cmd_logs() {
${cfg.sshKeySetup}
resolve_clone "''${1:-}"
clone_running || die "Clone not running for $PROJECT_DIR"
exec ${ssh} ${sshOpts} -p "''${SSH_PORT}" root@localhost journalctl -f
}
cmd_pause() {
resolve_clone "''${1:-}"
clone_running || die "Clone not running for $PROJECT_DIR"
${qmpScript} "''${CLONE_DIR}/qmp.sock" pause
echo "Paused ''${VM_NAME}" >&2
}
cmd_resume() {
resolve_clone "''${1:-}"
clone_running || die "Clone not running for $PROJECT_DIR"
${qmpScript} "''${CLONE_DIR}/qmp.sock" resume
echo "Resumed ''${VM_NAME}" >&2
}
cmd_completions() {
local shell="''${1:?Usage: vm completions <bash|zsh|fish>}"
case "$shell" in
bash)
cat <<'BASH_COMP'
_vm() {
local cur prev cmds
COMPREPLY=()
cur="''${COMP_WORDS[COMP_CWORD]}"
prev="''${COMP_WORDS[COMP_CWORD-1]}"
cmds="spawn enter init start save restore stop list status root-enter logs pause resume update help"
if [ "$COMP_CWORD" -eq 1 ]; then
COMPREPLY=($(compgen -W "$cmds" -- "$cur"))
COMPREPLY+=($(compgen -d -- "$cur"))
return
fi
case "$prev" in
spawn|enter|init|start|restore|stop|status|root-enter|logs|pause|resume)
COMPREPLY=($(compgen -d -- "$cur"))
;;
completions)
COMPREPLY=($(compgen -W "bash zsh fish" -- "$cur"))
;;
esac
}
complete -o filenames -F _vm vm
BASH_COMP
;;
zsh)
cat <<'ZSH_COMP'
_vm() {
local -a commands=(
'spawn:Spawn clone and enter'
'enter:Spawn clone and enter'
'init:Spawn clone and enter'
'start:Spawn clone and enter'
'save:Hibernate clone to disk'
'restore:Restore a saved clone'
'stop:Stop and remove clone'
'list:List all clones'
'status:Detailed status of a clone'
'root-enter:SSH as root'
'logs:Stream journalctl from clone'
'pause:Pause clone CPU'
'resume:Resume paused clone'
'update:Create/refresh base RAM snapshot'
'completions:Output shell completions'
'help:Show help'
)
if (( CURRENT == 2 )); then
_describe 'command' commands
_directories
return
fi
case "''${words[2]}" in
spawn|enter|init|start|restore|stop|status|root-enter|logs|pause|resume)
_directories
;;
completions)
_values 'shell' bash zsh fish
;;
esac
}
_vm "$@"
ZSH_COMP
;;
fish)
cat <<'FISH_COMP'
set -l commands spawn enter init start save restore stop list status root-enter logs pause resume update completions help
set -l dir_cmds spawn enter init start restore stop status root-enter logs pause resume
complete -c vm -f
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a spawn -d 'Spawn clone and enter'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a enter -d 'Enter clone'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a save -d 'Hibernate clone to disk'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a restore -d 'Restore saved clone'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a stop -d 'Stop clone'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a list -d 'List all clones'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a status -d 'Clone status'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a root-enter -d 'SSH as root'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a logs -d 'Stream logs'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a pause -d 'Pause clone'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a resume -d 'Resume clone'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a update -d 'Refresh base snapshot'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a help -d 'Show help'
complete -c vm -n "not __fish_seen_subcommand_from $commands" -a "(__fish_complete_directories)"
complete -c vm -n "__fish_seen_subcommand_from $dir_cmds" -a "(__fish_complete_directories)"
complete -c vm -n "__fish_seen_subcommand_from completions" -a "bash zsh fish"
FISH_COMP
;;
*)
die "Unknown shell: $shell (use bash, zsh, or fish)"
;;
esac
}
cmd_help() {
cat <<'HELP'
vm - NixOS dev VM manager (RAM snapshot/clone)
Usage: vm [command] [dir]
Commands:
(none) Spawn or enter clone for current directory
spawn [dir] Spawn clone and enter (aliases: enter, init, start)
save <description> Hibernate clone to disk (description >20 chars required)
restore [dir] Restore a saved clone (also happens automatically via spawn)
stop [dir] Stop and remove clone
list List all clones (running, saved, dead)
status [dir] Detailed status of a clone
root-enter [dir] SSH as root
logs [dir] Stream journalctl from clone
pause [dir] Pause clone CPU
resume [dir] Resume paused clone
update Create/refresh base RAM snapshot (requires sudo)
completions <shell> Output shell completions (bash|zsh|fish)
help Show this help
If [dir] is omitted, the current directory is used.
HELP
}
cmd="''${1:-spawn}"
shift || true
case "$cmd" in
spawn|enter|init|start) cmd_spawn "$@" ;;
save) cmd_save "$@" ;;
restore)
resolve_clone "''${1:-}"
[ -f "''${CLONE_DIR}/save.zst" ] || die "No saved state. Use 'vm save' first."
cmd_spawn "$@"
;;
stop) cmd_stop "$@" ;;
list|ls) cmd_list ;;
status) cmd_status "$@" ;;
root-enter) cmd_root_enter "$@" ;;
logs) cmd_logs "$@" ;;
pause) cmd_pause "$@" ;;
resume) cmd_resume "$@" ;;
update|snapshot) cmd_snapshot ;;
completions) cmd_completions "$@" ;;
help|--help|-h) cmd_help ;;
*)
if [ -d "$cmd" ]; then
cmd_spawn "$cmd" "$@"
else
die "Unknown command: $cmd (try 'vm help')"
fi
;;
esac
'';
in
{
options.modules.microvmClone = {
enable = lib.mkEnableOption "per-project RAM-snapshot microvm clones";
guestSystem = lib.mkOption {
type = lib.types.unspecified;
description = ''
Evaluated nixosSystem for the guest VM. Must include
`microvm.nixosModules.microvm` and configure an sshd the host can reach
over the forwarded port. See README.md for a minimal example guest.
'';
};
guestUser = lib.mkOption {
type = lib.types.str;
default = "dev";
description = "Unprivileged user to SSH into inside the guest (must exist in the guest system).";
};
guestWorkspace = lib.mkOption {
type = lib.types.str;
default = "/home/${cfg.guestUser}/workspace";
description = "Path inside the guest where the project dir (virtiofs workspace tag) is mounted; interactive sessions cd here.";
};
snapshotDir = lib.mkOption {
type = lib.types.str;
default = "/var/lib/microvm-clone-snapshot";
description = "Persistent directory holding the base RAM snapshot.";
};
clonesDir = lib.mkOption {
type = lib.types.str;
default = "/run/microvm-clones";
description = "Per-clone runtime state. Default under /run (tmpfs), so clones are disposable and vanish on reboot.";
};
snapshotAtBoot = lib.mkOption {
type = lib.types.bool;
default = true;
description = "Auto-create the base snapshot at boot via a systemd oneshot.";
};
direnv = lib.mkOption {
type = lib.types.bool;
default = false;
description = "Install a direnv `layout_devvm` that auto-spawns a clone on directory entry and drops vm/vm-run/vm-stop shims into .direnv/bin.";
};
sshExtraOpts = lib.mkOption {
type = lib.types.str;
default = "";
description = "Extra SSH options appended to all ssh commands (e.g., -i keypath for testing).";
};
sshKeySetup = lib.mkOption {
type = lib.types.str;
default = "";
description = "Shell commands run before SSH connections (e.g., copy a private key out of the nix store into a mode-0600 tempfile and add it to sshExtraOpts).";
};
sshEnterOpts = lib.mkOption {
type = lib.types.str;
default = "";
description = "Extra SSH options for interactive sessions only (e.g., GPG forwarding).";
};
forwardAgent = lib.mkOption {
type = lib.types.bool;
default = false;
description = ''
Forward the caller's ssh-agent (`ssh -A`) into interactive clone
sessions and the direnv shims. Off by default: anything running in the
guest — including the untrusted build steps or agents this VM is meant
to host — can use a forwarded agent socket to authenticate as you to any
host your keys reach, without ever seeing the key material. Only enable
when the code running inside the guest is trusted.
'';
};
};
config = lib.mkIf cfg.enable (
lib.mkMerge [
{
environment.systemPackages = [ vmCli ];
environment.etc."bash_completion.d/vm".source = pkgs.runCommand "vm-bash-comp" { } ''
${vmCli}/bin/vm completions bash > $out
'';
environment.etc."fish/vendor_completions.d/vm.fish".source = pkgs.runCommand "vm-fish-comp" { } ''
${vmCli}/bin/vm completions fish > $out
'';
environment.etc."zsh/site-functions/_vm".source = pkgs.runCommand "vm-zsh-comp" { } ''
${vmCli}/bin/vm completions zsh > $out
'';
systemd.services.microvm-clone-snapshot = lib.mkIf cfg.snapshotAtBoot {
description = "Create base RAM snapshot for microvm clone spawning";
wantedBy = [ "multi-user.target" ];
after = [ "network.target" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = "${vmCli}/bin/vm update";
TimeoutStartSec = "300";
};
};
# clonesDir is a shared runtime dir the vm CLI writes pid/socket/state
# files into. Sticky bit (1777) so any local user can create their own
# clone state but cannot rename or delete another user's — without the
# sticky bit, since clone identity is a predictable md5(project-dir), a
# local attacker could pre-create or clobber a victim's CLONE_DIR (plant
# a pidfile to make `vm stop`/`vm list` kill an arbitrary PID, or symlink
# the dir elsewhere). On a single-user workstation 0755 is also fine; set
# clonesDir per-user (e.g. under /run/user/$UID) to isolate fully.
systemd.tmpfiles.rules = [
"d ${cfg.snapshotDir} 0755 root root - -"
"d ${cfg.clonesDir} 1777 root root - -"
];
}
(lib.mkIf cfg.direnv {
environment.etc."devvm-direnv-layout.sh" = {
text = ''
layout_devvm() {
local project_dir="$PWD"
local port
port=$(vm spawn --no-enter "$project_dir" 2>/dev/null | tail -1)
if [ -z "$port" ]; then
log_error "vm spawn failed for $project_dir"
return 1
fi
export DEVVM_PORT="$port"
export DEVVM_NAME="devvm-$(echo "$project_dir" | md5sum | cut -c1-8)"
export DEVVM_PROJECT="$project_dir"
local bin_dir
bin_dir="$(direnv_layout_dir)/bin"
PATH_add "$bin_dir"
mkdir -p "$bin_dir"
cat > "$bin_dir/vm" <<VMSCRIPT
#!/usr/bin/env bash
exec ssh -t ${sshAgentOpt} \
-o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null \
-o LogLevel=ERROR \
${cfg.sshEnterOpts} \
-p $port ${guestUser}@localhost \
"cd ${workspace}; \''${*:-\$SHELL}"
VMSCRIPT
chmod +x "$bin_dir/vm"
cat > "$bin_dir/vm-run" <<VMSCRIPT
#!/usr/bin/env bash
exec ssh ${sshAgentOpt} \
-o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null \
-o LogLevel=ERROR \
${cfg.sshEnterOpts} \
-p $port ${guestUser}@localhost \
"cd ${workspace}; \$*"
VMSCRIPT
chmod +x "$bin_dir/vm-run"
cat > "$bin_dir/vm-stop" <<VMSCRIPT
#!/usr/bin/env bash
exec vm stop "$project_dir"
VMSCRIPT
chmod +x "$bin_dir/vm-stop"
log_status "devvm clone ready (port $port) — type 'vm' to enter"
}
'';
};
# Runs as root at every activation over user-owned homes. Treat the
# per-home path as hostile: derive the owner from the directory itself
# (not basename), refuse to traverse any user-planted symlink in the
# .config/direnv/lib chain, and never `chown -R` a user-controlled root
# — only create real dirs and chown the single symlink we own.
system.activationScripts.devvm-direnv-link = ''
for home in /home/*/; do
home="''${home%/}"
[ -d "$home" ] || continue
[ -L "$home" ] && continue
user="$(stat -c %U "$home" 2>/dev/null)" || continue
case "$user" in "" | UNKNOWN | root) continue ;; esac
# Bail out if any component is a symlink or a non-directory, so root
# never follows a symlink out of the user's home.
ok=1
dir="$home"
for part in .config direnv lib; do
dir="$dir/$part"
if [ -L "$dir" ] || { [ -e "$dir" ] && [ ! -d "$dir" ]; }; then
ok=0
break
fi
done
[ "$ok" = 1 ] || continue
install -d -o "$user" -m 0755 "$home/.config/direnv/lib" 2>/dev/null || continue
ln -sfn /etc/devvm-direnv-layout.sh "$home/.config/direnv/lib/devvm.sh"
chown -h "$user" "$home/.config/direnv/lib/devvm.sh" 2>/dev/null || true
done
'';
})
]
);
}