per-uid-egress-lockdown¶
Modules
Run an untrusted program — a code agent, a scraper, a vendored build script you did not read — as its own uid, inside bubblewrap, with no network at all except a loopback CONNECT proxy that enforces a domain allowlist.
The allowlist is enforced twice, and the second one is the only one that counts.
The problem¶
"Give this thing internet access to exactly these domains" is normally
implemented as a proxy plus HTTPS_PROXY in the environment. That is a
convention, not a control. Anything the program runs — a package manager's
post-install hook, a vendored binary, a Python module that builds its own
urllib3 pool — can simply not read the variable, open its own socket, and go
wherever it likes. Your allowlist becomes a suggestion the well-behaved parts of
the program follow.
The fix is to make "not using the proxy" mean "no network", in the kernel:
| Layer | What it enforces | How it is bypassed |
|---|---|---|
squid http_access allow CONNECT <allowlist> |
which domains | ignore HTTPS_PROXY |
nftables output chain, meta skuid <uid> drop |
that the proxy is used at all | you would need another uid |
Layer 2 is what makes layer 1 real. With the drop rule in place, a program that
ignores the proxy env vars gets ENETUNREACH on its very first connect() —
not the open internet.
Trap 1 — the proxy MUST run as a different uid¶
This is the corollary that catches people, and it is easy to get wrong in a way that silently reverts the whole design.
The drop rule is meta skuid <sandbox uid> drop. If the proxy runs as that same
uid, the proxy's own outbound packets match the drop rule and the proxy cannot
reach anything. The obvious "fix" is to punch a hole — allow tcp dport 443 for
the uid — and now the sandboxed program has direct HTTPS to the entire internet.
You have reintroduced exactly the bypass the table existed to close, and
everything still looks like it works, because the proxy also still works.
So: two uids, both pinned, and the module hard-asserts they differ:
The proxy uid gets its own, much narrower ruleset — DNS plus the CONNECT target ports, nothing else — so a compromised squid is not a general exfiltration channel either:
meta skuid 60900 oifname "lo" ct state established,related accept
meta skuid 60900 oifname "lo" tcp dport 3128 accept
meta skuid 60900 drop
meta skuid 60901 ct state established,related accept
meta skuid 60901 udp dport 53 accept
meta skuid 60901 tcp dport 53 accept
meta skuid 60901 tcp dport 443 accept
meta skuid 60901 oifname "lo" accept
meta skuid 60901 drop
Note the sandbox uid has no DNS. It cannot resolve anything; resolution
happens inside the proxy as part of CONNECT host:443. The failure signature
when a program tries to resolve directly is a name resolution error
(EAI_AGAIN, getaddrinfo failed, ENOTFOUND), never a connection error —
which reads like a broken DNS setup and sends people off debugging
resolv.conf. It is the lockdown working.
Why uid, not a dynamic user¶
meta skuid matches a number. DynamicUser = true or an unpinned
users.users.<n>.uid = null allocates from a pool and can renumber across
rebuilds or across hosts, at which point the table is filtering a uid nobody
runs as — fail-open, with no error anywhere. Both uids in this module are
required to be explicit integers, and they are the module's real interface.
skuid is the kernel uid that owns the socket. bubblewrap here uses
--unshare-all --share-net, so the sandbox does get a user namespace, and a
program can make itself uid 0 inside it. That changes nothing: the socket's
owner in the initial user namespace is still the sandbox uid, so the drop rule
still matches. Namespace root does not buy egress.
Trap 2 — deny CONNECT !SSL_ports and cache deny all¶
A domain allowlist on a CONNECT proxy without a port allowlist is an
arbitrary-port tunnel to every allowlisted host. CONNECT allowed.example:22
is an SSH session; :25 is a mail relay; :9000 is whatever that host also
runs. And a proxy without cache deny all quietly accumulates a copy of
everything the sandbox fetched, on disk, owned by the proxy user.
Both directives are in the generated config, and the ordering matters — the deny comes before the allow:
acl <name>_allowed dstdomain .example.com
acl SSL_ports port 443
acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports
http_access allow CONNECT <name>_allowed
http_access deny all
cache deny all
Verified against a live squid 7.6 from stock nixpkgs (access log, verbatim):
TCP_TUNNEL/200 CONNECT allowed.example:443 <- allowlisted domain, allowed port
TCP_DENIED/403 CONNECT other.example:443 <- not on the allowlist
TCP_DENIED/403 CONNECT allowed.example:22 <- allowlisted domain, WRONG port
TCP_DENIED/403 GET http://allowed.example/ <- plain HTTP through the proxy
That last line is a property worth keeping: because the only allow rule is
allow CONNECT …, non-CONNECT methods are refused outright. The sandbox cannot
speak cleartext HTTP through the proxy at all — everything is an end-to-end TLS
tunnel that squid never terminates. Which in turn means no SSL-bump, no
certificate generation, no CA to install, and the sandbox validates the real
server certificate itself.
Trap 3 — the source tree is read-only and lives outside the writable state¶
The sandboxed program's own code (sourceDir) is bind-mounted read-only,
and it is kept outside stateDir, the one place the sandbox can write.
If the program can write its own source tree — because the checkout sits inside the state directory, or because it was bind-mounted read-write "so it can update itself" — then one compromised run rewrites what the next run executes. The sandbox has bought you a delay, not a boundary: the attacker gets the same credentials, the same allowlist, and now also code execution that survives you noticing.
Writable outputs are punched into the read-only tree as individual mounts:
sourceDir = "/var/lib/agent-src/checkout"; # read-only
stateDir = "/var/lib/agent"; # the only writable place
launcher.stateMounts = {
"/src/out" = "out"; # -> /var/lib/agent/out
"/src/.auth.json" = ".auth.json"; # -> /var/lib/agent/.auth.json
};
The module asserts sourceDir is not under stateDir.
Note bubblewrap's requirement here: --bind of a file needs the target to
already exist on both sides. That is what stateFiles is for — the launcher
touches each one into existence before the run, or bwrap fails with a bare
Can't find source path.
Trap 4 — secrets fail at run time, not at eval time¶
The tempting way to wire a secret into a module like this is a builtins.pathExists
guard on the encrypted file, or an assertion that the decrypted path exists.
Both put a secret's lifecycle on the critical path of the host's evaluation:
forget to git add the new .age file, or rotate a key, and every deploy of
that machine fails at eval — including the deploys that have nothing to do with
this sandbox.
This module never reads secrets at eval time. launcher.secretEnvironment and
launcher.secretFiles take paths, and the generated wrapper checks them at
run time:
[ -r /run/secrets/api-token ] || { echo "missing or unreadable secret: /run/secrets/api-token" >&2; exit 1; }
A missing secret breaks this one command, with a message naming the file. The host still builds, still deploys, still boots.
Prefer secretFiles (bind-mounted read-only into the sandbox) over
secretEnvironment when the program can take a path: secretEnvironment reads
the file with $(cat …), so the value transits the launcher's own argv.
Trap 5 — a bare sudoers command permits any arguments¶
The launcher is meant to be invoked as the sandbox user, typically:
A sudoers entry of the form alice ALL=(agent) NOPASSWD: /nix/store/…/bin/agent
permits that command with any arguments whatsoever. If your launcher
forwards "$@" anywhere — into the sandboxed command, into a --setenv, into
the tool's own CLI — that is an injection surface handed to whoever is on the
sudo.users list.
sudo.forbidArguments (default true) appends the empty-string argument spec,
which is sudoers for "this command, with no arguments at all":
Set it to false only if the launcher genuinely takes arguments and you have
audited what it does with them.
Trap 6 — sharing the host network namespace is not free¶
This sandbox deliberately does not use --unshare-net: it needs to reach
the proxy on the host's loopback. (For total isolation with no allowlist at all,
see network-isolated-editor, which does use
--unshare-net.) Sharing the host netns has two consequences:
- Other loopback services are in reach of the packet filter, and only the
packet filter. Everything on
127.0.0.1— a database, an admin UI, another proxy, a metrics endpoint — is oneconnect()away. Themeta skuid <uid> droprule is what stops it, and every entry you add toextraLoopbackPortsis a hole punched straight through to a host service. Keep it empty. - Abstract-namespace AF_UNIX sockets are NOT filtered. They live in the
network namespace, but nftables cannot match them. A session bus, an
@/tmp/.X11-unix/X0, an agent socket bound to the abstract namespace is reachable by the sandbox regardless of every rule in this module. If the host has abstract sockets that matter, this recipe is not enough on its own — you want a real network namespace plus a socket relay, or a VM.
Trap 7 — the table is fail-open, and flushRuleset will open it¶
The chain is policy accept with explicit per-uid drops, deliberately: this
table narrows exactly two uids and never touches anyone else's traffic, so it
composes with whatever firewall the host already runs. The price is that
if the table is not there, nothing is filtered.
networking.nftables.flushRuleset runs flush ruleset on every start or
reload of nftables.service — including reloads triggered by entirely
unrelated firewall changes. That deletes this module's table, and the sandbox
uid is unfiltered until <name>-egress-lockdown.service next runs. The module
emits a config.warnings entry when it sees that combination. (It defaults on
for hosts with a system.stateVersion older than 23.11, and whenever
networking.nftables.ruleset/rulesetFile is used.)
Check the table is actually loaded before trusting it:
Disabling the module removes its unit from the config entirely, so nothing ever
runs the ExecStop; the table lingers in the kernel until you
nft delete table inet <name>-egress by hand. Harmless (it filters uids that no
longer exist) but worth knowing.
Trap 8 — squid 7 removed dns_v4_first¶
If you are porting an older config: dns_v4_first on is obsolete in squid
7.x. With stock nixpkgs squid 7.6 it produces, on every start:
ERROR: Directive 'dns_v4_first' is obsolete.
dns_v4_first : Remove this line. Squid no longer supports preferential treatment of DNS A records.
It is not fatal — squid starts and serves normally — and, notably, squid -k
parse still exits 0, so upstream's services.squid.validateConfig (which is
exactly squid -k parse -f, nixos/modules/services/networking/squid.nix:19)
does not catch it either. It is just a permanent error line in your cache log.
This module does not emit it; put it in proxy.extraConfig if you are pinned to
squid ≤ 6 and want it.
What upstream nixpkgs does not do¶
-
There is no per-uid egress option in NixOS.
networking.firewallhas no owner/uid match of any kind. The only uid-owner rules in the entire NixOS module set are innixos/modules/services/networking/sslh.nix:246(and its IPv6 twin at 265) — and those go throughnetworking.firewall.iptablesCommands, which the nftables backend hard-rejects (nixos/modules/services/networking/firewall-nftables.nix:65-70assertsextraCommands == ""/extraStopCommands == "").biboumi.nix:225contains a comment sketchingadd rule inet filter output meta skuid biboumi tcp acceptas something the reader might write themselves. That is the whole of upstream's per-uid egress story. -
services.squidcannot be used for this. Three independent blockers, all innixos/modules/services/networking/squid.nix: - It is a singleton — one
services.squid.enable, one unit, hard-coded/run/squid.pidand/var/log/squid. You cannot run a per-sandbox instance, let alone two. - Its unit has no
User=(lines 186–206). Squid starts as root and drops the worker to thesquidaccount internally viacache_effective_user squid squid(line 85). The uid that owns the master process is 0, and "uid 0" is not something a per-uid drop rule can be written around. There is noNoNewPrivileges,ProtectSystem, orProtectHomeeither. - Its default config ends in
http_access allow localnet/http_access allow localhost(lines 95–96), wherelocalnetis all of RFC1918 plus link-local plusfc00::/7(lines 38–43). Out of the box that is an open proxy for your entire LAN.
So this module runs squid directly: ExecStart = squid -f <conf> -N under
User=/Group= of a dedicated account, RuntimeDirectory,
ProtectSystem = "strict", ProtectHome, PrivateTmp, NoNewPrivileges.
-N keeps it in the foreground so systemd supervises the real process;
pid_filename has to point inside the runtime directory, because
ProtectSystem = "strict" makes squid's default /run/squid.pid unwritable.
-
Nothing upstream wires bubblewrap into a NixOS service.
bubblewrapappears in exactly one NixOS module (programs/opengamepadui.nix), as a runtime dependency of a game launcher. -
shutdown_lifetimedefaults to 30 seconds and squid honours it on SIGTERM. Left alone, everysystemctl restartof the proxy — i.e. every deploy — stalls for half a minute. This module defaults it to1 seconds.
Usage¶
{ pkgs, ... }:
{
imports = [ ./per-uid-egress-lockdown ];
services.perUidEgressLockdown = {
enable = true;
name = "agent"; # names the units, the nft table, /run/agent-squid
uid = 60900; # pinned: the nft rules match these numbers
proxyUid = 60901; # MUST differ from uid
allowedDomains = [ ".api.example.com" ];
stateDir = "/var/lib/agent"; # the only writable place
stateSubdirs = [ "out" "tmp" ];
stateFiles = [ ".auth.json" ];
sourceDir = "/var/lib/agent-src/checkout"; # read-only, outside stateDir
launcher = {
command = "exec npm run --silent build";
packages = with pkgs; [ coreutils bash nodejs_22 ];
workingDirectory = "/src";
stateMounts = {
"/src/out" = "out";
"/src/.auth.json" = ".auth.json";
};
environment = {
SSL_CERT_FILE = "/etc/ssl/certs/ca-bundle.crt";
NODE_EXTRA_CA_CERTS = "/etc/ssl/certs/ca-bundle.crt";
};
secretEnvironment.API_TOKEN = "/run/secrets/agent-api-token";
};
sudo.users = [ "alice" ]; # may run `sudo -u agent agent`
};
}
The sandbox starts from --clearenv, so nothing is inherited. TLS-using
programs generally need SSL_CERT_FILE set explicitly (Node additionally wants
NODE_EXTRA_CA_CERTS); the launcher always binds /etc/ssl read-only so the
bundle is there to point at.
Staging read-only inputs¶
When the sandbox needs to read trees that live somewhere awkward — a home
directory, a shared checkout area — mountStage bind-mounts them read-only into
a root-owned directory first, so the sandbox never has to be able to traverse
the parent:
mountStage = {
enable = true;
sources = [ "/srv/mirrors/upstream-a" "/srv/mirrors/upstream-b" ];
};
launcher.stageMountPoint = "/stage"; # -> /stage/upstream-a, /stage/upstream-b
The staging unit is a oneshot with RemainAfterExit; it is idempotent
(mountpoint -q || mount --bind -o ro) and unmounts in preStop. A source that
does not exist is skipped rather than failing the unit.
Options¶
| Option | Default | Meaning |
|---|---|---|
enable |
false |
Turn everything on. |
name |
"sandbox" |
Prefix for units (<name>-squid, <name>-egress-lockdown), the nft table inet <name>-egress, the squid ACL, /run/<name>-squid, and the wrapper binary. |
user / uid |
name / 60900 |
The sandboxed program's account. uid must be pinned. |
proxyUser / proxyUid |
"<name>-proxy" / 60901 |
The proxy's account. Must differ from the above. |
allowedDomains |
[ ] |
squid dstdomain allowlist. Leading dot = domain + subdomains. |
extraLoopbackPorts |
[ ] |
Extra host-loopback TCP ports the sandbox uid may reach. Each one is a hole; see Trap 6. |
proxy.package |
pkgs.squid |
Stock nixpkgs squid is sufficient — no SSL-bump features are used. |
proxy.listenAddress / .port |
"127.0.0.1" / 3128 |
Where the proxy listens. Loopback only. |
proxy.sslPorts |
[ 443 ] |
Ports CONNECT may target. |
proxy.egressPorts |
[ 443 ] |
Ports the proxy uid may reach off-box (nftables). |
proxy.allowDns |
true |
Let the proxy uid do DNS. The sandbox uid never can. |
proxy.accessLog |
stdio:/run/<name>-squid/access.log squid |
tmpfs by default: no persistent request log. stdio:/dev/stdout squid sends it to the journal instead. |
proxy.cacheLog |
/run/<name>-squid/cache.log |
Must be inside the runtime directory. |
proxy.shutdownLifetime |
"1 seconds" |
Upstream default is 30s; see above. |
proxy.extraConfig |
"" |
Extra squid directives, appended verbatim. |
stateDir |
/var/lib/<name> |
The only writable path. |
stateSubdirs / stateFiles |
[ ] |
Created/touched before the run. |
manageDirectories |
true |
Emit the systemd.tmpfiles rules. Off if you provision these via impermanence / a dataset / your own ruleset. |
sourceDir |
null |
Read-only source tree. Asserted to be outside stateDir. |
mountStage.{enable,dir,sources,unitName} |
off | Root-owned read-only bind-mount stage. |
launcher.enable |
true |
Build and install the bubblewrap wrapper. |
launcher.package |
null |
Escape hatch: bring your own launcher (see below). |
launcher.binName |
name |
Command name on PATH. |
launcher.command |
"" |
Shell run inside the sandbox (bash -c). |
launcher.packages |
[ ] |
Makes up PATH inside the sandbox. |
launcher.sourceMountPoint / .stageMountPoint / .homeMountPoint |
/src / /stage / /state |
Where things appear inside. |
launcher.workingDirectory |
"/" |
--chdir. |
launcher.stateMounts |
{ } |
in-sandbox path -> stateDir-relative path, read-write. |
launcher.roMounts |
{ } |
in-sandbox path -> host path, read-only. |
launcher.environment |
{ } |
Env inside the sandbox (starts from --clearenv). |
launcher.secretEnvironment |
{ } |
VAR -> file, read at run time. |
launcher.secretFiles |
{ } |
in-sandbox path -> host secret file, bind-mounted read-only. |
sudo.users |
[ ] |
Users who may run the launcher as user, NOPASSWD. |
sudo.forbidArguments |
true |
Append "" to the sudoers command spec; see Trap 5. |
launcher.package — bringing your own¶
The generated launcher covers the common shape, but the inner command of a
real workload often needs setup the option surface does not express. Setting
launcher.package to your own derivation (providing bin/<binName>) keeps
everything else — the two accounts, the nftables table, the proxy, the staging
mounts, the sudo rule — and replaces only the wrapper.
The kernel lockdown does not care which wrapper you use: it is bound to the
uid, not to this module's script. What you take on is reproducing the sandbox
properties yourself — read-only source, --clearenv plus the proxy variables,
no argument passthrough.
Firewall backend¶
This module does not use networking.firewall.extraCommands /
extraStopCommands, so it works on both the iptables and the nftables
backend without an assertion. It owns a self-contained inet <name>-egress
table applied by its own oneshot unit (nft -f <ruleset>, one atomic netlink
transaction), and it stays out of networking.nftables.tables — which deletes
and recreates every table it declares on each reload. See
egress-filter for the same finding in a
per-interface (rather than per-uid) filter, and for what to do when the
allowlist has to be expressed as IPs instead of a CONNECT target.
Ordering: <name>-egress-lockdown.service is after = [ "firewall.service" ]
and wantedBy = [ "multi-user.target" ]. It is a oneshot with
RemainAfterExit, so a switch that does not restart it leaves the running
table alone.
Test¶
test.nix is a four-node NixOS VM test built on
nixos-test-topology. Run it with:
or, from a flake, pkgs.callPackage ./modules/per-uid-egress-lockdown/test.nix {}.
It boots a confined host, a router, an allowlisted origin and a non-allowlisted origin. The two origins sit on a different subnet from the confined host, so every packet toward them has to transit the router — on one subnet the client would ARP the origin directly, the router would never see the traffic, and the filtering assertions would be tautologies.
What it asserts, and why none of it can be satisfied by accident:
| # | claim | how it is instrumented |
|---|---|---|
| 0 | uid == proxyUid is rejected |
eval-time; the module's own assertion must fire |
| 1 | confined uid reaches an allowlisted destination via the proxy | body matches, squid logged the CONNECT, the router forwarded packets, and the proxy uid — not the confined one — was the sender |
| 2 | confined uid cannot reach a non-allowlisted destination via the proxy | squid logs TCP_DENIED, and the router forwarded zero packets toward it |
| 3 | confined uid cannot bypass the proxy | direct curl --noproxy '*' to the destination IP hangs and times out |
| 4 | a different uid is unaffected | root and an ordinary uid run the identical command against the identical address and succeed, including to the non-allowlisted origin |
| 5 | the shipped bubblewrap launcher does 1–3 in one real run | reads what the sandboxed program wrote to its state dir |
| 6 | confined uid cannot resolve names either | DNS is the proxy uid's privilege |
Subtest 3 is the one worth copying. curl failing is satisfied by a typo in an
address, a missing route, or a dead server, so the failure alone proves nothing.
It is pinned down by two counters on opposite sides of the boundary: an
output-hook counter on the confined host at priority -300 (i.e. before the
module's chain at priority 0), matching meta skuid <confined uid>, and the
topology library's forward-hook counter on the router. The subtest requires
the first to be non-zero — the process really did emit a SYN at the
destination — and the second to be zero — nothing left the host. A drop that
is not in the packet path fails one or the other.
That is not theoretical. Mutating the module's chain from
hook output to hook forward — leaving a table that still loads and still
reads meta skuid <uid> drop in nft list table — keeps subtests 1, 2 and the
"table is loaded" check green and fails subtest 3 with
the confined uid reached the origin directly: 'ALLOWED-ORIGIN-PAYLOAD'.
Removing the module's uid != proxyUid assertion fails the test at eval;
nixpkgs alone only emits a Duplicate uid trace for that configuration, so
the module's assertion is the only thing that stops it.
On the "the proxy must run as a different uid" corollary: that is caught at eval
(subtest 0), before a VM boots. If the assertion did not exist, the runtime test
would still fail — with a shared uid the sandbox's drop rule matches squid's
own egress and subtest 1 fails — but it would fail as "the proxy is broken",
not as "the lockdown is void". The eval check is what names the actual fault.
Related recipes¶
network-isolated-editor— the all-or-nothing sibling: bubblewrap with--unshare-netand no proxy at all. Use it when the answer is "no network", not "these domains".egress-filter— domain allowlist for a whole interface (VM bridge, container network), enforced as a live IP set.nixos-hardening-tiers— host-level tiers; note its knobs and this module'ssystemdhardening are independent decisions.
Caveats¶
dstdomainis only as tight as the domain..example.commatches every subdomain. If an allowlisted host runs an open redirect, a proxy endpoint, or user-controlled subdomains, egress is effectively wider than the list reads.- The proxy sees hostnames, not content. CONNECT tunnels are opaque; there is no inspection of what flows through an allowed tunnel, by design (see the no-SSL-bump note above). This bounds where data can go, not what goes there.
- No egress accounting or rate limiting. Add
delay_poolsviaproxy.extraConfigif you need it. - The access log is on tmpfs by default — it does not survive a reboot.
That is deliberate (no long-term record of what the sandbox fetched) but it
also means no forensic trail; set
proxy.accessLog = "stdio:/dev/stdout squid"to route it to the journal. - Nothing here confines the filesystem beyond the bind mounts you declare.
/nix/storeis bound read-only in full, as it must be for anything to run. - The
stateDiris shared across runs. Two concurrent invocations of the launcher write to the same directory; the module does not lock. Wrap it in a systemd unit or a flock if that matters.
Source¶
modules/per-uid-egress-lockdown/default.nix
# per-uid-egress-lockdown
#
# Run an untrusted program as its own uid, inside bubblewrap, with NO network
# except a loopback CONNECT proxy that enforces a domain allowlist.
#
# The allowlist is enforced TWICE and the second one is the real one:
#
# 1. squid `http_access allow CONNECT <allowlist>` — a policy the program
# only obeys while it honours HTTPS_PROXY. Advisory.
# 2. an nftables `output` chain that drops EVERY packet owned by the sandbox
# uid except loopback traffic to the proxy port. Kernel-enforced; a
# program that ignores the proxy env vars gets ENETUNREACH, not the
# internet.
#
# Corollary, and the single easiest way to silently undo all of this: the proxy
# MUST run as a DIFFERENT uid. Same uid => the proxy's own outbound packets are
# matched by the sandbox's drop rule (proxy dies), and every rule you add to fix
# that reopens the internet for the sandboxed program. The module asserts this.
#
# See README.md for the full trap list.
{
config,
lib,
pkgs,
...
}:
let
cfg = config.services.perUidEgressLockdown;
inherit (lib)
mkOption
mkEnableOption
mkIf
types
optionals
optionalString
concatStringsSep
concatMapStringsSep
mapAttrsToList
escapeShellArg
;
name = cfg.name;
tableName = "${name}-egress";
aclName = "${name}_allowed";
proxyRuntimeDir = "${name}-squid";
proxyRunPath = "/run/${proxyRuntimeDir}";
# ---------------------------------------------------------------- proxy ---
# Deny-by-default. Only CONNECT, only to SSL_ports, only to allowlisted
# domains. `cache deny all` keeps squid from retaining anything on disk.
squidConf = pkgs.writeText "${name}-squid.conf" (
''
http_port ${cfg.proxy.listenAddress}:${toString cfg.proxy.port}
acl ${aclName} dstdomain ${concatStringsSep " " cfg.allowedDomains}
acl SSL_ports port ${concatMapStringsSep " " toString cfg.proxy.sslPorts}
acl CONNECT method CONNECT
http_access deny CONNECT !SSL_ports
http_access allow CONNECT ${aclName}
http_access deny all
cache deny all
access_log ${cfg.proxy.accessLog}
cache_log ${cfg.proxy.cacheLog}
pid_filename ${proxyRunPath}/squid.pid
shutdown_lifetime ${cfg.proxy.shutdownLifetime}
''
+ optionalString (cfg.proxy.extraConfig != "") (cfg.proxy.extraConfig + "\n")
);
# ------------------------------------------------------------- lockdown ---
# `policy accept` with explicit per-uid drops: this table only ever narrows
# the two uids it names and never touches anybody else's traffic.
#
# `meta skuid` is the KERNEL uid that owns the socket. A process that made
# itself "root" inside a user namespace is still the sandbox uid here.
ruleLines =
[
''meta skuid ${toString cfg.uid} oifname "lo" ct state established,related accept''
''meta skuid ${toString cfg.uid} oifname "lo" tcp dport ${toString cfg.proxy.port} accept''
]
++ map (p: ''meta skuid ${toString cfg.uid} oifname "lo" tcp dport ${toString p} accept'') cfg.extraLoopbackPorts
++ [
"meta skuid ${toString cfg.uid} drop"
""
"meta skuid ${toString cfg.proxyUid} ct state established,related accept"
]
++ optionals cfg.proxy.allowDns [
"meta skuid ${toString cfg.proxyUid} udp dport 53 accept"
"meta skuid ${toString cfg.proxyUid} tcp dport 53 accept"
]
++ map (p: "meta skuid ${toString cfg.proxyUid} tcp dport ${toString p} accept") cfg.proxy.egressPorts
++ [
''meta skuid ${toString cfg.proxyUid} oifname "lo" accept''
"meta skuid ${toString cfg.proxyUid} drop"
];
ruleBody = concatStringsSep "\n" (map (l: if l == "" then "" else " " + l) ruleLines);
# Written as `table` / `delete table` / `table { … }` so a re-run replaces the
# table atomically. The bare `table inet <name>` first line exists only so the
# `delete` cannot fail on a fresh boot where the table does not exist yet.
egressRules = pkgs.writeText "${name}-egress.nft" ''
table inet ${tableName}
delete table inet ${tableName}
table inet ${tableName} {
chain output {
type filter hook output priority 0; policy accept;
${ruleBody}
}
}
'';
# ------------------------------------------------------------- launcher ---
sandboxPath = lib.makeBinPath cfg.launcher.packages;
bindArg = flag: host: dest: "${flag} ${host} ${dest}";
roMountArgs = mapAttrsToList (dest: host: bindArg "--ro-bind" (escapeShellArg host) dest) cfg.launcher.roMounts;
stateMountArgs = mapAttrsToList (
dest: sub:
bindArg "--bind" (if (sub == "." || sub == "") then ''"$STATE"'' else ''"$STATE"/${sub}'') dest
) cfg.launcher.stateMounts;
envArgs = mapAttrsToList (k: v: "--setenv ${k} ${escapeShellArg v}") cfg.launcher.environment;
secretEnvArgs = mapAttrsToList (
k: file: ''--setenv ${k} "$(cat ${escapeShellArg file})"''
) cfg.launcher.secretEnvironment;
secretFileArgs = mapAttrsToList (
dest: file: "--ro-bind ${escapeShellArg file} ${dest}"
) cfg.launcher.secretFiles;
# Readability of every secret is checked at RUNTIME, not at eval. A missing
# secret must fail this one wrapper, never the whole host's evaluation.
secretChecks = mapAttrsToList (
_: file: ''[ -r ${escapeShellArg file} ] || { echo "missing or unreadable secret: ${file}" >&2; exit 1; }''
) (cfg.launcher.secretEnvironment // cfg.launcher.secretFiles);
stageBindArgs = optionalString cfg.mountStage.enable ''
for d in "$STAGE"/*/; do
[ -d "$d" ] || continue
stage_binds+=(--ro-bind "$d" "${cfg.launcher.stageMountPoint}/$(basename "$d")")
done
'';
# Assembled as a LIST and joined, never as a here-doc with interpolated
# optional lines. An optional line that expands to "" leaves a whitespace-only
# line with no trailing backslash, which ENDS the `exec bwrap` command: the
# next `--…` line then becomes a command name. With the default
# `sourceDir = null` + `mountStage.enable = false` that happened twice.
bwrapArgs =
[
"--unshare-all --share-net"
"--die-with-parent --new-session"
"--clearenv"
"--setenv PATH ${escapeShellArg sandboxPath}"
"--setenv HOME ${cfg.launcher.homeMountPoint}"
''--setenv HTTPS_PROXY "$PROXY"''
''--setenv HTTP_PROXY "$PROXY"''
''--setenv NO_PROXY ""''
]
++ envArgs
++ secretEnvArgs
++ [
"--proc /proc --dev /dev --tmpfs /tmp"
"--ro-bind /nix/store /nix/store"
"--ro-bind /etc/ssl /etc/ssl"
"--ro-bind /etc/resolv.conf /etc/resolv.conf"
]
++ roMountArgs
++ secretFileArgs
++ optionals (cfg.sourceDir != null) [
''--ro-bind "$SRC" ${cfg.launcher.sourceMountPoint}''
]
++ stateMountArgs
++ optionals cfg.mountStage.enable [ ''"''${stage_binds[@]}"'' ]
++ [
"--chdir ${cfg.launcher.workingDirectory}"
"-- ${pkgs.bash}/bin/bash -c ${escapeShellArg cfg.launcher.command}"
];
defaultLauncher = pkgs.writeShellApplication {
name = cfg.launcher.binName;
runtimeInputs = [
pkgs.bubblewrap
pkgs.coreutils
];
excludeShellChecks = [ "SC2016" ];
text = ''
set -euo pipefail
STATE=${escapeShellArg cfg.stateDir}
${optionalString (cfg.sourceDir != null) "SRC=${escapeShellArg cfg.sourceDir}"}
${optionalString cfg.mountStage.enable "STAGE=${escapeShellArg cfg.mountStage.dir}"}
PROXY="http://${cfg.proxy.listenAddress}:${toString cfg.proxy.port}"
${optionalString (
cfg.sourceDir != null
) ''[ -d "$SRC" ] || { echo "source tree not found at $SRC" >&2; exit 1; }''}
${optionalString cfg.mountStage.enable ''
[ -d "$STAGE" ] || { echo "mount stage $STAGE is not present (${cfg.mountStage.unitName})" >&2; exit 1; }''}
${concatStringsSep "\n" secretChecks}
${concatMapStringsSep "\n" (d: ''mkdir -p "$STATE"/${d}'') cfg.stateSubdirs}
${concatMapStringsSep "\n" (f: ''[ -f "$STATE"/${f} ] || : > "$STATE"/${f}'') cfg.stateFiles}
${optionalString cfg.mountStage.enable "stage_binds=()"}
${stageBindArgs}
exec bwrap \
${concatStringsSep " \\\n " bwrapArgs}
'';
};
launcherPkg = if cfg.launcher.package != null then cfg.launcher.package else defaultLauncher;
launcherBin = "${launcherPkg}/bin/${cfg.launcher.binName}";
in
{
options.services.perUidEgressLockdown = {
enable = mkEnableOption "per-uid kernel egress lockdown with a co-resident allowlist proxy";
name = mkOption {
type = types.str;
default = "sandbox";
description = ''
Prefix for everything this module names: the systemd units
(`<name>-squid`, `<name>-egress-lockdown`), the nftables table
(`inet <name>-egress`), the squid ACL, and `/run/<name>-squid`.
'';
};
user = mkOption {
type = types.str;
default = cfg.name;
defaultText = lib.literalExpression "config.services.perUidEgressLockdown.name";
description = "Unix user the untrusted program runs as.";
};
uid = mkOption {
type = types.int;
default = 60900;
description = ''
STABLE uid for the sandboxed program. The nftables rules match on this
number, so it must be pinned — an allocated (`uid = null`) system user
can renumber and the lockdown would then be filtering a uid nobody uses.
'';
};
proxyUser = mkOption {
type = types.str;
default = "${cfg.name}-proxy";
defaultText = lib.literalExpression ''"''${config.services.perUidEgressLockdown.name}-proxy"'';
description = "Unix user the allowlist proxy runs as. MUST NOT be `user`.";
};
proxyUid = mkOption {
type = types.int;
default = 60901;
description = ''
STABLE uid for the proxy. Must differ from `uid`: the proxy is the one
process allowed to leave the box, and it is separated from the
sandboxed program precisely so the sandbox's drop rule cannot be
loosened to keep the proxy alive.
'';
};
allowedDomains = mkOption {
type = types.listOf types.str;
default = [ ];
example = [
".example.com"
"api.example.org"
];
description = ''
squid `dstdomain` allowlist. A leading dot matches the domain and all
of its subdomains; without it the match is exact. Empty means the
sandbox can reach nothing at all (still a valid, if useless, config).
'';
};
extraLoopbackPorts = mkOption {
type = types.listOf types.port;
default = [ ];
description = ''
Extra loopback TCP ports the sandbox uid may reach, on top of the proxy
port. Every entry is a hole: anything listening on 127.0.0.1 of the
HOST is reachable, because the sandbox shares the host network
namespace. Prefer keeping this empty.
'';
};
proxy = {
package = lib.mkPackageOption pkgs "squid" { };
listenAddress = mkOption {
type = types.str;
default = "127.0.0.1";
description = ''
Loopback only. Binding this anywhere else publishes an open-ish proxy
to the network; the lockdown table does not protect other hosts.
'';
};
port = mkOption {
type = types.port;
default = 3128;
description = "Proxy port on `listenAddress`.";
};
sslPorts = mkOption {
type = types.listOf types.port;
default = [ 443 ];
description = ''
Ports CONNECT may target. Anything not listed is refused by
`http_access deny CONNECT !SSL_ports` — without which the proxy is a
general-purpose TCP tunnel to any port on an allowlisted host.
'';
};
egressPorts = mkOption {
type = types.listOf types.port;
default = [ 443 ];
description = ''
TCP ports the PROXY uid may reach off-box, enforced by nftables.
Normally the same set as `sslPorts`.
'';
};
allowDns = mkOption {
type = types.bool;
default = true;
description = ''
Let the proxy uid do DNS (53/udp + 53/tcp). The sandbox uid never
can — all name resolution happens inside the proxy, as part of
CONNECT.
'';
};
accessLog = mkOption {
type = types.str;
default = "stdio:${proxyRunPath}/access.log squid";
defaultText = lib.literalExpression ''"stdio:/run/''${name}-squid/access.log squid"'';
description = ''
Full squid `access_log` argument. `stdio:` avoids squid's
`log_file_daemon` helper. The default lands in a tmpfs, so the
request log does not persist across reboots — set
`stdio:/dev/stdout squid` to send it to the journal instead.
'';
};
cacheLog = mkOption {
type = types.str;
default = "${proxyRunPath}/cache.log";
defaultText = lib.literalExpression ''"/run/''${name}-squid/cache.log"'';
description = "squid `cache_log` path. Must be inside the runtime directory.";
};
shutdownLifetime = mkOption {
type = types.str;
default = "1 seconds";
description = ''
squid's `shutdown_lifetime`. The upstream default is 30 seconds and
squid honours it on SIGTERM, so every restart of this unit stalls for
half a minute unless it is lowered.
'';
};
extraConfig = mkOption {
type = types.lines;
default = "";
description = "Extra squid directives, appended verbatim.";
};
};
stateDir = mkOption {
type = types.str;
default = "/var/lib/${cfg.name}";
defaultText = lib.literalExpression ''"/var/lib/''${config.services.perUidEgressLockdown.name}"'';
description = ''
The ONLY writable place the sandboxed program has. Must not contain the
program's own source tree — see `sourceDir`.
'';
};
stateSubdirs = mkOption {
type = types.listOf types.str;
default = [ ];
example = [
"out"
"tmp"
];
description = "Subdirectories of `stateDir` to create (owned by `user`).";
};
stateFiles = mkOption {
type = types.listOf types.str;
default = [ ];
example = [ ".auth.json" ];
description = ''
Files under `stateDir` the launcher touches into existence before the
run. bubblewrap's `--bind` of a file requires the target to already
exist on both sides.
'';
};
manageDirectories = mkOption {
type = types.bool;
default = true;
description = ''
Emit `systemd.tmpfiles` rules for `stateDir` (+ `stateSubdirs`), the
mount stage, and the proxy runtime directory. Turn off if the adopter
provisions these another way (impermanence, a ZFS dataset, an existing
tmpfiles ruleset).
'';
};
sourceDir = mkOption {
type = types.nullOr types.str;
default = null;
example = "/var/lib/agent-src/checkout";
description = ''
The sandboxed program's own code, bind-mounted READ-ONLY. Keep it
OUTSIDE `stateDir`: if the program can write its own source tree, a
single compromised run rewrites what the next run executes, and the
sandbox buys you nothing but a delay.
'';
};
mountStage = {
enable = mkEnableOption "a root-owned stage of read-only bind mounts";
dir = mkOption {
type = types.str;
default = "/var/lib/${cfg.name}-stage";
defaultText = lib.literalExpression ''"/var/lib/''${config.services.perUidEgressLockdown.name}-stage"'';
description = "Root-owned directory the read-only mirrors are mounted under.";
};
sources = mkOption {
type = types.listOf types.str;
default = [ ];
description = ''
Host directories to bind-mount read-only into `dir`, each under its
own basename. Staging them means the sandbox never has to traverse
the parent directory (a home directory, a shared tree) to reach them.
'';
};
unitName = mkOption {
type = types.str;
default = "${cfg.name}-stage";
defaultText = lib.literalExpression ''"''${config.services.perUidEgressLockdown.name}-stage"'';
description = "systemd unit name for the staging mounts.";
};
};
launcher = {
enable = mkOption {
type = types.bool;
default = true;
description = "Install a wrapper that runs `command` inside bubblewrap.";
};
package = mkOption {
type = types.nullOr types.package;
default = null;
description = ''
Escape hatch: supply your own launcher package instead of the
generated one. It must provide `bin/<binName>`, and it is on you to
reproduce the sandbox properties (read-only source, proxy env, no
argument passthrough). The kernel lockdown still applies regardless,
because it is bound to the uid, not to this wrapper.
'';
};
binName = mkOption {
type = types.str;
default = cfg.name;
defaultText = lib.literalExpression "config.services.perUidEgressLockdown.name";
description = "Command name installed on PATH.";
};
command = mkOption {
type = types.str;
default = "";
example = "exec ./run.sh";
description = "Shell run inside the sandbox, as `bash -c`.";
};
packages = mkOption {
type = types.listOf types.package;
default = [ ];
description = "Packages whose `bin` directories make up PATH inside the sandbox.";
};
sourceMountPoint = mkOption {
type = types.str;
default = "/src";
description = "Where `sourceDir` appears inside the sandbox (read-only).";
};
stageMountPoint = mkOption {
type = types.str;
default = "/stage";
description = "Where the mount stage's entries appear inside the sandbox.";
};
homeMountPoint = mkOption {
type = types.str;
default = "/state";
description = "Value of `HOME` inside the sandbox.";
};
workingDirectory = mkOption {
type = types.str;
default = "/";
description = "`--chdir` for the sandboxed command.";
};
stateMounts = mkOption {
type = types.attrsOf types.str;
default = { };
example = {
"/src/out" = "out";
"/state" = ".";
};
description = ''
Writable mounts: in-sandbox path -> path relative to `stateDir`.
This is how a read-only source tree gets writable output
subdirectories punched into it without making the tree writable.
'';
};
roMounts = mkOption {
type = types.attrsOf types.str;
default = { };
example = {
"/etc/hosts" = "/etc/hosts";
};
description = "Extra read-only mounts: in-sandbox path -> host path.";
};
environment = mkOption {
type = types.attrsOf types.str;
default = { };
description = ''
Environment inside the sandbox. The sandbox starts from `--clearenv`,
so nothing is inherited: TLS-using programs generally need
`SSL_CERT_FILE` (and Node additionally `NODE_EXTRA_CA_CERTS`) set to
`/etc/ssl/certs/ca-bundle.crt`.
'';
};
secretEnvironment = mkOption {
type = types.attrsOf types.str;
default = { };
example = {
API_TOKEN = "/run/secrets/api-token";
};
description = ''
VAR -> file. The file is read at RUN time and passed with `--setenv`,
so a missing secret fails this command, not the host's evaluation.
The value transits `bwrap`'s argv; prefer `secretFiles` when the
program can read a path.
'';
};
secretFiles = mkOption {
type = types.attrsOf types.str;
default = { };
example = {
"/run/token" = "/run/secrets/api-token";
};
description = "in-sandbox path -> host secret file, bind-mounted read-only.";
};
};
sudo = {
users = mkOption {
type = types.listOf types.str;
default = [ ];
description = "Users allowed to run the launcher as `user` with NOPASSWD.";
};
forbidArguments = mkOption {
type = types.bool;
default = true;
description = ''
Append `""` to the sudoers command spec, which means "this command
with NO arguments". A bare command in sudoers permits ANY arguments,
so without this the sudo rule is an argument-injection surface into
whatever the launcher forwards.
'';
};
};
};
config = mkIf cfg.enable {
assertions = [
{
assertion = cfg.uid != cfg.proxyUid;
message = ''
services.perUidEgressLockdown: uid (${toString cfg.uid}) and proxyUid must differ.
Running the proxy as the sandboxed uid means the sandbox's drop rule kills
the proxy, and every rule that revives it hands the internet back to the
sandboxed program.
'';
}
{
assertion = cfg.user != cfg.proxyUser;
message = "services.perUidEgressLockdown: user and proxyUser must differ.";
}
{
assertion = cfg.sourceDir == null || !(lib.hasPrefix (cfg.stateDir + "/") cfg.sourceDir);
message = ''
services.perUidEgressLockdown: sourceDir (${toString cfg.sourceDir}) is inside
stateDir (${cfg.stateDir}), which the sandbox can write. Keep the code the
sandbox runs outside everything the sandbox can modify.
'';
}
{
assertion = cfg.launcher.package == null -> !cfg.launcher.enable || cfg.launcher.command != "";
message = "services.perUidEgressLockdown: launcher.command is empty and no launcher.package was supplied.";
}
];
warnings =
lib.optional (config.networking.nftables.enable && config.networking.nftables.flushRuleset) ''
services.perUidEgressLockdown: networking.nftables.flushRuleset is on. Every
start or reload of nftables.service runs `flush ruleset`, deleting the
inet ${tableName} table until ${name}-egress-lockdown.service next runs.
The sandbox uid is UNFILTERED (fail-open) in that window.
''
++ lib.optional (cfg.allowedDomains == [ ]) ''
services.perUidEgressLockdown: allowedDomains is empty; the sandbox has no
reachable destination at all.
'';
users.groups.${cfg.user} = { };
users.groups.${cfg.proxyUser} = { };
users.users.${cfg.user} = {
uid = cfg.uid;
group = cfg.user;
isSystemUser = true;
home = cfg.stateDir;
createHome = true;
description = "${name} sandbox (no direct egress)";
};
users.users.${cfg.proxyUser} = {
uid = cfg.proxyUid;
group = cfg.proxyUser;
isSystemUser = true;
description = "${name} egress-allowlist proxy";
};
systemd.tmpfiles.rules = mkIf cfg.manageDirectories (
[
"d ${cfg.stateDir} 0750 ${cfg.user} ${cfg.user} - -"
]
++ map (d: "d ${cfg.stateDir}/${d} 0750 ${cfg.user} ${cfg.user} - -") cfg.stateSubdirs
++ optionals cfg.mountStage.enable [
"d ${cfg.mountStage.dir} 0755 root root - -"
]
++ [
"d ${proxyRunPath} 0755 ${cfg.proxyUser} ${cfg.proxyUser} - -"
]
);
# NOTE the shape: `systemd.services.<n> = mkIf false { … }` still creates
# the ATTRIBUTE, and an attrsOf-submodule then materialises an empty unit
# with that name. The mkIf has to sit on the attrset, not on the unit.
systemd.services = lib.mkMerge [
(mkIf cfg.mountStage.enable {
${cfg.mountStage.unitName} = {
description = "read-only bind mounts for the ${name} sandbox stage";
wantedBy = [ "multi-user.target" ];
before = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
script = concatStringsSep "\n" (
map (
d:
let
base = baseNameOf d;
in
''
if [ -d ${escapeShellArg d} ]; then
mkdir -p ${cfg.mountStage.dir}/${base}
${pkgs.util-linux}/bin/mountpoint -q ${cfg.mountStage.dir}/${base} \
|| ${pkgs.util-linux}/bin/mount --bind -o ro ${escapeShellArg d} ${cfg.mountStage.dir}/${base}
fi
''
) cfg.mountStage.sources
);
preStop = concatStringsSep "\n" (
map (
d: "${pkgs.util-linux}/bin/umount ${cfg.mountStage.dir}/${baseNameOf d} || true"
) cfg.mountStage.sources
);
};
})
{
"${name}-squid" = {
description = "${name} egress-allowlist proxy (CONNECT to allowlisted domains only)";
wantedBy = [ "multi-user.target" ];
after = [
"network.target"
"nss-lookup.target"
];
serviceConfig = {
User = cfg.proxyUser;
Group = cfg.proxyUser;
RuntimeDirectory = proxyRuntimeDir;
ExecStart = "${cfg.proxy.package}/bin/squid -f ${squidConf} -N";
Restart = "on-failure";
NoNewPrivileges = true;
ProtectSystem = "strict";
ProtectHome = true;
PrivateTmp = true;
ReadWritePaths = [ proxyRunPath ];
};
};
"${name}-egress-lockdown" = {
description = "per-uid egress lockdown for the ${name} sandbox (nft table)";
wantedBy = [ "multi-user.target" ];
after = [ "firewall.service" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = "${pkgs.nftables}/bin/nft -f ${egressRules}";
ExecStop = "${pkgs.nftables}/bin/nft delete table inet ${tableName}";
};
};
}
];
environment.systemPackages = mkIf cfg.launcher.enable [ launcherPkg ];
security.sudo.extraRules = mkIf (cfg.launcher.enable && cfg.sudo.users != [ ]) [
{
users = cfg.sudo.users;
runAs = cfg.user;
commands = [
{
command = launcherBin + optionalString cfg.sudo.forbidArguments " \"\"";
options = [ "NOPASSWD" ];
}
];
}
];
};
}