spool-dir-credential-broker¶

Modules

Keep a bearer token out of unprivileged sandboxes. Producers only drop JSON manifests into a shared, sticky spool directory; one small hardened watcher holds the credential and is the sole process that forwards those manifests to an upstream REST API.

The problem¶

You have a bunch of unprivileged, possibly sandboxed processes (CI jobs, per-user agent sessions, ephemeral containers) that each need to register something with a central API — say, announce a session so it can be routed to. The API is authenticated with a bearer token.

The naive approach hands that token to every producer. Now the secret lives in every sandbox: any one of them can read it, exfiltrate it, or call the API with privileges far beyond "register my own session." And rotating it means touching every producer.

The design¶

Invert it. The producers never see the token. Instead:

unprivileged producers ──drop <id>.json──▶  spool dir (1777, sticky)
                                                  │  inotify
                                                  ▼
                                       broker unit (holds token)
                                                  │  Bearer <token>
                                                  ▼
                                          upstream REST API

Producers write a manifest <id>.json into a sticky 1777 spool directory and delete it when done. That is their entire interface. They hold no credential and make no network calls.
The broker is one hardened systemd unit running as a dedicated user whose only privilege is read access to the token. It watches the spool with inotify and translates filesystem events into authenticated API calls:
a new/written <id>.json → POST <createPath> with the manifest body
a removed <id>.json → DELETE <deletePath>/<id>

The blast radius of a compromised producer is now "write a JSON file"; the token lives in exactly one confined place.

Key insights / traps¶

The token is re-read on every request¶

The broker reads tokenFile fresh for each API call, not once at startup. So when your secret manager rotates the token on disk, the very next forwarded manifest uses the new value — no restart, no reload. This is the single most important detail; a naive implementation that caches the token in a variable would silently start failing after every rotation.

The spool must be sticky (1777) — and this module does not create it¶

The whole security story depends on the spool being world-writable with the sticky bit, so any local process can drop its own manifest but cannot delete or overwrite another producer's. Provision it yourself, e.g.:

systemd.tmpfiles.rules = [
  "d /var/lib/spool-broker/inbox 1777 root root -"
];

The broker unit orders itself after = [ "systemd-tmpfiles-setup.service" ] so the directory exists before it starts.

The delete id comes from the filename, not the file¶

On delete the manifest content is already gone, so the resource id is recovered from the filename stem (<id>.json → <id>). Producers must name the file after the same id they put in the idField of the JSON, or teardown will target the wrong resource. Create and delete are deliberately keyed the same way.

Startup reconciliation¶

Before it starts watching, the broker sweeps every *.json already in the spool and re-forwards it. This means a broker restart re-announces manifests that were dropped while it was down, instead of losing them. It also means your upstream POST handler should be idempotent for an id that already exists.

inotify events cover both write styles¶

It listens for close_write and moved_to: close_write catches producers that write in place, moved_to catches the safer write-to-temp-then- rename(2) pattern (which never exposes a half-written manifest). delete and moved_from both trigger teardown. Dotfiles and non-.json names are ignored.

Hardening¶

The unit runs with ProtectSystem=strict, ProtectHome, PrivateTmp, NoNewPrivileges, the token mounted ReadOnlyPaths and the spool the only ReadWritePaths. The point: this process holds the one real credential in the system, so it should be able to touch nothing else. Run it as a dedicated low-privilege user.

The bearer token is never placed on curl's command line. Process arguments are readable via /proc/<pid>/cmdline by every local user on a stock host, so a token passed as -H "Authorization: Bearer …" would be scrapable by the very unprivileged producers this module is meant to confine. Instead the Authorization header is piped to curl as a config on stdin (curl -K -), so the secret never shows up in the process arguments. If you adapt the api function for a different upstream, keep the token off argv the same way.

Usage¶

{
  imports = [ ./spool-dir-credential-broker ];

  # You create the sticky spool dir:
  systemd.tmpfiles.rules = [
    "d /var/lib/spool-broker/inbox 1777 root root -"
  ];

  users.users.spool-broker = {
    isSystemUser = true;
    group = "spool-broker";
  };
  users.groups.spool-broker = { };

  services.spoolCredentialBroker = {
    enable = true;
    spoolDir = "/var/lib/spool-broker/inbox";
    tokenFile = "/run/secrets/upstream-token"; # readable by the user below
    upstreamUrl = "https://api.example.com";
    user = "spool-broker";
    # optional, defaults shown:
    # createPath = "/api/sessions";
    # deletePath = "/api/sessions";
    # idField    = "slug";
  };
}

A producer registers itself by writing (atomically, ideally):

tmp=$(mktemp)
printf '{"slug":"job-42","cmd":"..."}' > "$tmp"
mv "$tmp" /var/lib/spool-broker/inbox/job-42.json

and unregisters by removing job-42.json.

Options¶

Option	Default	Meaning
`enable`	`false`	Turn the broker on.
`spoolDir`	`/var/lib/spool-broker/inbox`	Sticky dir producers drop manifests into (create it yourself).
`tokenFile`	(required)	File holding the bearer token; readable only by `user`.
`upstreamUrl`	(required)	Base URL of the upstream REST API.
`createPath`	`/api/sessions`	Path POSTed to with the manifest body on create.
`deletePath`	`/api/sessions`	Base path DELETEd as `<deletePath>/<id>` on remove.
`idField`	`slug`	JSON field carrying the id (also the filename stem).
`user`	(required)	Dedicated user the broker runs as.
`group`	`= user`	Group the broker runs as.

Caveats¶

The upstream contract here is a simple POST create / DELETE by id REST shape with the id in a JSON field. If your API differs (different verbs, id in the URL for create, envelope format), adapt the api/handle_* shell functions — the pattern (spool → confined watcher → authenticated forward) is what's reusable, not the exact endpoints.
There is no backpressure or retry queue: a manifest that fails to forward is logged and dropped until the next inotify event or restart. If you need at-least-once delivery, add a retry/spool-of-failures on top.
Producers can spoof each other's ids (they share the dir). The sticky bit stops them deleting each other's files, but not writing a manifest claiming someone else's id. Trust boundary is "local processes"; if that's too broad, give each producer its own drop dir.

Source¶

modules/spool-dir-credential-broker/default.nix

# spool-dir-credential-broker — keep a bearer token out of unprivileged
# sandboxes by putting one hardened watcher between them and a REST API.
#
# Unprivileged producers (sandboxed jobs, agents, user sessions) only ever
# drop a JSON manifest into a shared, sticky 1777 spool directory. They never
# see the credential. A single hardened, credential-holding systemd unit
# watches that directory with inotify and is the *only* process that forwards
# the manifests to the upstream REST API.
#
# The token is re-read from `tokenFile` on every request, so rotating the
# secret takes effect with no restart of the watcher.
#
# This module is self-contained: import it, set `enable = true`, and provide
# `tokenFile`, `upstreamUrl`, and `user`. You are responsible for creating the
# spool directory as 1777 (see README) — this unit only reads/writes it.
{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.services.spoolCredentialBroker;

  watcherScript = pkgs.writeShellApplication {
    name = "spool-credential-broker";
    runtimeInputs = with pkgs; [
      inotify-tools
      curl
      jq
      coreutils
    ];
    text = ''
      set -euo pipefail

      SPOOL="''${SPOOL:?SPOOL required}"
      TOKEN_FILE="''${TOKEN_FILE:?TOKEN_FILE required}"
      UPSTREAM_URL="''${UPSTREAM_URL:?UPSTREAM_URL required}"
      CREATE_PATH="''${CREATE_PATH:-/api/sessions}"
      DELETE_PATH="''${DELETE_PATH:-/api/sessions}"
      ID_FIELD="''${ID_FIELD:-slug}"

      [ -d "$SPOOL" ]      || { echo "[broker] spool $SPOOL missing" >&2; exit 1; }
      [ -r "$TOKEN_FILE" ] || { echo "[broker] token $TOKEN_FILE not readable" >&2; exit 1; }

      # Re-read the token on every call: this is the whole point — a rotated
      # secret takes effect without restarting the unit.
      api() {
        local method="$1" path="$2" body="''${3:-}" token
        token=$(tr -d '\n\r' < "$TOKEN_FILE")
        # Never put the token on curl's argv: process arguments are world-
        # readable via /proc/<pid>/cmdline on a stock host, so a local
        # unprivileged producer — exactly the adversary this module confines —
        # could scrape the bearer token out of the in-flight curl. Instead feed
        # the Authorization header through a curl config on stdin (`-K -`), which
        # never appears in the process arguments.
        if [ -n "$body" ]; then
          printf 'header = "Authorization: Bearer %s"\n' "$token" \
          | curl -sS -f -K - -X "$method" \
            -H "Content-Type: application/json" \
            --data "$body" \
            "$UPSTREAM_URL$path"
        else
          printf 'header = "Authorization: Bearer %s"\n' "$token" \
          | curl -sS -f -K - -X "$method" \
            "$UPSTREAM_URL$path"
        fi
      }

      handle_create() {
        local f="$1" body id
        body=$(cat "$SPOOL/$f" 2>/dev/null || return 0)
        id=$(printf '%s' "$body" | jq -r ".$ID_FIELD // empty" 2>/dev/null || true)
        [ -n "$id" ] || { echo "[broker] no .$ID_FIELD in $f, skip" >&2; return 0; }
        if api POST "$CREATE_PATH" "$body" >/dev/null 2>&1; then
          echo "[broker] registered $id ← $f"
        else
          echo "[broker] upstream registration failed for $id" >&2
        fi
      }

      handle_delete() {
        local f="$1"
        # The manifest content is already gone on delete, so the resource id is
        # recovered from the filename (drop the .json suffix).
        local id="''${f%.json}"
        if api DELETE "$DELETE_PATH/$id" >/dev/null 2>&1; then
          echo "[broker] unregistered $id"
        fi
      }

      # Reconcile anything already present before we start watching, so a
      # restart re-forwards manifests that were dropped while we were down.
      shopt -s nullglob
      for f in "$SPOOL"/*.json; do
        bn=$(basename "$f")
        case "$bn" in .*) continue ;; esac
        handle_create "$bn"
      done
      shopt -u nullglob

      echo "[broker] watching $SPOOL → $UPSTREAM_URL"
      # close_write catches finished writes; moved_to catches atomic
      # write-tmp-then-rename producers. Both delete and moved_from tear down.
      inotifywait -m -e close_write,moved_to,delete,moved_from \
        --format '%e %f' "$SPOOL" \
      | while IFS=' ' read -r ev fn; do
          case "$fn" in *.json) ;; *) continue ;; esac
          case "$fn" in .*) continue ;; esac
          case "$ev" in
            CLOSE_WRITE|MOVED_TO) handle_create "$fn" ;;
            DELETE|MOVED_FROM)    handle_delete "$fn" ;;
          esac
        done
    '';
  };
in
{
  options.services.spoolCredentialBroker = {
    enable = lib.mkEnableOption "spool-dir credential broker: forward manifests dropped into a sticky spool dir to an upstream REST API";

    spoolDir = lib.mkOption {
      type = lib.types.path;
      default = "/var/lib/spool-broker/inbox";
      description = ''
        Directory where unprivileged producers drop JSON manifests. This should
        be a sticky (1777) directory so any local process can write its own
        manifest but not touch another's. This module does NOT create it — see
        the README for a systemd.tmpfiles rule.
      '';
    };

    tokenFile = lib.mkOption {
      type = lib.types.path;
      description = ''
        File containing the bearer token for the upstream API. Keep it out of
        the Nix store (use a secret manager) and readable only by `user`.
        Re-read on every request, so rotation needs no restart.
      '';
      example = "/run/secrets/upstream-token";
    };

    upstreamUrl = lib.mkOption {
      type = lib.types.str;
      description = "Base URL of the upstream REST API the broker forwards to.";
      example = "https://api.example.com";
    };

    createPath = lib.mkOption {
      type = lib.types.str;
      default = "/api/sessions";
      description = "Path POSTed to (with the full manifest body) when a manifest appears.";
    };

    deletePath = lib.mkOption {
      type = lib.types.str;
      default = "/api/sessions";
      description = "Base path DELETEd (as `<deletePath>/<id>`) when a manifest is removed.";
    };

    idField = lib.mkOption {
      type = lib.types.str;
      default = "slug";
      description = ''
        JSON field in each manifest carrying the resource id. Also the manifest
        filename stem (`<id>.json`) so deletes can recover the id from the name.
      '';
    };

    user = lib.mkOption {
      type = lib.types.str;
      description = "User the broker runs as. Must have read access to `tokenFile`.";
      example = "spool-broker";
    };

    group = lib.mkOption {
      type = lib.types.str;
      default = cfg.user;
      defaultText = lib.literalExpression "cfg.user";
      description = "Group the broker runs as.";
    };
  };

  config = lib.mkIf cfg.enable {
    systemd.services.spool-credential-broker = {
      description = "Forward spool-dir manifests to an upstream REST API";
      wantedBy = [ "multi-user.target" ];
      after = [
        "network-online.target"
        "systemd-tmpfiles-setup.service"
      ];
      wants = [ "network-online.target" ];
      environment = {
        SPOOL = cfg.spoolDir;
        TOKEN_FILE = toString cfg.tokenFile;
        UPSTREAM_URL = cfg.upstreamUrl;
        CREATE_PATH = cfg.createPath;
        DELETE_PATH = cfg.deletePath;
        ID_FIELD = cfg.idField;
      };
      serviceConfig = {
        ExecStart = "${watcherScript}/bin/spool-credential-broker";
        User = cfg.user;
        Group = cfg.group;
        Restart = "on-failure";
        RestartSec = 5;
        # Hardening: if the broker is ever compromised it holds the token, so
        # confine it to nothing but reading the token and writing the spool.
        ProtectSystem = "strict";
        ProtectHome = true;
        PrivateTmp = true;
        NoNewPrivileges = true;
        ReadOnlyPaths = [ (toString cfg.tokenFile) ];
        ReadWritePaths = [ cfg.spoolDir ];
      };
    };
  };
}