tailscale-derp-server

Modules

A NixOS module that self-hosts a Tailscale DERP relay behind nginx, with ACME certificates and abuse filtering.

What problem it solves

DERP is Tailscale's relay of last resort: when two nodes can't establish a direct WireGuard path (symmetric NAT, restrictive firewalls), their encrypted packets bounce through a DERP relay instead. Running your own relay keeps that traffic on your own infrastructure rather than Tailscale's public DERP mesh, and — fronted on port 443 — it works through corporate firewalls that block non-standard ports.

NixOS ships no DERP module, so this wraps the derper binary (from pkgs.tailscale.derper) into a hardened systemd service plus an nginx reverse proxy.

The insight & the traps

The value here is the wiring, not the binary. Several things are non-obvious:

• -certmode=manual fed by symlinks. derper's default cert mode wants to run its own LetsEncrypt autocert. Instead we let NixOS's ACME manage the cert and hand derper the files. ACME and derper disagree on filenames, so an ExecStartPre symlinks fullchain.pem/key.pem to the <hostname>.crt/ .key names derper expects.

• Nothing runs as root — not even the setup step. The obvious way to write that ExecStartPre is with systemd's + prefix so it can chown freely. That is unnecessary: StateDirectory=derper already creates the state directory owned by the service user, and the certificate is reachable by joining the group that owns it. Dropping the prefix also makes the readability check real — as root, test -r always succeeds and has to be faked with su, which silently tests the wrong thing if the user's groups are wrong. Run as the service user and the check is the same access the daemon will perform a second later, so a group misconfiguration fails the unit immediately with a clear message rather than surfacing minutes later as a TLS handshake error.

• The owning group is derived, not assumed. It is tempting to hardcode nginx, since a web server usually provisions the cert. But that group is a property of the certificate, not of this module: it comes from security.acme.certs.<dir>.group, whose NixOS default is acme. The module reads that value and joins whatever it actually is, and asserts at build time that it could determine it — so a mismatch is a build error with a fix in the message, never a running relay that cannot read its own key.

• Poll for the cert. The unit orders after the acme-finished-<host>.target, but the files can still lag that target. The ExecStartPre polls up to ~60s for them before symlinking — without the loop, a race leaves derper starting with no cert.

• HTTP/2 must be OFF on the nginx vhost. DERP negotiates over HTTP/1.1 Upgrade semantics; leaving HTTP/2 on breaks the relay handshake.

• Only proxy the real DERP endpoints; 444 everything else. The vhost enumerates exactly /derp (websocket, with 10-minute timeouts), /derp/probe, /derp/latency-check, /generate_204, /robots.txt, and /bootstrap-dns. Any other path hits location "/", gets logged to a probes log, and returns 444 (nginx closes the connection with no response). This keeps the box from looking like a live web server to internet scanners.

• Gate clients to your tailnet with -verify-clients. Without it your relay is an open DERP anyone can bounce traffic through. With it, derper checks each client against the local Tailscale daemon (which must be running on the host).

• Split-port trap. All the nginx and fail2ban machinery is gated on port != 443. If you run derper directly on 443 (no reverse proxy), the vhost and jails silently vanish — that's intentional, but surprising.

• Backend SSL verify off + buffering off. nginx talks to derper over loopback HTTPS; proxy_ssl_verify off avoids validating the loopback cert, and buffering off keeps long-lived relay streams from stalling.

Certificate setup

The module reads a cert from /var/lib/acme/<acmeHost or hostname>/. You still have to arrange for that cert to exist; the module works out who may read it.

Nothing in this module runs as root, so access is purely by group membership. The module looks up the matching security.acme.certs entry and adds the derper user to that certificate's group — whatever it is. Declare the cert however you normally would:

security.acme.certs."derp.example.com".group = "nginx";

The NixOS default for that option is acme, not nginx; either works, since the group is read rather than assumed. When nginx fronts the relay you will usually want nginx anyway so the proxy can read the same files, and the nginx gid is pinned (via mkDefault) so it is stable on minimal hosts.

If the certificate is provisioned outside the NixOS ACME module there is nothing to look up, and the build fails with an assertion telling you to name the group yourself:

services.derp-server.acmeGroup = "some-group";

Usage

{
  imports = [ ./modules/tailscale-derp-server ];

  services.derp-server = {
    enable   = true;
    hostname = "derp.example.com";   # MUST match the served TLS cert
    port     = 8443;                 # derper listener; nginx proxies 443 -> here
    stunPort = 3478;
    verifyClients = true;            # gate to your tailnet
    # acmeHost = "example.com";      # if a different-named cert covers hostname
  };

  # Optional: ban scanners (needs services.fail2ban.enable = true)
  services.derp-server.fail2ban.enable = true;
}

Then point your Tailscale/Headscale control plane at the relay in your DERP map (region with HostName = "derp.example.com", DERPPort = 443, STUNPort = 3478).

Options

Option	Default	Purpose
enable	false	Turn the module on.
hostname	(required)	DERP hostname; must match the TLS cert.
port	8443	derper's own HTTPS listener. 443 = run direct, no nginx.
stunPort	3478	STUN port (opened on the firewall).
verifyClients	true	Pass -verify-clients; restrict relay to your tailnet.
acmeHost	null → hostname	Which /var/lib/acme/<dir> cert to read.
acmeGroup	null → cert's own group	Group owning the ACME files, which user joins to read them. Auto-detected from security.acme.certs.<dir>.group; set only when the cert is provisioned outside the ACME module.
user / group	derper	System user/group derper runs as. Never root.
fail2ban.enable	false	Register the probe + bad-TLS jails (needs nginx + services.fail2ban).
fail2ban.action	null	fail2ban action for the jails; null = global default.
fail2ban.bantime	"168h"	Ban duration.

fail2ban jails

When fail2ban.enable is set (and nginx is fronting derper), two jails are registered with matching /etc/fail2ban/filter.d filters:

• derp-probes — bans IPs that trip the nginx 444 responses (i.e. hit paths that aren't real DERP endpoints), read from /var/log/nginx/derp-probes.log.
• derp-bad-tls — bans IPs producing cert mismatch TLS handshake errors in the derper journal.

Leave fail2ban.action at null to use fail2ban's global banaction, or set it to a named action of your own.

Security notes

The systemd unit is tightly sandboxed: ProtectSystem=strict, ProtectHome=true, NoNewPrivileges=true, /var/lib/acme mounted read-only, and the capability set narrowed to just CAP_NET_BIND_SERVICE (the ambient cap that lets the unprivileged user bind the low STUN/relay ports).

No step in the unit runs as root, including ExecStartPre — the service user is derper throughout, and two build-time assertions keep it that way: one rejects user = "root", the other refuses to build unless the owning group of the ACME material could be determined.

Source

modules/tailscale-derp-server/default.nix

# Self-hosted Tailscale DERP relay behind nginx.
#
# Wraps Tailscale's `derper` binary in a hardened systemd service, feeds it
# ACME-managed certificates via `-certmode=manual`, and (unless it runs
# directly on 443) fronts it with nginx so the relay is reachable on the
# standard HTTPS port. See README.md for the why and the traps.
#
# Usage:
#   imports = [ ./modules/tailscale-derp-server ];
#   services.derp-server = {
#     enable   = true;
#     hostname = "derp.example.com";  # must match the served TLS cert
#     # acmeHost = "example.com";     # optional: which ACME cert dir to read
#     # port     = 8443;              # derper's own listener (nginx proxies 443 -> this)
#     # stunPort = 3478;
#     # verifyClients = true;         # gate to your tailnet
#   };
#
# You must arrange the ACME certificate yourself, e.g.:
#   security.acme.certs."derp.example.com".group = "nginx";
{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.services.derp-server;

  derper = pkgs.tailscale.derper;

  # Whether nginx fronts derper. When derper listens directly on 443 there is
  # no reverse proxy, so the nginx vhost + abuse filtering are irrelevant.
  useNginx = cfg.port != 443;

  # Directory under /var/lib/acme that holds fullchain.pem / key.pem.
  acmeDir = if cfg.acmeHost != null then cfg.acmeHost else cfg.hostname;
  acmeCertPath = "/var/lib/acme/${acmeDir}";

  # The group that owns the ACME material. Nothing here runs as root, so the
  # relay reads its certificate purely through group membership -- which means
  # this group has to be right, and is asserted below rather than assumed.
  acmeCert = config.security.acme.certs.${acmeDir} or null;
  certGroup =
    if cfg.acmeGroup != null then
      cfg.acmeGroup
    else if acmeCert != null then
      acmeCert.group
    else
      null;
in
{
  options.services.derp-server = {
    enable = lib.mkOption {
      type = lib.types.bool;
      default = false;
      description = "Enable DERP relay server";
    };

    hostname = lib.mkOption {
      type = lib.types.str;
      example = "derp.example.com";
      description = "Hostname for DERP server (must match the TLS certificate served to clients)";
    };

    port = lib.mkOption {
      type = lib.types.port;
      default = 8443;
      description = ''
        HTTPS port derper listens on. When set to anything other than 443,
        nginx is enabled and proxies public 443 traffic to this port. Set to
        443 to run derper directly with no reverse proxy.
      '';
    };

    stunPort = lib.mkOption {
      type = lib.types.port;
      default = 3478;
      description = "STUN port (opened on the firewall)";
    };

    verifyClients = lib.mkOption {
      type = lib.types.bool;
      default = true;
      description = ''
        Pass `-verify-clients` to derper so it only relays for nodes in the
        local tailnet. Requires the Tailscale daemon to be running on this
        host. Prevents the relay from being abused as an open DERP.
      '';
    };

    acmeHost = lib.mkOption {
      type = lib.types.nullOr lib.types.str;
      default = null;
      example = "example.com";
      description = ''
        Name of the ACME certificate directory under /var/lib/acme to read
        the cert from. Defaults to `hostname`. Use this when a single
        wildcard/SAN cert (issued for a different primary name) covers the
        DERP hostname.
      '';
    };

    user = lib.mkOption {
      type = lib.types.str;
      default = "derper";
      description = "System user derper runs as. Must not be root.";
    };

    acmeGroup = lib.mkOption {
      type = lib.types.nullOr lib.types.str;
      default = null;
      example = "nginx";
      description = ''
        Group owning the ACME certificate files, which the derper user joins in
        order to read them. Defaults to the `group` of the matching
        `security.acme.certs` entry, so normally you do not set this. Set it
        explicitly when the certificate is provisioned outside the NixOS ACME
        module and therefore cannot be discovered.
      '';
    };

    group = lib.mkOption {
      type = lib.types.str;
      default = "derper";
      description = "Primary group for the derper user.";
    };

    fail2ban = {
      enable = lib.mkOption {
        type = lib.types.bool;
        default = false;
        description = ''
          Register fail2ban jails that ban scanners hitting the relay: one on
          nginx 444 responses (probe requests to non-DERP paths) and one on
          derper "cert mismatch" TLS journal errors. Only active when nginx
          fronts derper (port != 443) and `services.fail2ban.enable` is true.
        '';
      };

      action = lib.mkOption {
        type = lib.types.nullOr lib.types.str;
        default = null;
        example = "iptables-allports";
        description = ''
          fail2ban action for the DERP jails. `null` uses fail2ban's global
          default (`services.fail2ban.banaction`). Set to a named action if
          you want these jails to use a specific one.
        '';
      };

      bantime = lib.mkOption {
        type = lib.types.str;
        default = "168h";
        description = "Ban duration for the DERP jails.";
      };
    };
  };

  config = lib.mkIf cfg.enable {
    networking.firewall = {
      allowedUDPPorts = [
        config.services.tailscale.port
        cfg.stunPort
      ];
    };

    systemd.services.derp-server = {
      description = "DERP (Designated Encrypted Relay for Packets) server";
      after = [
        "network-online.target"
        "acme-finished-${acmeDir}.target"
      ] ++ lib.optional useNginx "nginx.service";
      wants = [
        "network-online.target"
        "acme-finished-${acmeDir}.target"
      ];
      wantedBy = [ "multi-user.target" ];

      path = [ pkgs.coreutils ];

      serviceConfig = {
        Type = "simple";
        Restart = "always";
        RestartSec = 5;

        # Runs as the service user, NOT root: `StateDirectory` already creates
        # /var/lib/derper owned by it, and the certificate is reached through
        # group membership. Polls first because the ACME files can lag the
        # acme-finished target, then symlinks them to the <hostname>.crt/.key
        # names derper expects.
        #
        # The readability test is the real thing rather than a root-side `su`
        # emulation of it, so a group misconfiguration fails the unit here with
        # a precise message instead of surfacing later as a TLS handshake error.
        ExecStartPre = pkgs.writeShellScript "derper-cert-links" ''
          set -euo pipefail

          i=0
          while [ $i -lt 30 ]; do
            if [ -r "${acmeCertPath}/fullchain.pem" ] && [ -r "${acmeCertPath}/key.pem" ]; then
              break
            fi
            i=$((i + 1))
            sleep 2
          done

          if [ ! -r "${acmeCertPath}/fullchain.pem" ] || [ ! -r "${acmeCertPath}/key.pem" ]; then
            echo "derp-server: cannot read ${acmeCertPath}/{fullchain,key}.pem as $(id -un):$(id -gn)." >&2
            echo "derp-server: that material is owned by group ${
              if certGroup != null then certGroup else "<unknown>"
            }; the ${cfg.user} user must be a member of it." >&2
            exit 1
          fi

          ln -sfn "${acmeCertPath}/fullchain.pem" "/var/lib/derper/${cfg.hostname}.crt"
          ln -sfn "${acmeCertPath}/key.pem" "/var/lib/derper/${cfg.hostname}.key"
        '';

        # -http-port=-1 disables plain HTTP; -certmode=manual reads the
        # symlinked ACME cert instead of derper's LetsEncrypt autocert.
        # When nginx fronts derper, bind the HTTPS listener to loopback only
        # (nginx proxies from 127.0.0.1) so the backend port can never be hit
        # directly even if the host firewall/cloud SG leaves it open. On the
        # direct-443 path bind all interfaces since there is no proxy.
        ExecStart = "${derper}/bin/derper -c=/var/lib/derper/derper.key -hostname=${cfg.hostname} -a=${if useNginx then "127.0.0.1" else ""}:${toString cfg.port} -http-port=-1 -stun-port=${toString cfg.stunPort} -certmode=manual -certdir=/var/lib/derper ${lib.optionalString cfg.verifyClients "-verify-clients"}";

        StateDirectory = "derper";
        StateDirectoryMode = "0700";

        User = cfg.user;
        Group = cfg.group;
        PrivateTmp = true;
        ProtectSystem = "strict";
        ProtectHome = true;
        NoNewPrivileges = true;
        ReadOnlyPaths = [ "/var/lib/acme" ];
        ReadWritePaths = [ "/var/lib/derper" ];
        # Ambient cap lets the unprivileged user bind low STUN/relay ports.
        AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
        CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
      };
    };

    assertions = [
      {
        assertion = cfg.user != "root";
        message = ''
          services.derp-server.user must not be root. The relay reads its
          certificate through membership of the group that owns it, so it never
          needs privilege.
        '';
      }
      {
        assertion = certGroup != null;
        message = ''
          services.derp-server cannot determine which group owns the ACME
          certificate in ${acmeCertPath}, so it cannot grant the ${cfg.user}
          user read access to it.

          Either define the certificate through the NixOS ACME module, e.g.
            security.acme.certs."${acmeDir}".group = "nginx";
          (its `group` is picked up automatically), or, if the certificate is
          provisioned some other way, name the owning group explicitly:
            services.derp-server.acmeGroup = "<group>";
        '';
      }
    ];

    users.groups.${cfg.group} = { };
    # Pin nginx gid so it is stable on hosts that don't otherwise define it.
    users.groups.nginx = lib.mkIf useNginx (lib.mkDefault { gid = 60; });
    users.users.${cfg.user} = {
      isSystemUser = true;
      group = cfg.group;
      # Read access to the ACME material comes from joining whichever group owns
      # it -- derived from the cert definition rather than assumed to be nginx,
      # because that only happens to be true when a web server provisions it.
      extraGroups = lib.optional (certGroup != null) certGroup;
    };

    services.fail2ban.jails = lib.mkIf (cfg.fail2ban.enable && useNginx) {
      derp-bad-tls.settings = {
        enabled = true;
        filter = "derp-bad-tls";
        backend = "systemd";
        maxretry = 3;
        findtime = 600;
        bantime = cfg.fail2ban.bantime;
      } // lib.optionalAttrs (cfg.fail2ban.action != null) { action = cfg.fail2ban.action; };
      derp-probes.settings = {
        enabled = true;
        filter = "derp-probes";
        logpath = "/var/log/nginx/derp-probes.log";
        backend = "auto";
        maxretry = 3;
        findtime = 600;
        bantime = cfg.fail2ban.bantime;
      } // lib.optionalAttrs (cfg.fail2ban.action != null) { action = cfg.fail2ban.action; };
    };

    environment.etc = lib.mkIf (cfg.fail2ban.enable && useNginx) {
      "fail2ban/filter.d/derp-bad-tls.local".text = ''
        [Definition]
        failregex = ^.*http: TLS handshake error from <HOST>:[0-9]+: .*cert mismatch.*$
        journalmatch = _SYSTEMD_UNIT=derp-server.service
      '';
      "fail2ban/filter.d/derp-probes.local".text = ''
        [Definition]
        failregex = ^<HOST> - .* "(GET|POST|PUT|DELETE|HEAD|OPTIONS|PATCH|CONNECT) [^"]*" 444
      '';
    };

    services.nginx.enable = lib.mkDefault useNginx;
    services.nginx.virtualHosts.${cfg.hostname} = lib.mkIf useNginx (
      let
        upstream = "https://127.0.0.1:${toString cfg.port}";
        # Backend SSL verification off (derper serves a cert for `hostname` on
        # loopback); buffering off so relay streams don't stall.
        sharedProxyConfig = ''
          proxy_ssl_verify off;
          proxy_ssl_protocols TLSv1.2 TLSv1.3;
          proxy_ssl_server_name on;
          proxy_ssl_name ${cfg.hostname};
          proxy_ssl_session_reuse on;
          proxy_set_header Host ${cfg.hostname};
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_buffering off;
          proxy_request_buffering off;
        '';
        plainLocation = {
          proxyPass = upstream;
          recommendedProxySettings = false;
          extraConfig = sharedProxyConfig;
        };
        upgradeLocation = {
          proxyPass = upstream;
          proxyWebsockets = true;
          recommendedProxySettings = false;
          extraConfig = sharedProxyConfig + ''
            proxy_connect_timeout 10m;
            proxy_send_timeout 10m;
            proxy_read_timeout 10m;
          '';
        };
      in
      {
        forceSSL = true;
        useACMEHost = acmeDir;
        # HTTP/2 OFF: DERP uses HTTP/1.1 Upgrade semantics.
        http2 = false;
        extraConfig = ''
          server_tokens off;
          access_log /var/log/nginx/derp-access.log combined;
          error_log /var/log/nginx/derp-error.log warn;
        '';
        # Enumerate only real DERP endpoints; everything else returns 444
        # (connection closed, no response) so the box isn't a probe target.
        locations = {
          "= /derp" = upgradeLocation;
          "= /derp/probe" = plainLocation;
          "= /derp/latency-check" = plainLocation;
          "= /generate_204" = plainLocation;
          "= /robots.txt" = plainLocation;
          "= /bootstrap-dns" = plainLocation;
          "/" = {
            extraConfig = ''
              access_log /var/log/nginx/derp-probes.log combined;
              return 444;
            '';
          };
        };
      }
    );
  };
}