jaeger-oci-tracing¶

Modules

Run Jaeger distributed tracing as an OCI/Docker container on NixOS, with the UI behind nginx+TLS and the OTLP ingest ports left raw for trusted collectors.

The problem¶

There is no native nixpkgs service for Jaeger, so it runs as a container. A tracing backend has two very different network surfaces, and treating them the same gets you either a broken UI or an open door:

The UI (port 16686) is a web app. It wants TLS and a domain, so it goes through nginx like any other web service. Its container port is published on 127.0.0.1, so it is reachable only through the nginx TLS vhost — never as raw plaintext HTTP on an external interface.
The OTLP ingest ports (4317 gRPC, 4318 HTTP) are where your services push spans. They are unauthenticated. Proxying them through nginx buys nothing and adds latency; what they need is to be reachable only from your collectors — a VPN, a private subnet, a trusted interface — never the public internet. They are published on otlpListenAddress, which defaults to loopback; point it at a trusted-network interface to reach the host remotely.

So the module deliberately splits the surface: nginx proxies the UI only, and the OTLP ports are published on a bind address you control.

Docker bypasses the NixOS firewall. Docker publishes container ports with a DNAT rule in the DOCKER iptables chain that skips the networking.firewall INPUT chain. So what actually determines exposure is the publish bind address (127.0.0.1 for the UI, otlpListenAddress for OTLP), not openFirewall. A 0.0.0.0 bind is reachable on every interface regardless of the firewall toggle. This is why the defaults bind to loopback.

The traps this encodes¶

The docker network must be created before the container starts. The container joins a named docker network. NixOS's generated docker-jaeger unit has no idea that network needs to exist first, so a docker-network-jaeger oneshot (Type = "oneshot", RemainAfterExit = true) creates the network and is ordered before the container, which requires it. Drop the ordering and, on boot, the container comes up attached to nothing.
Enabling without a UI domain fails eval, on purpose. domain is asserted non-null. Without the assertion, forgetting to set it would silently leave the UI with no nginx vhost — reachable only on loopback — with no error to tell you why.
The OTLP ports are unauthenticated. They default to a loopback bind (otlpListenAddress = "127.0.0.1"), so out of the box nothing off-host can reach them. To let remote collectors in, set otlpListenAddress to a trusted-network interface address (VPN / private subnet). Because Docker's published-port DNAT bypasses the NixOS firewall, this bind address — not openFirewall — is what governs exposure. openFirewall defaults to false and only opens the OTLP ports in the firewall INPUT chain; enable it only where the network already restricts those ports to trusted collectors.

Usage¶

Import the module and enable it:

{
  imports = [ ./modules/jaeger-oci-tracing ];

  modules.services.jaeger = {
    enable   = true;
    domain   = "jaeger.example.com";  # UI vhost (required)
    acmeHost = "example.com";         # cert to reuse, or null
    image    = "jaegertracing/all-in-one:latest";

    # OTLP ingest binds to loopback by default. To accept spans from other
    # hosts, publish it on a trusted-network interface (VPN / private subnet):
    # otlpListenAddress = "10.0.0.1";
  };
}

Browse the UI at https://jaeger.example.com. With the default otlpListenAddress, OTLP ingest is loopback-only — point exporters running on the same host at 127.0.0.1:4317 (gRPC) or 127.0.0.1:4318 (HTTP). To ingest from other hosts, set otlpListenAddress to a trusted-interface IP and point exporters at that address.

Options¶

Option	Default	Purpose
`enable`	`false`	Turn the module on.
`domain`	`null`	UI vhost domain. Required (asserted non-null).
`acmeHost`	`null`	ACME host whose cert the vhost reuses (`useACMEHost`); `null` to manage TLS elsewhere.
`image`	`jaegertracing/all-in-one:latest`	OCI image reference.
`imageFile`	`null`	Pre-built image tarball (e.g. a pinned `dockerTools.pullImage`) to load instead of pulling `image`.
`user` / `group`	`jaeger`	System user/group owning the data dir.
`uid` / `gid`	`1322`	IDs for that user/group.
`dataDir`	`/var/lib/jaeger`	Data directory (created `0700`).
`uiPort`	`16686`	Host port for the UI (proxied).
`otlpGrpcPort`	`4317`	Host port for OTLP gRPC ingest (raw).
`otlpHttpPort`	`4318`	Host port for OTLP HTTP ingest (raw).
`network`	`jaeger`	Docker network name the container joins.
`otlpListenAddress`	`127.0.0.1`	Host interface the OTLP ports are published on. Loopback by default; set to a trusted-interface IP for remote collectors. Governs exposure (Docker DNAT bypasses the firewall).
`openFirewall`	`false`	Open the OTLP ports in the firewall INPUT chain (opt-in; partial control — see the Docker/firewall note). The UI is never opened raw.

Pinning the image¶

For reproducible deploys, build the image tarball with pkgs.dockerTools.pullImage (or your own buildImage) and pass it as imageFile; the container runtime loads it instead of pulling by reference:

modules.services.jaeger.imageFile = pkgs.dockerTools.pullImage {
  imageName = "jaegertracing/all-in-one";
  imageDigest = "sha256:...";
  sha256 = "...";
};

Caveats¶

Requires virtualisation.docker.enable = true and an nginx that serves the configured vhost.
The all-in-one image keeps traces in memory by default; for retention, switch to a Jaeger image backed by a persistent store and mount dataDir.
OTLP ingest is unauthenticated — see trap 3.

Source¶

modules/jaeger-oci-tracing/default.nix

# jaeger-oci-tracing
#
# Run Jaeger distributed tracing as an OCI/Docker container behind nginx.
#
# The pattern, and the two traps it defends against:
#
#   1. Split the surface. Only the Jaeger UI (16686) is proxied through nginx
#      for TLS; its container port is published on 127.0.0.1 so it is never
#      reachable as raw plaintext HTTP. The OTLP ingest ports (4317 gRPC, 4318
#      HTTP) are published on `otlpListenAddress` (loopback by default) so
#      collectors reach the container directly. Those ports are UNAUTHENTICATED
#      — point `otlpListenAddress` at a trusted-network interface (VPN /
#      private subnet) to reach them remotely. Do not expose them to the public
#      internet. NOTE: Docker publishes ports via a DNAT rule in the DOCKER
#      iptables chain that BYPASSES the NixOS firewall INPUT chain, so exposure
#      is controlled by the publish bind address, not by `openFirewall`.
#
#   2. Order the docker network before the container. The container joins a
#      named docker network; if the `docker-network-*` oneshot does not run
#      first, the container boots with no network. The `before`/`after`/
#      `requires` wiring below makes that ordering explicit.
#
# `domain` is asserted non-null so that enabling the module without a UI domain
# fails at eval time rather than silently leaving the UI unproxied.
#
# Import into a host config and set, at minimum, `domain` and one of
# `image` / `imageFile`.
{
  config,
  lib,
  pkgs,
  ...
}:
with lib;
let
  cfg = config.modules.services.jaeger;
in
{
  options.modules.services.jaeger = {
    enable = mkEnableOption "Jaeger tracing service";

    domain = mkOption {
      type = types.nullOr types.str;
      default = null;
      example = "jaeger.example.com";
      description = "Domain name for the Jaeger UI (proxied via nginx with TLS).";
    };

    acmeHost = mkOption {
      type = types.nullOr types.str;
      default = null;
      example = "example.com";
      description = ''
        ACME host whose certificate the nginx vhost should reuse
        (services.nginx.virtualHosts.<domain>.useACMEHost). Set to null to
        manage the certificate elsewhere.
      '';
    };

    image = mkOption {
      type = types.str;
      default = "jaegertracing/all-in-one:latest";
      description = "OCI image reference for the Jaeger container.";
    };

    imageFile = mkOption {
      type = types.nullOr types.package;
      default = null;
      example = literalExpression "pkgs.dockerTools.pullImage { /* ... */ }";
      description = ''
        Optional pre-built image tarball to load instead of pulling `image`
        from a registry (e.g. a pinned `dockerTools.pullImage` result). When
        null, the container runtime pulls `image` by reference.
      '';
    };

    user = mkOption {
      type = types.str;
      default = "jaeger";
      description = "System user that owns the data directory.";
    };

    group = mkOption {
      type = types.str;
      default = "jaeger";
      description = "Primary group for the Jaeger service user.";
    };

    uid = mkOption {
      type = types.int;
      default = 3300;
      description = "User ID for the Jaeger service user.";
    };

    gid = mkOption {
      type = types.int;
      default = 3300;
      description = "Group ID for the Jaeger service.";
    };

    dataDir = mkOption {
      type = types.str;
      default = "/var/lib/jaeger";
      description = "Directory to store Jaeger data.";
    };

    uiPort = mkOption {
      type = types.port;
      default = 16686;
      description = "Host port for the Jaeger UI (proxied through nginx).";
    };

    otlpGrpcPort = mkOption {
      type = types.port;
      default = 4317;
      description = ''
        Host port for OTLP gRPC ingest. Opened raw in the firewall and
        UNAUTHENTICATED — restrict to a trusted network.
      '';
    };

    otlpHttpPort = mkOption {
      type = types.port;
      default = 4318;
      description = ''
        Host port for OTLP HTTP ingest. Opened raw in the firewall and
        UNAUTHENTICATED — restrict to a trusted network.
      '';
    };

    network = mkOption {
      type = types.str;
      default = "jaeger";
      description = "Name of the docker network the container joins.";
    };

    otlpListenAddress = mkOption {
      type = types.str;
      default = "127.0.0.1";
      example = "10.0.0.1";
      description = ''
        Host interface address the OTLP ingest ports (gRPC/HTTP) are published
        on. Docker publishes container ports with an iptables DNAT rule in the
        DOCKER chain that BYPASSES the NixOS `networking.firewall` INPUT chain,
        so a `0.0.0.0` bind is reachable on every interface regardless of
        `openFirewall`. Because OTLP ingest is UNAUTHENTICATED, this defaults to
        loopback. Set it to the address of a trusted interface (a VPN or private
        subnet address, e.g. "10.0.0.1") to let remote collectors reach the
        host. Only use "0.0.0.0" if a firewall in front of the box already
        restricts these ports to trusted collectors.
      '';
    };

    openFirewall = mkOption {
      type = types.bool;
      default = false;
      description = ''
        Open the OTLP ingest ports in the `networking.firewall` INPUT chain.

        Note this is only a partial control: Docker's published-port DNAT rules
        live in the DOCKER iptables chain and bypass the NixOS firewall INPUT
        chain, so the effective exposure of the OTLP ports is governed by
        `otlpListenAddress`, not by this toggle. Opening the firewall is only
        meaningful for ports bound to a non-loopback address. OTLP ingest is
        UNAUTHENTICATED — leave this off unless a trusted network already
        restricts access. The UI is never opened here; it is reached only via
        the nginx TLS vhost.
      '';
    };
  };

  config = mkIf cfg.enable {
    assertions = [
      {
        assertion = cfg.domain != null;
        message = "modules.services.jaeger: domain must be set when enabled";
      }
    ];

    systemd.services.docker-jaeger = {
      after = [ "docker-network-jaeger.service" ];
      requires = [ "docker-network-jaeger.service" ];
    };

    users.users.${cfg.user} = {
      inherit (cfg) uid;
      isSystemUser = true;
      home = cfg.dataDir;
      group = cfg.group;
    };

    users.groups.${cfg.group}.gid = cfg.gid;

    systemd.tmpfiles.rules = [
      "d ${cfg.dataDir} 0700 ${toString cfg.uid} ${toString cfg.gid} - -"
    ];

    # Only the OTLP ingest ports are ever opened here, and only opt-in. The UI
    # is never opened raw — it is reached exclusively through the nginx TLS
    # vhost below (the UI publish is bound to loopback).
    networking.firewall.allowedTCPPorts = mkIf cfg.openFirewall [
      cfg.otlpGrpcPort
      cfg.otlpHttpPort
    ];

    # The container joins the `${cfg.network}` docker network by name. This
    # oneshot must create it BEFORE the container starts, or the container
    # boots with no network.
    systemd.services."docker-network-jaeger" = {
      serviceConfig = {
        Type = "oneshot";
        RemainAfterExit = true;
      };
      before = [ "docker-jaeger.service" ];
      wantedBy = [ "multi-user.target" ];
      script = ''
        ${pkgs.docker}/bin/docker network inspect ${cfg.network} > /dev/null 2>&1 \
          || ${pkgs.docker}/bin/docker network create ${cfg.network}
      '';
    };

    virtualisation.oci-containers = {
      backend = "docker";
      containers.jaeger = {
        inherit (cfg) image;
        imageFile = mkIf (cfg.imageFile != null) cfg.imageFile;
        # The UI is bound to loopback so it is reachable ONLY through the nginx
        # TLS vhost, never as raw plaintext HTTP on an external interface.
        # OTLP ports are bound to `otlpListenAddress` (loopback by default);
        # both binds matter because Docker DNAT bypasses the NixOS firewall.
        ports = [
          "127.0.0.1:${toString cfg.uiPort}:16686"
          "${cfg.otlpListenAddress}:${toString cfg.otlpGrpcPort}:4317"
          "${cfg.otlpListenAddress}:${toString cfg.otlpHttpPort}:4318"
        ];
        extraOptions = [
          "--network=${cfg.network}"
        ];
        autoStart = true;
      };
    };

    # Only the UI is proxied for TLS. The OTLP ingest ports stay raw so
    # collectors talk to the container directly.
    services.nginx.virtualHosts.${cfg.domain} = {
      forceSSL = true;
      useACMEHost = cfg.acmeHost;
      locations."/" = {
        proxyPass = "http://127.0.0.1:${toString cfg.uiPort}/";
        proxyWebsockets = true;
      };
    };
  };
}