hydra-ci-server¶

Modules

Self-host Hydra — the Nix-native CI/CD server — behind nginx, backed by a PostgreSQL database you provision declaratively instead of letting Hydra bootstrap its own.

The module is written against upstream NixOS options only (services.hydra, services.postgresql, services.nginx), so it drops into any NixOS host. The reason it exists as a recipe is the set of five non-obvious traps between "enabled" and "actually works behind a proxy on a DB you control." Each is annotated inline in default.nix.

The problem¶

services.hydra.enable = true gets you a process, but:

Hydra runs as three system users and expects to own its database.
Its search needs a Postgres extension that isn't there by default.
It builds every link and redirect from a header most proxies don't set.
If you provision the DB yourself, its first-run bootstrap fights you.

Each of those is a silent failure — the service starts, then misbehaves.

The five traps¶

1. Three-user peer-auth ident map → one DB role¶

Hydra runs as hydra, hydra-queue-runner, and hydra-www (fixed upstream). Postgres peer auth authenticates by OS user name, so without a mapping only a role literally named hydra-queue-runner could connect. The module installs a pg_ident.conf map so all three — plus root, for out-of-band psql — auth as the single hydra role, and a matching pg_hba.conf line that enables the map:

hydra  hydra               hydra
hydra  hydra-queue-runner  hydra
hydra  hydra-www           hydra
hydra  root                hydra

2. `pg_trgm` extension¶

Hydra's job/build search uses trigram indexes. Without pg_trgm the schema initialization fails and Hydra won't run evaluations. Upstream Postgres has no per-database setup hook, so the module runs CREATE EXTENSION IF NOT EXISTS pg_trgm in a oneshot ordered before hydra-init.

3. `X-Request-Base` header¶

Hydra composes every absolute URL — redirects, page links, notification bodies — from the X-Request-Base request header, not from Host. Behind a reverse proxy that header is absent unless you set it, and links break in subtle ways (a login redirect lands on http://127.0.0.1:3000/...). The nginx location sets it explicitly to https://<domain>.

4. `.db-created` sentinel¶

hydra-init bootstraps its own database on first run: it creates the role and database and applies the schema. But here the role and database are provisioned declaratively via ensureDatabases / ensureUsers, so that bootstrap would try to CREATE objects that already exist — and fail. The module's preStart pre-touches the .db-created sentinel Hydra checks, so it skips the bootstrap and goes straight to running against the DB you gave it.

5. Import-from-derivation + large `maxOutputSize`¶

allow-import-from-derivation is passed both to nix.settings and into Hydra's evaluator env (NIX_CONFIG via extraEnv) — the nix.conf setting alone does not reach the evaluator. Needed for jobsets that fetch/generate inputs during evaluation. Caveat: IFD evaluations can't be gated by Hydra's --no-build dry pass, so IFD jobsets must be built, not dry-evaluated.
max_output_size defaults to 8 GiB. Heavy builds (large ML / CUDA closures) otherwise trip Hydra's "output limit exceeded"; raise it as needed.

Usage¶

{
  imports = [ ./hydra-ci-server ];

  services.hydra-ci-server = {
    enable = true;
    domain = "hydra.example.com";
    # acmeHost = "example.com";     # reuse an existing cert; else nginx gets its own
    # buildMachinesFile = "/etc/nix/machines";
    # maxOutputSize = 16 * 1024 * 1024 * 1024;
  };
}

Options¶

Option	Default	Purpose
`enable`	`false`	Turn the module on.
`domain`	(required)	Public host; used for the vhost and `X-Request-Base`.
`port`	`3000`	Loopback port nginx proxies to.
`stateDir`	`/var/lib/hydra`	Hydra state directory.
`dbName`	`hydra`	Postgres database + role (also the ident-map target).
`notificationSender`	`hydra@<domain>`	From-address for failure emails.
`acmeHost`	`null`	Reuse this ACME cert; if null, nginx requests its own for `domain`.
`buildMachinesFile`	`null`	Optional remote-builder list; null = build locally.
`maxOutputSize`	`8 GiB`	Per-output size cap before Hydra rejects a build.
`allowImportFromDerivation`	`true`	Enable IFD in nix + the Hydra evaluator.

Caveats¶

Postgres must be on the same host — the peer/ident auth is local-socket only. For a remote database you'd switch to scram-sha-256 and drop the ident map.
Hydra's three user names are fixed by upstream. Don't rename them; the ident map depends on them.
acmeHost = null requires nginx to be able to complete an ACME challenge (open :80, resolvable DNS). If your certs come from elsewhere, point acmeHost at the cert you already manage.
The module sets nginx.enable and postgresql.enable with mkDefault, so it coexists with an existing server config but won't fight an explicit = false.

Source¶

modules/hydra-ci-server/default.nix

# hydra-ci-server
#
# Self-host Hydra (the Nix-native CI/CD system) behind nginx with a
# declaratively-provisioned PostgreSQL backend. This module is written against
# *upstream* NixOS options only (services.hydra, services.postgresql,
# services.nginx), so it is a drop-in you can import anywhere.
#
# The value here is not the wiring — it is the five non-obvious traps it takes
# to make Hydra actually run behind a reverse proxy on a DB you provisioned
# yourself. Each is called out inline below. See README.md for the "why".
#
# Usage:
#   imports = [ ./hydra-ci-server ];
#   services.hydra-ci-server = {
#     enable = true;
#     domain = "hydra.example.com";
#   };

{
  config,
  lib,
  pkgs,
  ...
}:

let
  cfg = config.services.hydra-ci-server;
  inherit (lib)
    mkEnableOption
    mkIf
    mkOption
    mkBefore
    types
    optional
    ;

  # Hydra's three fixed system users (created by upstream services.hydra) plus
  # root (for out-of-band psql), all mapped to the single `dbName` DB role.
  # These OS user names are fixed by Hydra upstream — do not rename them.
  # The third column is the target PG role, so it tracks cfg.dbName.
  identMapName = "hydra";
  identMapLines = lib.concatStringsSep "\n" [
    "${identMapName} hydra               ${cfg.dbName}"
    "${identMapName} hydra-queue-runner  ${cfg.dbName}"
    "${identMapName} hydra-www           ${cfg.dbName}"
    "${identMapName} root                ${cfg.dbName}"
  ];
in
{
  options.services.hydra-ci-server = {
    enable = mkEnableOption "self-hosted Hydra CI/CD server behind nginx";

    domain = mkOption {
      type = types.str;
      example = "hydra.example.com";
      description = ''
        Public domain name for the Hydra web interface. Used both for the
        nginx virtual host and for the X-Request-Base header (see below) that
        Hydra uses to build absolute URLs.
      '';
    };

    stateDir = mkOption {
      type = types.str;
      default = "/var/lib/hydra";
      description = "Directory to store Hydra state.";
    };

    port = mkOption {
      type = types.port;
      default = 3000;
      description = "Loopback port Hydra listens on (nginx proxies to it).";
    };

    dbName = mkOption {
      type = types.str;
      default = "hydra";
      description = ''
        PostgreSQL database and role name. Hydra's system users are peer-mapped
        onto this role; changing it also changes the ident map target.
      '';
    };

    notificationSender = mkOption {
      type = types.str;
      default = "hydra@${cfg.domain}";
      description = "From-address Hydra uses for failure notification emails.";
    };

    acmeHost = mkOption {
      type = types.nullOr types.str;
      default = null;
      example = "example.com";
      description = ''
        If set, the nginx vhost reuses this ACME certificate
        (services.nginx.virtualHosts.<domain>.useACMEHost). If null, nginx
        requests its own certificate for `domain` via enableACME. Either way
        the vhost is forced to SSL, because Hydra's absolute-URL scheme is
        https.
      '';
    };

    buildMachinesFile = mkOption {
      type = types.nullOr types.path;
      default = null;
      description = ''
        Optional path to a Nix build-machines file listing remote builders
        Hydra may dispatch jobs to. Null = build only on the local machine.
      '';
    };

    maxOutputSize = mkOption {
      type = types.int;
      # 8 GiB. Raise for heavy builds (e.g. CUDA/ML closures) that otherwise
      # trip Hydra's "output limit exceeded".
      default = 8 * 1024 * 1024 * 1024;
      description = "Max size (bytes) of a single build output before Hydra rejects it.";
    };

    allowImportFromDerivation = mkOption {
      type = types.bool;
      default = true;
      description = ''
        Enable import-from-derivation (IFD) for evaluations that need it.
        Note: IFD evaluations cannot be gated by Hydra's `--no-build` pass, so
        jobsets that pull in IFD must be built rather than dry-evaluated.
      '';
    };
  };

  config = mkIf cfg.enable {
    services.hydra = {
      enable = true;
      hydraURL = "https://${cfg.domain}";
      listenHost = "127.0.0.1";
      port = cfg.port;
      notificationSender = cfg.notificationSender;
      buildMachinesFiles = optional (cfg.buildMachinesFile != null) cfg.buildMachinesFile;
      useSubstitutes = true;

      # TRAP 5 (IFD): Hydra evaluates in its own environment; the nix.conf
      # setting alone does not reach the evaluator, so pass it explicitly.
      extraEnv = mkIf cfg.allowImportFromDerivation {
        NIX_CONFIG = "allow-import-from-derivation = true";
      };

      extraConfig = ''
        max_output_size = ${toString cfg.maxOutputSize}
      '';
    };

    # ---- Declarative PostgreSQL backend --------------------------------------
    # The DB and role are provisioned here, NOT by hydra-init. See TRAP 4.
    services.postgresql = {
      enable = lib.mkDefault true;

      ensureDatabases = [ cfg.dbName ];
      ensureUsers = [
        {
          name = cfg.dbName;
          ensureDBOwnership = true;
        }
      ];

      # TRAP 1 (ident map): Hydra runs as three separate system users
      # (hydra, hydra-queue-runner, hydra-www). Peer auth authenticates by the
      # OS user name, so without a map only a role literally named
      # "hydra-queue-runner" etc. could connect. This map lets all three — plus
      # root, for manual psql — connect as the single `${cfg.dbName}` role.
      # (identMap is a plain string that appends to pg_ident.conf upstream.)
      identMap = identMapLines;

      # Peer-auth rule that activates the map above for local connections as
      # the hydra role. Appended to pg_hba.conf ahead of the default catch-all.
      authentication = mkBefore ''
        local all ${cfg.dbName} peer map=${identMapName}
      '';
    };

    # TRAP 2 (pg_trgm): Hydra's job/build search relies on trigram indexes.
    # Without the pg_trgm extension the schema init fails and Hydra will not
    # start evaluations. Upstream has no per-database setup hook, so create the
    # extension in a oneshot ordered before hydra-init.
    systemd.services.hydra-pg-trgm = {
      description = "Ensure pg_trgm extension exists in the Hydra database";
      after = [ "postgresql.service" ];
      requires = [ "postgresql.service" ];
      before = [ "hydra-init.service" ];
      requiredBy = [ "hydra-init.service" ];
      serviceConfig = {
        Type = "oneshot";
        RemainAfterExit = true;
        User = "postgres";
        Group = "postgres";
      };
      script = ''
        ${config.services.postgresql.package}/bin/psql -d ${cfg.dbName} \
          -c 'CREATE EXTENSION IF NOT EXISTS pg_trgm'
      '';
    };

    # ---- nginx reverse proxy -------------------------------------------------
    services.nginx = {
      enable = lib.mkDefault true;
      virtualHosts.${cfg.domain} = {
        forceSSL = true;
        useACMEHost = mkIf (cfg.acmeHost != null) cfg.acmeHost;
        enableACME = cfg.acmeHost == null;
        locations."/" = {
          proxyPass = "http://127.0.0.1:${toString cfg.port}";
          proxyWebsockets = true;
          extraConfig = ''
            proxy_set_header X-Forwarded-Host   $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto  $scheme;
            # TRAP 3 (X-Request-Base): Hydra builds every absolute URL
            # (redirects, links, notification bodies) from THIS header, not
            # from Host. Omit it and links/redirects break behind the proxy.
            proxy_set_header X-Request-Base     "https://${cfg.domain}";
          '';
        };
      };
    };

    systemd.tmpfiles.rules = [
      "d ${cfg.stateDir} 0755 hydra hydra -"
      "d /nix/var/nix/gcroots/hydra 0755 hydra hydra -"
    ];

    nix.settings = {
      trusted-users = [
        "hydra"
        "@hydra"
      ];
      allow-import-from-derivation = cfg.allowImportFromDerivation;
    };

    # TRAP 4 (.db-created sentinel): hydra-init bootstraps its own database on
    # first run — creating the role and DB and running its schema. Since we
    # already provisioned both declaratively above, pre-touch the sentinel it
    # checks so it skips that bootstrap (which would otherwise try to CREATE a
    # role/DB that already exist, and fail). mkBefore keeps this ahead of the
    # upstream preStart body.
    systemd.services.hydra-init.preStart = mkBefore ''
      mkdir -p ${cfg.stateDir}
      touch ${cfg.stateDir}/.db-created
    '';
  };
}