patroni-leader-proxy

Modules

A NixOS module: a local HAProxy that gives PostgreSQL clients a fixed local endpoint that always lands on the current Patroni leader. Failover needs no client reconfiguration, no DNS change, no restart.

The problem

You run PostgreSQL in HA with Patroni. Patroni elects a leader and, on failure, promotes a replica — but the leader is now a different host. Every write client has to find the new one. Chasing that with DNS TTLs, floating VIPs, or client-side multi-host connection strings is either slow, fiddly, or unsupported by your driver.

The insight

Patroni already exposes a REST API that answers, per node, "am I the leader?":

• GET /primary → 200 only on the current leader, 503 otherwise
• GET /replica → 200 only on a running replica

So point HAProxy in TCP mode at the raw PostgreSQL port of every node, but use the REST API as the health check. The read-write pool then holds exactly one "healthy" server — the leader — and HAProxy re-points to a newly promoted leader within inter × fall seconds. Clients only ever talk to 127.0.0.1:<port>; the proxy quietly follows the leader around the cluster.

An optional read-only pool health-checks /replica and round-robins across live replicas.

  client ──▶ 127.0.0.1:5432 (HAProxy, this host)
                 │  TCP-forwards PG port
                 │  health-checks Patroni REST /primary
                 ├─▶ pg1:5432   (/primary → 200  ← leader, gets traffic)
                 ├─▶ pg2:5432   (/primary → 503  ← replica, no traffic)
                 └─▶ pg3:5432   (/primary → 503  ← replica, no traffic)

Usage

{
  imports = [ ./modules/patroni-leader-proxy ];

  services.patroni-leader-proxy = {
    enable = true;
    nodes = {
      pg1 = "10.0.0.11";
      pg2 = "10.0.0.12";
      pg3 = "10.0.0.13";
    };
    # readPort = 5433;              # optional round-robin replica pool
  };
}

Clients connect to 127.0.0.1:5432 for writes (always the leader) and, if you set readPort, 127.0.0.1:5433 for reads (any live replica).

Key options

Option	Default	Purpose
nodes	(required)	name -> address of every Patroni member.
pgPort	5432	PostgreSQL port on each node (forwarded to).
restApiPort	8008	Patroni REST API port (health-checked).
port	5432	Local read-write listen port.
readPort	null	Local read-only (replica round-robin) port; off by default.
bindAddresses	[ "127.0.0.1" ]	Where HAProxy listens (see below).
extraAfterUnits	[ ]	Extra after= units, e.g. a VPN (see below).
checkInter / checkTimeout / checkFall / checkRise	5s / 8s / 5 / 2	Health-check tuning (see below).

Traps and tunings (do not naively "tighten")

Servers start DOWN, on purpose

HAProxy's default is to consider a health-checked server UP until a check proves otherwise. For a pool whose entire job is "only the leader", that default is backwards: for up to checkInter × checkFall after HAProxy starts — 25 seconds at the defaults here — every node is in the RW pool, and a write can be round-robined onto a replica.

default-server init-state down inverts it: a node joins the pool only after checkRise successful checks say it is the primary. The cost is that the RW port refuses connections for up to one checkInter after a proxy restart. That is the right trade: a brief, obvious outage beats a silent write to a replica.

This requires HAProxy 3.1 or newer. On older builds HAProxy rejects the unknown keyword and refuses to start, which is at least a loud failure rather than a quiet misroute.

WAN-tolerant timings. If the proxy health-checks nodes across a high-latency link (a stretched cross-region cluster), a healthy /primary check can take 1–2s over a ~150ms RTT. A tight inter 3s / timeout check 3s flaps the whole pool DOWN on every jitter spike — dropping all writes for no reason. The defaults here are inter 5s fall 5 rise 2 (≈25s to mark a node down) with timeout check 8s. Raise them further for slower links; do not lower them because the LAN case "looks fine".

Live connections survive blips. HAProxy's on-marked-down shutdown-sessions is deliberately not set. On a transient health-check failure you want existing PostgreSQL connections to survive, not be killed and re-pooled. (The leader itself hasn't moved — only a check timed out.)

Boot ordering when checks ride a VPN. If the Patroni nodes are reachable only over an overlay network (Tailscale, WireGuard, ...), HAProxy starting before that interface is up trips every server to "No route to host" and leaves the RW pool empty for ~30s until checks recover. Pass the overlay's unit via extraAfterUnits = [ "tailscaled.service" ]; (or your WireGuard unit). network-online.target is always ordered before HAProxy.

Log-noise suppression. option dontlog-normal drops the clean-termination line every PostgreSQL connection emits. On a busy host this is tens of thousands of lines per boot that otherwise drown the DOWN/UP/retry events you actually care about.

Binding for containers / VMs

bindAddresses defaults to loopback, for host-local consumers. To let containers or microvms on the same host reach the proxy, add the bridge / VM gateway IP:

services.patroni-leader-proxy.bindAddresses = [ "127.0.0.1" "172.20.0.1" ];

If that bridge IP may not exist yet when HAProxy starts, allow non-local binds:

boot.kernel.sysctl."net.ipv4.ip_nonlocal_bind" = 1;

Notes

• This is the write-path companion to your Patroni setup — Patroni elects the leader; this module finds it. It does not manage PostgreSQL or Patroni itself.
• Running the proxy on the same host as a Patroni member is fine — just set port to something other than 5432 (e.g. 15432) so it doesn't collide with the local PostgreSQL.
• A Patroni member tagged nofailover still answers /replica while it streams, so it will appear in a readPort pool. Exclude it from nodes if you don't want reads routed there.

Source

modules/patroni-leader-proxy/default.nix

# patroni-leader-proxy
#
# A local HAProxy that gives PostgreSQL clients a fixed endpoint which always
# lands on the CURRENT Patroni leader. HAProxy TCP-forwards the PG port but
# health-checks Patroni's REST API: `GET /primary` returns 200 only on the
# leader and `GET /replica` returns 200 only on running replicas. The RW pool
# therefore holds exactly one live server and re-points to a new leader within
# `(inter x fall)` seconds of a failover — no client reconfig, DNS, or restart.
#
# Import it, set `nodes` to your Patroni members, enable it, and point clients
# at 127.0.0.1:<port>.
{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.services.patroni-leader-proxy;

  serverLines = lib.concatStringsSep "\n        " (
    lib.mapAttrsToList (
      name: addr: "server ${name} ${addr}:${toString cfg.pgPort} check port ${toString cfg.restApiPort}"
    ) cfg.nodes
  );

  bindLines =
    port: lib.concatStringsSep "\n        " (map (addr: "bind ${addr}:${toString port}") cfg.bindAddresses);

  rwBlock = ''
    listen patroni-rw
        ${bindLines cfg.port}
        option httpchk
        http-check send meth GET uri /primary
        http-check expect status 200
        timeout check ${cfg.checkTimeout}
        default-server init-state down inter ${cfg.checkInter} fall ${toString cfg.checkFall} rise ${toString cfg.checkRise}
        ${serverLines}
  '';

  roBlock = lib.optionalString (cfg.readPort != null) ''

    listen patroni-ro
        ${bindLines cfg.readPort}
        balance roundrobin
        option httpchk
        http-check send meth GET uri /replica
        http-check expect status 200
        timeout check ${cfg.checkTimeout}
        default-server init-state down inter ${cfg.checkInter} fall ${toString cfg.checkFall} rise ${toString cfg.checkRise}
        ${serverLines}
  '';
in
{
  options.services.patroni-leader-proxy = {
    enable = lib.mkEnableOption "Local HAProxy that routes PostgreSQL to the current Patroni leader";

    nodes = lib.mkOption {
      type = lib.types.attrsOf lib.types.str;
      example = {
        pg1 = "10.0.0.11";
        pg2 = "10.0.0.12";
        pg3 = "10.0.0.13";
      };
      description = ''
        The Patroni members, as an attrset of `name -> address`. `name` is the
        HAProxy server label (shown in logs / stats); `address` is the host or
        IP where that node's PostgreSQL and Patroni REST API listen. Every node
        appears in both the RW and RO pools — Patroni's REST API decides which
        one is live for each role, so you never edit this on failover.
      '';
    };

    pgPort = lib.mkOption {
      type = lib.types.port;
      default = 5432;
      description = "Port each Patroni node's PostgreSQL listens on (forwarded to by HAProxy).";
    };

    restApiPort = lib.mkOption {
      type = lib.types.port;
      default = 8008;
      description = "Port each Patroni node's REST API listens on (used for the role health check).";
    };

    port = lib.mkOption {
      type = lib.types.port;
      default = 5432;
      description = ''
        Local TCP port for read-write (leader) connections. Set this to
        something other than 5432 (e.g. 15432) if this host itself runs a
        PostgreSQL/Patroni that already owns 5432.
      '';
    };

    readPort = lib.mkOption {
      type = lib.types.nullOr lib.types.port;
      default = null;
      example = 5433;
      description = "If set, a local TCP port that round-robins across running replicas.";
    };

    bindAddresses = lib.mkOption {
      type = lib.types.listOf lib.types.str;
      default = [ "127.0.0.1" ];
      description = ''
        Addresses HAProxy binds the read-write (and read) pools on. Defaults to
        loopback for host-local consumers. Add bridge / VM gateway IPs (e.g.
        "172.20.0.1" or "192.168.121.1") so containers and microvms on this host
        can reach the proxy. Binding a not-yet-existing bridge IP requires
        `boot.kernel.sysctl."net.ipv4.ip_nonlocal_bind" = 1;`.
      '';
    };

    checkInter = lib.mkOption {
      type = lib.types.str;
      default = "5s";
      description = ''
        HAProxy `inter` — how often the Patroni REST health check runs.
        Combined with `checkFall`, a node is marked down after
        `checkInter x checkFall`. Keep this generous over high-latency /
        cross-region links: a too-tight `inter` flaps the pool down on jitter.
      '';
    };

    checkTimeout = lib.mkOption {
      type = lib.types.str;
      default = "8s";
      description = ''
        HAProxy `timeout check`. Must comfortably exceed the worst-case latency
        of a `/primary` REST response over your slowest link (a healthy check
        across a ~150ms RTT WAN can take 1-2s).
      '';
    };

    checkFall = lib.mkOption {
      type = lib.types.int;
      default = 5;
      description = "Consecutive failed checks before a node is marked down (see `checkInter`).";
    };

    checkRise = lib.mkOption {
      type = lib.types.int;
      default = 2;
      description = "Consecutive successful checks before a node is marked up again.";
    };

    extraAfterUnits = lib.mkOption {
      type = lib.types.listOf lib.types.str;
      default = [ ];
      example = [ "tailscaled.service" ];
      description = ''
        Extra systemd units to order the HAProxy service `after`. If the health
        checks reach the Patroni nodes over a VPN / overlay network (Tailscale,
        WireGuard, ...), add that unit here: starting before the overlay is up
        trips every server to "No route to host" and leaves the RW pool empty
        for ~30s until checks recover. `network-online.target` is always included.
      '';
    };
  };

  config = lib.mkIf cfg.enable {
    services.haproxy = {
      enable = true;
      config = ''
        global
            maxconn 2000
            log /dev/log local0

        defaults
            mode tcp
            log global
            option tcplog
            option dontlog-normal
            timeout connect 10s
            timeout client 1h
            timeout server 1h

        ${rwBlock}${roBlock}
      '';
    };

    systemd.services.haproxy = {
      after = [ "network-online.target" ] ++ cfg.extraAfterUnits;
      wants = [ "network-online.target" ];
    };
  };
}