forgejo-bidirectional-safe-sync¶

Modules

A NixOS module that keeps two Forgejo instances mirrored to each other from a neutral third box — so neither forge depends on the other for backup or HA.

Each timer tick, the sync engine walks every repo on both sides via the Forgejo API and reconciles refs symmetrically:

fast-forward a ref wherever it is safe (one side strictly ahead of the other),
propagate brand-new repos and refs to the peer,
and alert on genuine divergence (both sides moved, neither is an ancestor of the other) instead of force-pushing over either side.

Why a third box?¶

If forge A backed up to forge B (or vice versa), losing one forge degrades the other's recovery story, and the natural "just push" fix is a force-push waiting to clobber real work. Running the reconciler on an independent host makes the relationship peer-to-peer: when either forge is down, the survivor still holds a complete copy, and the box quietly re-converges once the peer returns. This is exactly the machinery you want during an unplanned outage of one forge.

Safety guarantees¶

The reconciler is built to never destroy history. It:

issues no --force / --force-with-lease pushes — a non-fast-forward push is refused by Forgejo and logged;
never deletes refs — deleting a branch on one side does not propagate;
never deletes or renames repos — a repo present on one side but missing on the other (after having been seen on both) is flagged as rename-or-delete-suspected and skipped, not re-created or removed;
never auto-creates owners — if a new repo's owner doesn't exist on the target, it is skipped and alerted (optionally routed to a fallback org, see unownedReposTarget).

Worst-case failure mode: a stale ref on one side until the next cycle. True conflicts surface as structured diverged / alert-divergence log events; a human pushes a merge or rebase and the next cycle converges.

The load-bearing trap: `EnvironmentFile = "-…"`¶

The service materializes its config (including the admin password) into a tmpfs env file under /run via an ExecStartPre, then runs the sync engine with that file as its EnvironmentFile. (Both steps run unprivileged — see the next section.)

Because /run is tmpfs, the env file does not exist on the first cycle after boot, and systemd loads EnvironmentFile before it runs ExecStartPre (the step that creates it). If the path is listed without a leading -, the unit dies with Failed to load environment files before the pre-step can ever run — a permanent boot-time loop.

The fix is the single - optional-load prefix:

EnvironmentFile = "-/run/forgejo-bisync/env";

The first load no-ops, ExecStartPre writes the file, and ExecStart reads it. Keep that dash.

Nothing in this unit runs as root¶

ExecStartPre used to run with systemd's + prefix purely so it could read passwordFile and own the tmpfs env file it writes. Neither actually needs root: RuntimeDirectory = cfg.user already creates /run/forgejo-bisync owned by the sync user (replacing a manual install -d -o -g), and LoadCredential = "password:${passwordFile}" lets systemd's PID1 — still root at that point — read the secret on the unit's behalf and hand it over via $CREDENTIALS_DIRECTORY. That works no matter how restrictive passwordFile's own permissions are, so ExecStartPre and ExecStart both run as the unprivileged forgejo-bisync user throughout, and an assertion rejects user = "root" at build time.

Usage¶

{
  imports = [ ./forgejo-bidirectional-safe-sync ];

  services.forgejo-bisync = {
    enable = true;

    # Your reconciliation engine (see "The sync engine" below).
    syncScript = ./sync.ts;

    instances = [
      { name = "forge-a"; baseUrl = "https://forge-a.example.com"; }
      { name = "forge-b"; baseUrl = "https://forge-b.example.com"; }
    ];

    # An admin account that exists with the SAME password on BOTH forges.
    username = "bisync";
    passwordFile = config.age.secrets.bisync-password.path; # or sops/systemd-creds/etc.

    interval = "5min";
    # excludeOwners = [ "mirrors" ];
    # unownedReposTarget = "archive"; # org on the target for owner-less repos
  };
}

Run this module on a host that is neither forge. You are responsible for provisioning the username admin account (with passwordFile's password) on both Forgejo instances — a tiny declarative bootstrap per forge, or a manual admin user, both work. The account needs admin scope so the API can enumerate every repo and create repos on either side.

Options¶

Option	Default	Purpose
`enable`	`false`	Turn the timer + sync service on.
`syncScript`	(required)	Path to the reconciliation engine.
`interpreter`	`node --experimental-strip-types`	Argv prefix to run the script; `null` to exec it directly.
`instances`	(required)	Exactly two `{ name; baseUrl; }` forges. `name` is the git remote name and log label.
`user`	`forgejo-bisync`	System user/group and `/run` subdir name.
`username`	`bisync`	Admin login present on both forges.
`passwordFile`	(required)	File with the shared admin password.
`interval`	`5min`	`OnUnitActiveSec` between cycles.
`workDir`	`/var/lib/forgejo-bisync`	Bare clones + `state.json`.
`parallelism`	`1`	Repos processed concurrently.
`alertOnDivergeAfterMs`	`1800000`	Grace period before a diverged ref escalates to `alert-divergence`.
`excludeRepos`	`[ ]`	`owner/name` strings to skip.
`excludeOwners`	`[ ]`	Whole owners (users/orgs) to skip.
`unownedReposTarget`	`null`	Fallback org on the target for repos whose owner is missing there; `null` = skip + alert.

The sync engine¶

The module is deliberately just the packaging (system user, secret plumbing, hardened oneshot service, and timer). The reconciliation logic is an external script you supply via syncScript. It receives all of its configuration from environment variables in the tmpfs env file — no CLI args, no config file:

Env var	From option
`BISYNC_USERNAME`	`username`
`BISYNC_PASSWORD`	contents of `passwordFile`
`BISYNC_A_NAME` / `BISYNC_A_BASE_URL`	`instances` element 0
`BISYNC_B_NAME` / `BISYNC_B_BASE_URL`	`instances` element 1
`BISYNC_WORK_DIR`	`workDir`
`BISYNC_STATE_FILE`	`${workDir}/state.json`
`BISYNC_PARALLELISM`	`parallelism`
`BISYNC_ALERT_DIVERGE_MS`	`alertOnDivergeAfterMs`
`BISYNC_EXCLUDE_REPOS`	comma-joined `excludeRepos`
`BISYNC_EXCLUDE_OWNERS`	comma-joined `excludeOwners`
`BISYNC_UNOWNED_TARGET`	`unownedReposTarget` (empty when null)

A conformant engine, per cycle, does roughly:

Discover — GET /api/v1/repos/search on both instances (paginated) using HTTP Basic Auth with username:password. The admin account sees every repo.
For each repo present on both sides, fetch all heads and tags into a bare clone (one namespaced remote per instance), then per ref:
equal SHAs → nothing to do;
present on one side only → fast-forward push to the other;
both present, one is a strict ancestor of the other → fast-forward the behind side up to the ahead side;
both moved, neither an ancestor → diverged: log it, never force.
For a repo present on one side only — if never seen before, create it on the target (respecting unownedReposTarget) and seed every ref; if seen before, treat as a suspected rename/delete and skip.
Persist a small state.json (last-synced ref SHAs, divergedSince timestamps) so divergence can be aged before escalating to alert-divergence, and so a vanished repo can be distinguished from a brand-new one.

Emit one structured JSON log line per event (synced, ff-pushed, created, diverged, alert-divergence, skipped, error) so a log pipeline can alert on divergence.

Git-over-HTTPS gotchas worth keeping¶

Two settings the reference engine sets on every fetch/push, learned from large repos:

-c http.version=HTTP/1.1
-c http.postBuffer=1048576000

Multi-GB repositories fail over HTTP/2 with curl 92 stream reset by server / early EOF; forcing HTTP/1.1 avoids the multiplexed-stream reset, and the large postBuffer keeps big pushes from chunking into a server-rejected size.

Also: some Forgejo versions reject sort=newest on /repos/search with a 422 Invalid sort mode — omit the sort param, ordering is irrelevant when you collect results into a map.

Caveats¶

Exactly two instances, with distinct name values (both asserted).
The admin password is the same on both forges by design — it is the shared identity the reconciler authenticates as. Scope it to a dedicated bisync account, not a human admin.
The service is a oneshot with Restart=no; transient network/API failures are expected and simply retried on the next timer tick.
Runs hardened (ProtectSystem=strict, filtered syscalls, no new privileges); the only writable locations are workDir and the unit's own RuntimeDirectory under /run (where the env file is written, mode 0400).

Source¶

modules/forgejo-bidirectional-safe-sync/default.nix

# forgejo-bidirectional-safe-sync
#
# A NixOS module that bidirectionally mirrors two Forgejo instances from a
# neutral third box. Each timer tick walks every repo on both sides via the
# Forgejo API and reconciles refs *symmetrically*:
#
#   - fast-forward a ref wherever it is safe (one side strictly ahead),
#   - propagate brand-new repos/refs,
#   - and ALERT on genuine divergence instead of force-pushing over either side.
#
# Because the sync runs on a third host, neither forge depends on the other for
# backup / HA: if one goes down, the survivor keeps its full copy and the box
# reconciles once the peer returns.
#
# This module is only the packaging (system user, secret plumbing, timer +
# hardened oneshot service). The actual reconciliation engine is an external
# script you point `syncScript` at; the env-var contract it must consume is
# documented in the README and materialized by `envSetup` below.
{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.services.forgejo-bisync;

  # Runtime dir lives on tmpfs (/run) — see the EnvironmentFile trap below.
  runDir = "/run/${cfg.user}";
  envFile = "${runDir}/env";

  # Runs as cfg.user, not root: RuntimeDirectory already creates ${runDir}
  # owned by that user, and the secret is read through LoadCredential, which
  # lets systemd's PID1 (still root) read passwordFile on the unit's behalf
  # and hand it over via $CREDENTIALS_DIRECTORY -- so this works regardless of
  # whether passwordFile is readable by anyone but root, and nothing here
  # needs the `+` root-prefix it used to.
  envSetup = pkgs.writeShellScript "forgejo-bisync-env" ''
    set -eu
    PW=$(cat "$CREDENTIALS_DIRECTORY/password")
    umask 077
    cat > ${envFile} <<EOF
    BISYNC_USERNAME=${cfg.username}
    BISYNC_PASSWORD=$PW
    BISYNC_A_NAME=${(builtins.elemAt cfg.instances 0).name}
    BISYNC_A_BASE_URL=${(builtins.elemAt cfg.instances 0).baseUrl}
    BISYNC_B_NAME=${(builtins.elemAt cfg.instances 1).name}
    BISYNC_B_BASE_URL=${(builtins.elemAt cfg.instances 1).baseUrl}
    BISYNC_WORK_DIR=${cfg.workDir}
    BISYNC_STATE_FILE=${cfg.workDir}/state.json
    BISYNC_PARALLELISM=${toString cfg.parallelism}
    BISYNC_ALERT_DIVERGE_MS=${toString cfg.alertOnDivergeAfterMs}
    BISYNC_EXCLUDE_REPOS=${lib.concatStringsSep "," cfg.excludeRepos}
    BISYNC_EXCLUDE_OWNERS=${lib.concatStringsSep "," cfg.excludeOwners}
    BISYNC_UNOWNED_TARGET=${lib.optionalString (cfg.unownedReposTarget != null) cfg.unownedReposTarget}
    EOF
    chmod 0400 ${envFile}
  '';
in
{
  options.services.forgejo-bisync = {
    enable = lib.mkEnableOption "forgejo-bisync: bidirectional safe-sync between two Forgejos";

    syncScript = lib.mkOption {
      type = lib.types.path;
      description = ''
        Path to the reconciliation engine. It is invoked once per timer tick and
        reads its configuration entirely from the BISYNC_* environment variables
        materialized into the tmpfs env file (see README for the full contract).
        The default runner assumes a Node/TypeScript script executed with
        `node --experimental-strip-types`; override `interpreter` for anything
        else (e.g. a standalone executable or a Python script).
      '';
    };

    interpreter = lib.mkOption {
      type = lib.types.nullOr (lib.types.listOf lib.types.str);
      default = [
        "${pkgs.nodejs}/bin/node"
        "--experimental-strip-types"
      ];
      description = ''
        Argv prefix used to run `syncScript`. Set to null to execute the script
        directly (it must be executable and carry its own shebang).
      '';
    };

    instances = lib.mkOption {
      type = lib.types.listOf (
        lib.types.submodule {
          options = {
            name = lib.mkOption {
              type = lib.types.str;
              description = "Short name used in logs and as the git remote name. Lowercase, no spaces.";
              example = "forge-a";
            };
            baseUrl = lib.mkOption {
              type = lib.types.str;
              description = "https://… base URL of the Forgejo instance (no trailing slash).";
              example = "https://forge-a.example.com";
            };
          };
        }
      );
      description = "Exactly two Forgejo instances. Sync is symmetric.";
    };

    user = lib.mkOption {
      type = lib.types.str;
      default = "forgejo-bisync";
      description = "System user/group the sync service runs as. Also names the /run subdir.";
    };

    username = lib.mkOption {
      type = lib.types.str;
      default = "bisync";
      description = ''
        Admin user that exists on BOTH Forgejo instances with the SAME password.
        The daemon uses this username + passwordFile for HTTP Basic Auth against
        both instances (repo discovery via the API, and git over HTTPS). Give it
        an admin token/role so the API can enumerate every repo and create repos
        on either side. You are responsible for provisioning this identical
        account on both forges (e.g. a small declarative bootstrap on each host).
      '';
    };

    passwordFile = lib.mkOption {
      type = lib.types.path;
      description = ''
        Path to a file containing the shared admin password. Use whatever secret
        manager you like (agenix, sops-nix, systemd credentials, …); the module
        only needs a readable path. The SAME password must be set for `username`
        on both Forgejo instances.
      '';
    };

    interval = lib.mkOption {
      type = lib.types.str;
      default = "5min";
      description = "systemd OnUnitActiveSec — how often to run a sync cycle.";
    };

    workDir = lib.mkOption {
      type = lib.types.str;
      default = "/var/lib/forgejo-bisync";
      description = "Working directory (bare clones + state.json).";
    };

    parallelism = lib.mkOption {
      type = lib.types.int;
      default = 1;
      description = "Repos processed concurrently. The reference engine is sequential (1).";
    };

    alertOnDivergeAfterMs = lib.mkOption {
      type = lib.types.int;
      default = 30 * 60 * 1000;
      description = "Emit `alert-divergence` only after a ref has been diverged this long (ms).";
    };

    excludeRepos = lib.mkOption {
      type = lib.types.listOf lib.types.str;
      default = [ ];
      example = [ "owner/private-repo" ];
      description = "owner/name strings to skip entirely.";
    };

    excludeOwners = lib.mkOption {
      type = lib.types.listOf lib.types.str;
      default = [ ];
      example = [ "mirrors" ];
      description = "Whole owners (users or orgs) to skip.";
    };

    unownedReposTarget = lib.mkOption {
      type = lib.types.nullOr lib.types.str;
      default = null;
      description = ''
        If a source-side repo's owner doesn't exist on the target side, create
        the propagated repo under THIS org instead. Must be an existing org on
        the target. Default null = skip + alert on owner mismatch.
      '';
    };
  };

  config = lib.mkIf cfg.enable {
    assertions = [
      {
        assertion = builtins.length cfg.instances == 2;
        message = "services.forgejo-bisync.instances must list exactly 2 Forgejo instances.";
      }
      {
        assertion = (builtins.elemAt cfg.instances 0).name != (builtins.elemAt cfg.instances 1).name;
        message = "forgejo-bisync: the two instances must have distinct `name` values.";
      }
      {
        assertion = cfg.user != "root";
        message = ''
          services.forgejo-bisync.user must not be root. The sync cycle reads its
          shared password via a systemd credential and never needs privilege.
        '';
      }
    ];

    users.users.${cfg.user} = {
      isSystemUser = true;
      group = cfg.user;
      home = cfg.workDir;
      description = "forgejo-bisync sync daemon";
    };
    users.groups.${cfg.user} = { };

    systemd.tmpfiles.rules = [
      "d ${cfg.workDir}        0750 ${cfg.user} ${cfg.user} - -"
      "d ${cfg.workDir}/work   0750 ${cfg.user} ${cfg.user} - -"
    ];

    systemd.services.forgejo-bisync = {
      description = "Forgejo bidirectional safe-sync cycle";
      after = [ "network-online.target" ];
      wants = [ "network-online.target" ];
      path = [
        pkgs.git
        pkgs.coreutils
        pkgs.openssh
      ];
      serviceConfig = {
        Type = "oneshot";
        User = cfg.user;
        Group = cfg.user;
        # RuntimeDirectory replaces the manual `install -d -o -g` that used to
        # need root; LoadCredential replaces the root-only `cat` of
        # passwordFile. Neither ExecStartPre nor ExecStart runs privileged.
        RuntimeDirectory = cfg.user;
        RuntimeDirectoryMode = "0750";
        LoadCredential = "password:${cfg.passwordFile}";
        ExecStartPre = [ "${envSetup}" ];
        ExecStart = lib.concatStringsSep " " (
          (lib.optionals (cfg.interpreter != null) cfg.interpreter) ++ [ (toString cfg.syncScript) ]
        );
        # NOTE the leading `-`: optional-load. /run is tmpfs, so on the first
        # cycle after boot the env file does not exist yet, and systemd loads
        # EnvironmentFile BEFORE running ExecStartPre (which creates it).
        # Without the `-`, the unit dies with "Failed to load environment files"
        # before envSetup can run — a permanent boot-time loop. The `-` lets the
        # first load no-op; envSetup then writes the file for ExecStart to read.
        EnvironmentFile = "-${envFile}";
        WorkingDirectory = cfg.workDir;

        ProtectSystem = "strict";
        ProtectHome = true;
        ReadWritePaths = [ cfg.workDir ];
        PrivateTmp = true;
        NoNewPrivileges = true;
        ProtectKernelTunables = true;
        ProtectKernelModules = true;
        ProtectControlGroups = true;
        RestrictAddressFamilies = [
          "AF_INET"
          "AF_INET6"
          "AF_UNIX"
        ];
        SystemCallFilter = [
          "@system-service"
          "~@privileged"
          "~@resources"
        ];
        # Transient network/API failures are expected and fine: no Restart, the
        # next timer tick retries. A stale ref survives at most one interval.
        Restart = "no";
      };
    };

    systemd.timers.forgejo-bisync = {
      description = "Periodic Forgejo bidirectional safe-sync";
      wantedBy = [ "timers.target" ];
      timerConfig = {
        OnBootSec = "2min";
        OnUnitActiveSec = cfg.interval;
        AccuracySec = "30s";
        Unit = "forgejo-bisync.service";
      };
    };
  };
}