skip-flaky-tests-overlay

Overlays

A small, reusable nixpkgs overlay toolbox for surgically disabling package checks that fail for reasons unrelated to the package being broken — most commonly test suites that time out, OOM, or outright crash the CPU emulator when you build for a foreign architecture under qemu-user (binfmt cross builds), or that flake under a heavily loaded parallel builder.

The value here is not the package list — that is pinned to whatever nixpkgs revision you happen to be on. The value is four helper functions and the ordering/import-check traps they encode.

The problem

When you build a package whose upstream binary you can't (or won't) use — a foreign-arch build running under qemu-user, a source build on a busy builder — the checkPhase becomes a liability:

• suites time out because the emulator is 10-50x slower than native,
• pytest gets OOM-killed at high job counts,
• native extension modules trip spurious emulator debug assertions,
• and none of it indicates an actual regression in the package.

You want to drop the offending check while keeping the build, and do it with the smallest possible attribute change (see the caveat about hashes below).

The helpers

Helper	What it does	Use when
skipChecks pkg	clears doCheck + doInstallCheck via overrideAttrs	a C / meson / cargo / generic suite is flaky or slow
skipPyChecks pkg	same, via overridePythonAttrs	a Python package's whole pytest run is the problem
skipPyAllChecks pkg	also clears pythonImportsCheck and disables the import hook	the native module import itself crashes (see trap)
disablePyTests [names] pkg	appends to disabledTests (pytest -k deselection)	only one or two named tests are flaky — keep the rest

Prefer disablePyTests over skipPyChecks whenever a single test is the culprit: you keep coverage for everything else.

The key trap: pythonImportsCheck runs before pytest

buildPythonPackage has a pythonImportsCheckHook that runs during installCheck and imports every module listed in pythonImportsCheck as a smoke test. That import happens before your test suite runs.

So for a package whose native extension module crashes on import — e.g. a compiled extension that trips an emulator assertion the moment it's loaded — setting doCheck = false is not enough. The crash fires inside the import hook, not in pytest, and your build still fails. You have to clear the import list and disable the hook as well. That is exactly what skipPyAllChecks does, and why it exists as a separate helper:

skipPyAllChecks = pkg:
  pkg.overridePythonAttrs (_: {
    doCheck = false;
    doInstallCheck = false;
    pythonImportsCheck = [ ];        # nothing for the hook to import
    dontUsePythonImportsCheck = true; # and drop the hook itself
  });

The canonical example: a plotting extension whose native import aborts the emulator, plus a second package that imports it during its own import check — both need skipPyAllChecks, and skipPyChecks silently fails to fix either.

Usage

Import it as a nixpkgs overlay:

# plain nixpkgs
import <nixpkgs> {
  overlays = [ (import ./skip-flaky-tests-overlay) ];
}

# NixOS module
{ ... }: {
  nixpkgs.overlays = [ (import ./skip-flaky-tests-overlay) ];
}

# flake
pkgs = import nixpkgs {
  inherit system;
  overlays = [ (import ./skip-flaky-tests-overlay) ];
};

Then replace the illustrative entries in default.nix with your own. Each entry should carry a one-line comment saying why it needs the skip — a skip without a reason is impossible to retire later.

Notes on the non-Python examples in default.nix:

• meson: some packages compile their test binaries behind a build flag (-Dtests=true) independently of doCheck. Flip the flag too, or you still pay to compile tests you never run.
• cmake: you can usually exclude a single failing target (EXCLUDE_TESTS=...) instead of the whole suite.
• postPatch rename: when a package exposes no deselection knob, renaming test_foo → no_test_foo in-source makes the collector skip it.
• compose: helpers return a package, so you can chain a further .overridePythonAttrs to also wire back a missing dependency.

Caveat: overriding changes the hash

Any overrideAttrs / overridePythonAttrs changes the derivation's output hash, so the result no longer matches the upstream binary cache and will be built from source everywhere it's referenced. That is fine — often necessary — for packages you are already building locally. It is a trap for packages you were getting from the cache for free: overriding one of those turns a download into a full (possibly emulated, possibly brutal) source build across your whole fleet.

Rule of thumb: only reach for these helpers on packages that were going to be built from source anyway. Leave cache-hitting packages untouched.

When to retire an entry

These skips are load-bearing but temporary. The right fix is usually upstream (a patched emulator, a fixed test, a nixpkgs bump) — the overlay just unblocks you meanwhile. Keep the per-entry "why" comments so that when you bump nixpkgs you can tell which skips are now obsolete and drop them.

Source

overlays/skip-flaky-tests-overlay/default.nix

# A nixpkgs overlay: a small toolbox for surgically disabling package checks
# that fail under CPU emulation (qemu-user cross builds) or heavy parallel
# builder load — without perturbing the derivation hash more than necessary.
#
# Import it as a nixpkgs overlay, e.g.:
#
#   nixpkgs.overlays = [ (import ./skip-flaky-tests-overlay) ];
#
# or in a flake:
#
#   pkgs = import nixpkgs { inherit system; overlays = [ (import ./skip-flaky-tests-overlay) ]; };
#
# The entries below the helpers are ILLUSTRATIVE examples only. Delete them and
# add your own — the reusable value is the four helper functions and the
# ordering/import-check traps they encode, not this particular package list.
#
# WARNING: dropping doCheck (or otherwise mutating attrs) changes the output
# hash, so the package no longer matches the upstream binary cache and rebuilds
# from source everywhere. Only override packages you are already building
# locally; leave cache-hitting packages untouched.

_: prev:
let
  # Drop the standard check + installCheck phases (C / meson / cargo / generic).
  skipChecks =
    pkg:
    pkg.overrideAttrs (_: {
      doCheck = false;
      doInstallCheck = false;
    });

  # Same, for a Python package (uses overridePythonAttrs so buildPythonPackage
  # picks the change up correctly).
  skipPyChecks =
    pkg:
    pkg.overridePythonAttrs (_: {
      doCheck = false;
      doInstallCheck = false;
    });

  # The important one. Some packages have a native extension module whose
  # *import* crashes the emulator (or otherwise aborts) before pytest ever
  # runs. buildPythonPackage's pythonImportsCheckHook imports every module in
  # `pythonImportsCheck` during installCheck — so clearing doCheck alone is not
  # enough; the crash fires in the import hook, not in the test suite. This
  # helper clears the import list AND disables the hook as well.
  skipPyAllChecks =
    pkg:
    pkg.overridePythonAttrs (_: {
      doCheck = false;
      doInstallCheck = false;
      pythonImportsCheck = [ ];
      dontUsePythonImportsCheck = true;
    });

  # Keep the suite, drop only specific flaky tests by name/pattern (pytest -k
  # deselection via nixpkgs' disabledTests). Preferred over skipPyChecks when a
  # single test is the problem — you keep coverage for everything else.
  disablePyTests =
    tests: pkg:
    pkg.overridePythonAttrs (old: {
      disabledTests = (old.disabledTests or [ ]) ++ tests;
    });
in
{
  # ---- Non-Python examples -------------------------------------------------

  # A C/meson package whose suite times out or is flaky under emulation.
  age = skipChecks prev.age;
  libsecret = skipChecks prev.libsecret;

  # Some meson packages gate tests behind a build flag as well as doCheck —
  # flip the flag too, or the test binaries still get compiled (slow under
  # emulation) even though they are never run.
  power-profiles-daemon = prev.power-profiles-daemon.overrideAttrs (old: {
    doCheck = false;
    doInstallCheck = false;
    mesonFlags = builtins.map (
      f: if f == "-Dtests=true" then "-Dtests=false" else f
    ) (old.mesonFlags or [ ]);
  });

  # CMake packages often let you exclude a single failing test target rather
  # than the whole suite.
  thrift = prev.thrift.overrideAttrs (old: {
    cmakeFlags = (old.cmakeFlags or [ ]) ++ [
      (prev.lib.cmakeFeature "EXCLUDE_TESTS" "TServerIntegrationTest")
    ];
    disabledTests = (old.disabledTests or [ ]) ++ [ "TServerIntegrationTest" ];
    doCheck = false;
  });

  # Rename a single failing test in-source via postPatch (turns test_foo into
  # no_test_foo so the collector skips it) when the package has no clean
  # deselection knob.
  kitty = prev.kitty.overrideAttrs (old: {
    postPatch = (old.postPatch or "") + ''
      substituteInPlace kitty_tests/check_build.py \
        --replace-quiet test_macos_dictation_forwarding no_test_macos_dictation_forwarding
    '';
  });

  # ---- Python examples -----------------------------------------------------
  #
  # Extend every python interpreter's package set. Using
  # pythonPackagesExtensions (rather than overriding a single pythonPackages)
  # applies the fix across python3Packages, the vendored sets inside other
  # packages, etc.
  pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
    (_: pprev: {
      # Whole-suite skip: OOM-killed / aborts under load.
      aiohttp = skipPyChecks pprev.aiohttp;
      twisted = skipPyChecks pprev.twisted;

      # Single-test deselection — keeps the rest of the coverage.
      rich = disablePyTests [ "test_brokenpipeerror" ] pprev.rich;
      dulwich = disablePyTests [
        "test_no_decode_encode"
        "test_cyrillic"
      ] pprev.dulwich;

      # THE TRAP: contourpy's native import crashes the emulator itself, and
      # matplotlib imports contourpy during its own import check — both need
      # the import hook cleared, not just doCheck. This is exactly the case
      # skipPyChecks does NOT cover and skipPyAllChecks does.
      contourpy = skipPyAllChecks pprev.contourpy;
      matplotlib = skipPyAllChecks pprev.matplotlib;

      # Compose helpers with a further override when a package needs both a
      # check skip and a missing dependency wired back in.
      slicer = (skipPyChecks pprev.slicer).overridePythonAttrs (old: {
        build-system = (old.build-system or [ ]) ++ [ pprev.setuptools ];
        nativeBuildInputs = (old.nativeBuildInputs or [ ]) ++ [ pprev.setuptools ];
      });

      # Runtime-deps metadata check failing because nixpkgs didn't wire a
      # wheel-declared dep: either add the dep back...
      shap = pprev.shap.overridePythonAttrs (old: {
        dependencies = (old.dependencies or [ ]) ++ [ pprev.typing-extensions ];
      });
      # ...or, when the dep genuinely isn't needed at build time, just turn
      # the runtime-deps check off.
      outlines = pprev.outlines.overridePythonAttrs (_: {
        dontCheckRuntimeDeps = true;
      });
    })
  ];
}