Package mlx-vlm on Nix Without the Torch/OpenCV Chain

Packages

Apple's mlx-vlm runs vision-language models on Apple Silicon via MLX. Packaging it for Nix is mostly routine buildPythonPackage work — except for one trap that, if you don't handle it, drags a huge, mostly-useless dependency chain into the closure.

The problem

Recent versions of transformers resolve an AutoVideoProcessor whenever a processor is constructed for a VLM. Building that video processor imports torch, torchvision, and (transitively) opencv-python. For image-plus-text inference you never process a single frame of video — but the import happens anyway, at model-load time, purely as a side effect of AutoProcessor.from_pretrained(...).

On Nix this is worse than a fat wheel: torch + torchvision from PyPI want their own binary provenance and CUDA-shaped assumptions, and opencv-python fights with the C++ opencv4 that nixpkgs already ships. You end up either patching a wheel or building a second OpenCV.

The insight

You don't need the video processor to exist — you need it to not run. So monkey-patch it out at build time, surgically, around the one call that triggers it:

1. Wrap the single processor = AutoProcessor.from_pretrained(...) line in mlx_vlm/utils.py.
2. Just before it, replace AutoVideoProcessor.from_pretrained with a no-op that returns None, and relax ProcessorMixin.check_argument_for_proper_class so it accepts that None instead of type-checking the absent processor.
3. Immediately after the call, restore both originals, so nothing else in the running process is affected.

With the video path neutered, the torch / torchvision / opencv-python chain is never imported, and you can supply cv2 from the nixpkgs opencv4 C++ build while stripping the PyPI opencv-python pin (pythonRemoveDeps).

Why the patch is deliberately narrow

The replacement targets exactly one source line via substituteInPlace ... --replace-fail. Two reasons:

• Fail loud on upstream drift. --replace-fail errors the build if that line ever changes shape in a new mlx-vlm release, instead of silently patching nothing and leaving you to discover the torch import at runtime.
• Easy to re-audit. A one-line surgical wrap is trivial to eyeball after a version bump.

The indentation gotcha

The replaced statement lives inside an indented function body. Every injected line after the first therefore carries a hard-coded four-space indent in the Nix string — the first line inherits the original statement's indentation (it takes its place), the rest must supply their own. Drop that indentation and you get a Python IndentationError at import time, not at build time.

Usage

# Simplest: let callPackage wire the arguments.
mlx-vlm = pkgs.callPackage ./default.nix { };

# If you need an mlx-lm other than the one nixpkgs ships, override that
# derivation rather than re-packaging it from PyPI:
mlx-vlm = import ./default.nix {
  inherit (pkgs) python3Packages;
  mlx-lm = pkgs.python3Packages.mlx-lm.overridePythonAttrs (old: rec {
    version = "0.31.4";
    src = old.src.override {
      tag = "v${version}";
      hash = "sha256-...";
    };
  });
};

Options

Argument	Default	Purpose
python3Packages	—	The Python package set to build against.
mlx-lm	python3Packages.mlx-lm	mlx-vlm's core runtime; pass an overridePythonAttrs of the nixpkgs package if you need a different release.
version	"0.4.2"	PyPI release to fetch.
hash	sha256 for 0.4.2	Override when you bump version (run the build once, copy the expected hash).

To run the bundled OpenAI-compatible server, wrap it:

pkgs.writeShellScriptBin "mlx-vlm-server" ''
  exec ${pkgs.python3.withPackages (_: [ mlx-vlm ])}/bin/python -m mlx_vlm.server "$@"
''

Caveats

• Video inference is gone by design. This package is for image + text. If you need video, don't apply this patch — take the torch/torchvision cost.
• Version-coupled patch. The wrapped line is specific to the packaged mlx-vlm version. On a bump, expect --replace-fail to catch a changed line; re-point old/new at the new call site.
• opencv4, not opencv-python. The nixpkgs C++ OpenCV satisfies the cv2 import; the pythonRemoveDeps = [ "opencv-python" ] line keeps the PyPI pin from re-introducing the wheel.
• Apple Silicon only. MLX targets Metal; this builds and runs on macOS aarch64.

Source

packages/mlx-vlm-nix-package/default.nix

# Packaging Apple MLX's vision-language library (mlx-vlm) for Nix.
#
# The interesting part is the `postPatch` block: it monkey-patches
# transformers' AutoVideoProcessor at *build time* so that loading a model
# never tries to construct a video processor. That single edit lets us drop
# the entire torch / torchvision / opencv-python dependency chain that the
# video path would otherwise drag in. See README.md for the full why.
#
# Usage (called with the standard buildPythonPackage convention):
#
#   mlx-vlm = pkgs.callPackage ./default.nix { };
#   # or, if you need an mlx-lm other than the one nixpkgs ships:
#   mlx-vlm = import ./default.nix {
#     inherit (pkgs) python3Packages;
#     mlx-lm = python3Packages.mlx-lm.overridePythonAttrs (old: { /* ... */ });
#   };

{
  python3Packages,
  # mlx-vlm depends on mlx-lm, which nixpkgs ships as python3Packages.mlx-lm.
  # Override that derivation and pass it here if you need a different build;
  # re-packaging it from PyPI just to change a version is not worth the
  # hand-maintained dependency list and hash.
  mlx-lm ? python3Packages.mlx-lm,
  version ? "0.4.2",
  # sha256 of the PyPI sdist for the given version. Override when you bump.
  hash ? "sha256-MchLQyHI8XzssEV/oY1cBomCCmavGRkm0kDn35dWRT4=",
}:

python3Packages.buildPythonPackage rec {
  pname = "mlx-vlm";
  inherit version;
  pyproject = true;

  src = python3Packages.fetchPypi {
    pname = "mlx_vlm";
    inherit version hash;
  };

  build-system = [ python3Packages.setuptools ];

  dependencies = with python3Packages; [
    mlx-lm
    mlx
    numpy
    transformers
    pillow
    requests
    fastapi
    uvicorn
    tqdm
    datasets
    soundfile
    miniaudio
    # opencv4 (the nixpkgs C++ build) satisfies mlx-vlm's cv2 import without
    # pulling opencv-python (which would want a wheel + its own deps).
    opencv4
  ];

  # Upstream pins opencv-python from PyPI; we satisfy cv2 via nixpkgs opencv4
  # instead, so strip the PyPI pin from the metadata.
  pythonRemoveDeps = [ "opencv-python" ];

  # --- The load-bearing trap -------------------------------------------------
  # transformers >= 4.5x resolves an AutoVideoProcessor whenever a processor is
  # built for a VLM. Constructing that video processor imports torch /
  # torchvision (and, transitively, opencv-python). For image + text inference
  # we never touch video, so we neutralise AutoVideoProcessor.from_pretrained
  # for the duration of the AutoProcessor call, then restore it.
  #
  # We wrap only the single `AutoProcessor.from_pretrained(...)` line so the
  # patch is surgical and easy to re-verify after an upstream bump. If a future
  # mlx-vlm release changes that line, `--replace-fail` makes the build fail
  # loudly instead of silently no-op'ing.
  #
  # NOTE the leading four-space indentation baked into every line *after* the
  # first: the replaced statement lives inside an indented function body, so the
  # injected statements must carry that indentation to stay syntactically valid.
  postPatch =
    let
      old = "processor = AutoProcessor.from_pretrained(model_path, use_fast=True, **kwargs)";
      new = builtins.concatStringsSep "\n" [
        "from transformers.models.auto import video_processing_auto as _vpa"
        "    from transformers import processing_utils as _pu"
        "    _orig_vp = _vpa.AutoVideoProcessor.from_pretrained"
        "    _orig_check = _pu.ProcessorMixin.check_argument_for_proper_class"
        # Make AutoVideoProcessor.from_pretrained a no-op returning None ...
        "    _vpa.AutoVideoProcessor.from_pretrained = classmethod(lambda cls, *a, **kw: None)"
        # ... and let ProcessorMixin accept that None where it would otherwise
        # type-check the (now absent) video processor.
        "    def _skip_none_check(self, name, arg):"
        "        if arg is None: return type(None)"
        "        return _orig_check(self, name, arg)"
        "    _pu.ProcessorMixin.check_argument_for_proper_class = _skip_none_check"
        "    processor = AutoProcessor.from_pretrained(model_path, use_fast=True, **kwargs)"
        # Restore the originals so nothing else in the process is affected.
        "    _vpa.AutoVideoProcessor.from_pretrained = _orig_vp"
        "    _pu.ProcessorMixin.check_argument_for_proper_class = _orig_check"
      ];
    in
    ''
      substituteInPlace mlx_vlm/utils.py \
        --replace-fail \
          '${old}' \
          '${new}'
    '';

  # No test suite worth running at build time (needs model weights + Metal).
  doCheck = false;

  meta = {
    description = "Apple MLX vision-language model inference (image + text), packaged for Nix without the torch/torchvision/opencv-python chain";
    homepage = "https://github.com/Blaizzy/mlx-vlm";
  };
}