Skip to content

grafana-matrix-alert-relay

Modules

A small, single-file NixOS module that forwards Grafana alerts into a Matrix room, with no bridge, no bot framework, and no third-party dependency — just Python stdlib on loopback.

The problem

Grafana's webhook contact point speaks one dialect: it POSTs a JSON payload to a URL. The Matrix send API speaks another:

PUT /_matrix/client/v3/rooms/{roomId}/send/m.room.message/{txnId}

It requires PUT (not POST) and a caller-supplied transaction id in the path. Grafana can produce neither. So you can't point Grafana straight at Matrix — the verb is wrong and the txn id is missing.

The fix

Run a minimal relay on 127.0.0.1. Grafana POSTs alerts to it; the relay re-shapes each alert into a well-formed Matrix PUT and sends it with a bearer token.

The insight worth stealing: deterministic transaction ids

Matrix's txn id exists specifically for idempotency — resend the same PUT .../send/m.room.message/{txn} and the server treats it as the same message rather than a new one. Most relays throw a random UUID at it, which defeats the mechanism: every Grafana retry or duplicate delivery becomes another line in the room.

This relay derives the txn id instead:

txn = f"grafana-{floor(now/300)}-{sha1(body)[:16]}"

The message body is hashed, and the current time is bucketed into 5-minute windows. Identical alerts delivered or retried inside the same window reuse the same txn id, so the Matrix server dedupes them for you — no local state, no seen-cache, no cron cleanup. The 5-minute bucket bounds how long a duplicate is suppressed; a genuinely re-firing alert in a later window posts again, as it should.

Secret handling

The bearer token is delivered through systemd LoadCredential, materialised into the per-unit credentials directory (%d/matrix-token) at 0400, owned by the DynamicUser. It is never placed in the unit environment or on the command line, so it doesn't leak into systemctl show, /proc/<pid>/environ, or the journal. You supply the token file with whatever secret manager you use (agenix, sops-nix, a deploy step); the module only needs a path.

Usage

{
  imports = [ ./grafana-matrix-alert-relay ];

  services.grafana-matrix-alert-relay = {
    enable = true;
    matrixBase = "https://matrix.example.com";
    room = "!aBcDeFgHiJkLmNoPqR:example.com"; # internal room id, not the #alias
    tokenFile = "/run/secrets/matrix-alert-token";
    # port = 9099; # default
  };
}

Then add a Grafana webhook contact point pointing at http://127.0.0.1:9099/alert (any path works) and a notification policy that routes the alerts you care about to it. Example provisioning:

# grafana contact point
apiVersion: 1
contactPoints:
  - orgId: 1
    name: matrix
    receivers:
      - uid: matrix-relay
        type: webhook
        settings:
          url: http://127.0.0.1:9099/alert
          httpMethod: POST

Options

Option Type Default Notes
enable bool false
port port 9099 Loopback port the relay listens on.
matrixBase str — Homeserver client-server API base URL.
room str — Internal room id (!...), not the #alias. The bot must be joined.
tokenFile path — File with the raw bearer token; loaded via LoadCredential.

Getting the token and room id

  • Token: log in as the bot account once and grab its access token (e.g. via POST /_matrix/client/v3/login, or from an Element session's Help & About). Treat it like a password.
  • Room id: the internal id starts with ! and is stable; the #name:server alias is just a pointer. In Element it's under Room settings → Advanced → Internal room ID. The bot account must be a member of the room before the relay can post.

Message format

Alerts render as one line each:

[FIRING] HighCPU host=web-01: CPU above 90% for 5m

Status maps to a [FIRING] / [RESOLVED] / [PENDING] prefix; the summary is taken from annotations.summary, falling back to annotations.description then the alert name, and truncated at 400 chars. Adjust format_alert in default.nix if you want HTML formatting (m.room.message also accepts format: org.matrix.custom.html with a formatted_body).

Caveats

  • Loopback only. The relay binds 127.0.0.1. Run it on the same host as Grafana. It does no auth on the inbound side — anything that can reach the port can post to your room, so don't expose it.
  • Egress is open. The unit hardening (DynamicUser, ProtectSystem, RestrictNamespaces, ...) is tight, but IPAddressAllow permits all outbound because the relay needs DNS plus the homeserver. The token-gated room is the real boundary. Front it with an egress proxy if you need per-hostname control.
  • Fire-and-forget. A failed Matrix POST is logged to stderr and dropped; there's no retry queue. For alerting this is usually fine (the next evaluation re-notifies), but it is not a guaranteed-delivery pipe.
  • One room. Multi-room / severity-based routing would mean running more than one instance (they can share a port only if you change it), or teaching format_alert/post_matrix to pick a room from labels.

Source

modules/grafana-matrix-alert-relay/default.nix 
# grafana-matrix-alert-relay
#
# A tiny NixOS module that bridges Grafana's webhook contact point to a Matrix
# room. Grafana can only POST JSON to a URL; the Matrix send API is
# `PUT /_matrix/client/v3/rooms/{room}/send/m.room.message/{txn}` and needs a
# caller-supplied transaction id Grafana cannot generate. This relay listens on
# loopback, re-shapes each alert into a well-formed Matrix PUT, and uses a
# deterministic (sha1-of-body, 5-minute bucket) transaction id so retries and
# duplicate deliveries collapse server-side instead of spamming the room.
#
# Import it, set the options, and point a Grafana webhook contact point at
# http://127.0.0.1:<port>/alert (any path works — the relay accepts every POST).
{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.services.grafana-matrix-alert-relay;

  relayScript = pkgs.writeText "grafana-matrix-relay.py" ''
    """Webhook -> Matrix relay for Grafana alerts."""
    import datetime as dt
    import hashlib
    import http.server
    import json
    import os
    import sys
    import urllib.parse
    import urllib.request

    PORT = int(os.environ.get("RELAY_PORT", "9099"))
    MATRIX_BASE = os.environ["MATRIX_BASE"].rstrip("/")
    ROOM = os.environ["MATRIX_ROOM"]
    with open(os.environ["MATRIX_TOKEN_FILE"]) as _f:
        TOKEN = _f.read().strip()

    STATUS_PREFIX = {
        "firing": "[FIRING]",
        "resolved": "[RESOLVED]",
        "pending": "[PENDING]",
    }


    def format_alert(a: dict) -> str:
        labels = a.get("labels", {}) or {}
        annot = a.get("annotations", {}) or {}
        status = a.get("status", "?")
        name = labels.get("alertname") or annot.get("summary") or "alert"
        prefix = STATUS_PREFIX.get(status, f"[{status}]")
        summary = annot.get("summary") or annot.get("description") or name
        if len(summary) > 400:
            summary = summary[:400] + "..."
        host = labels.get("host") or labels.get("instance") or ""
        host_str = f" host={host}" if host else ""
        return f"{prefix} {name}{host_str}: {summary}"


    def deterministic_txn(body: str) -> str:
        # sha1 of the body, bucketed into 5-minute windows. Identical alerts
        # delivered/retried within the same window reuse the same txn id, so the
        # Matrix server dedupes them instead of posting the message twice.
        h = hashlib.sha1(body.encode()).hexdigest()
        bucket = int(dt.datetime.now().timestamp() // 300)
        return f"grafana-{bucket}-{h[:16]}"


    def post_matrix(body: str) -> int:
        txn = deterministic_txn(body)
        encoded_room = urllib.parse.quote(ROOM, safe="")
        url = f"{MATRIX_BASE}/_matrix/client/v3/rooms/{encoded_room}/send/m.room.message/{txn}"
        payload = json.dumps({"msgtype": "m.text", "body": body}).encode()
        req = urllib.request.Request(
            url, data=payload, method="PUT",
            headers={
                "Authorization": f"Bearer {TOKEN}",
                "Content-Type": "application/json",
            },
        )
        with urllib.request.urlopen(req, timeout=15) as resp:
            return resp.status


    class Handler(http.server.BaseHTTPRequestHandler):
        def do_POST(self):
            length = int(self.headers.get("Content-Length", "0") or 0)
            raw = self.rfile.read(length).decode("utf-8", "replace")
            try:
                payload = json.loads(raw)
            except Exception:
                self.send_response(400)
                self.end_headers()
                return
            alerts = payload.get("alerts") or []
            if not alerts:
                self.send_response(204)
                self.end_headers()
                return
            for a in alerts:
                try:
                    body = format_alert(a)
                    post_matrix(body)
                except Exception as exc:
                    sys.stderr.write(f"matrix post failed: {exc}\n")
            self.send_response(200)
            self.end_headers()

        def log_message(self, fmt, *args):
            sys.stderr.write("relay " + (fmt % args) + "\n")


    def main():
        srv = http.server.HTTPServer(("127.0.0.1", PORT), Handler)
        sys.stderr.write(f"relay listening on 127.0.0.1:{PORT} -> room {ROOM}\n")
        srv.serve_forever()


    if __name__ == "__main__":
        main()
  '';
in
{
  options.services.grafana-matrix-alert-relay = {
    enable = lib.mkEnableOption "webhook -> Matrix relay for Grafana alerts";

    port = lib.mkOption {
      type = lib.types.port;
      default = 9099;
      description = "Loopback port the relay listens on. Point Grafana's webhook contact point at http://127.0.0.1:<port>/alert.";
    };

    matrixBase = lib.mkOption {
      type = lib.types.str;
      example = "https://matrix.example.com";
      description = "Base URL of the Matrix homeserver's client-server API.";
    };

    room = lib.mkOption {
      type = lib.types.str;
      example = "!aBcDeFgHiJkLmNoPqR:example.com";
      description = ''
        Internal Matrix room ID (starts with `!`, not the human `#alias`) that
        alerts are posted to. Not a secret — room IDs are public identifiers;
        the bearer token is what gates posting. The relay's bot account must
        already be joined to this room.
      '';
    };

    tokenFile = lib.mkOption {
      type = lib.types.path;
      example = "/run/secrets/matrix-alert-token";
      description = ''
        Path to a file containing the Matrix bearer access token as its raw
        value (no `KEY=` env-file prefix, no trailing newline required). It is
        handed to the unit via systemd `LoadCredential`, so it never appears in
        the environment or on the command line. Provide it with your secret
        manager of choice (agenix, sops-nix, a deploy step, ...).
      '';
    };
  };

  config = lib.mkIf cfg.enable {
    systemd.services.grafana-matrix-alert-relay = {
      description = "Webhook -> Matrix relay for Grafana alerts";
      after = [ "network-online.target" ];
      wants = [ "network-online.target" ];
      wantedBy = [ "multi-user.target" ];

      environment = {
        RELAY_PORT = toString cfg.port;
        MATRIX_BASE = cfg.matrixBase;
        MATRIX_ROOM = cfg.room;
        # %d is the systemd credentials directory; see LoadCredential below.
        MATRIX_TOKEN_FILE = "%d/matrix-token";
      };

      serviceConfig = {
        Type = "simple";
        DynamicUser = true;
        # Token is materialised into the per-unit credentials dir (0400, owned
        # by the DynamicUser), readable at %d/matrix-token — never in env/argv.
        LoadCredential = "matrix-token:${cfg.tokenFile}";
        ExecStart = "${pkgs.python3}/bin/python3 ${relayScript}";
        Restart = "on-failure";
        RestartSec = "10s";

        NoNewPrivileges = true;
        ProtectHome = true;
        ProtectSystem = "strict";
        PrivateTmp = true;
        RestrictNamespaces = true;
        RestrictRealtime = true;
        RestrictSUIDSGID = true;
        LockPersonality = true;
        SystemCallArchitectures = "native";

        # Egress is left open (0.0.0.0/0) because the relay needs DNS plus the
        # Matrix homeserver, and the token-gated room is the real security
        # boundary — a tighter allowlist buys little here. If you want
        # per-hostname egress control, front it with an egress proxy instead.
        IPAddressDeny = "any";
        IPAddressAllow = [
          "127.0.0.0/8"
          "::1/128"
          "0.0.0.0/0"
          "::/0"
        ];
      };
    };
  };
}