Sending Alerts from Grafana to Matrix

Published on 2024/04/07

Over the past year, I’ve been slowly working on getting my own infrastructure up and running so I could host stuff for my friends and myself.

I’d finished setting up grafana for all the monitoring, and then I realized that it didn’t have a built in way of sending alerts to matrix. This is pretty important to me because matrix is where my friends and I are most easily reachable, so if I could send alerts there, I’d know if something’s going wrong almost immediately.

Sidenote, don’t take this as an endorsement of matrix, I actually have a lot of problems with it which I’ll talk about in a whole other post.

Anyway, after looking around a little, I found matrix-webhook. It looked really simple to deploy and it supported encrypted rooms, so I just ran with it.

Configuring matrix-webhook

You’re gonna need a dedicated account on your matrix homeserver for grafana.

Once you have that ready, you just run:

matrix-webhook \
    --host <ip on the host to bind to> \
    --port <port on the host to bind to> \
    --matrix-url <matrix homeserver url> \
    --matrix-id <matrix username> \
    --matrix-pw <matrix password> \
    --api-key <A Secret Token>

The “Secret Token” can be literally anything sufficiently long. I just generated a 60 character password in bitwarden and used it.

Once matrix-webhook starts running, you can hop into Grafana and create a new contact point with a webhook integration.

The URL for the webhook is going to look like:

http://<the ip/domain where matrix-webhook is>:<the port>/<the matrix room id where you want the alerts to appear>?formatter=grafana&key=<the secret key you generated>

So, assuming you ran:

matrix-webhook \
    --host 10.0.1.2 \
    --port 6000 \
    --matrix-url https://matrix.glitched.systems \
    --matrix-id grafana \
    --matrix-pw grafanapassword \
    --api-key supersecrettoken

Then your webhook URL will look like:

http://10.0.1.2:6000/!arsotWfpoeham:glitched.systems?formatter=grafana&key=supersecrettoken

Aaaaand that’s it!

You can now configure your grafana alerts to use this contact point and it’ll send alert notifications straight to your matrix room.

One improvement you can make to this is writing a systemd service to run matrix-webhook automatically.