Checkmk

Checkmk notifies through scripts. EvoHub provides one: a problem opens an EvoHub alert and the recovery resolves it. The script uses only Python's standard library and sends a fixed set of fields — never the URL, the rule parameters or contact details.

Set it up

Create the integration

In EvoHub, go to On-Call → Integrations → + Add Integration, choose Checkmk, pick an Escalation Policy and click Create Integration. Copy the Webhook URL.

Install the script

As the site user (omd su <site>), save the script below as ~/local/share/check_mk/notifications/evohub and make it executable with chmod +x. Its second line, # EvoHub On-Call, is the name it gets in the notification method list.

Test the connection

Run it once by hand. EvoHub answers with success and opens no alert:

~/local/share/check_mk/notifications/evohub --test 'https://evohub.io/ingest/checkmk?key=YOUR_INTEGRATION_KEY'

Create the notification rule

Go to Setup → Notifications → Add notification rule. Choose the notification method EvoHub On-Call and enter your webhook URL as the first parameter. For contacts, select one user (for example a dedicated evohub user): the script runs once per selected contact. Include the changes back to OK and UP in the triggering events — the recovery is what resolves the alert.

Activate changes

Activate the pending changes.

Script (~/local/share/check_mk/notifications/evohub):

#!/usr/bin/env python3
# EvoHub On-Call
# Checkmk notification script. Install as the site user:
#   ~/local/share/check_mk/notifications/evohub   (chmod +x)
# Rule parameter 1 = your EvoHub integration URL.
# Test by hand (as the site user): ./evohub --test '<URL>'
import json
import os
import sys
import urllib.error
import urllib.request

# Only these variables are sent: never the rule parameters (they hold the
# URL and its key) nor contact details.
FIELDS = (
    "WHAT", "NOTIFICATIONTYPE", "HOSTNAME", "HOSTALIAS", "HOSTADDRESS",
    "HOSTSTATE", "HOSTSTATEID", "HOSTOUTPUT", "LONGHOSTOUTPUT",
    "SERVICEDESC", "SERVICESTATE", "SERVICESTATEID", "SERVICEOUTPUT",
    "LONGSERVICEOUTPUT", "HOSTTAGS", "OMD_SITE", "SHORTDATETIME",
    "HOSTURL", "SERVICEURL", "NOTIFICATIONAUTHOR", "NOTIFICATIONCOMMENT",
)


def post(url, payload):
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode("utf-8"),
        method="POST",
        headers={"Content-Type": "application/json", "User-Agent": "evohub-checkmk/1"},
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            resp.read()
        print("EvoHub: delivered")
        return 0
    except urllib.error.HTTPError as e:
        print("EvoHub answered HTTP %d" % e.code)
        # 429 and 5xx are worth a retry (exit 1); other refusals are final.
        return 1 if e.code == 429 or e.code >= 500 else 2
    except (urllib.error.URLError, OSError) as e:
        print("Could not reach EvoHub: %s" % getattr(e, "reason", e))
        return 1


def main(argv):
    if len(argv) == 3 and argv[1] == "--test":
        return post(argv[2], {"evohub_test": True})
    url = os.environ.get("NOTIFY_PARAMETER_1", "").strip()
    if not url:
        print("Set the EvoHub integration URL as the rule's first parameter")
        return 2
    payload = {}
    for name in FIELDS:
        value = os.environ.get("NOTIFY_" + name)
        if value:
            payload["NOTIFY_" + name] = value
    return post(url, payload)


if __name__ == "__main__":
    sys.exit(main(sys.argv))

The script exits with 0 when EvoHub accepted the notification, 1 when it is worth retrying (network error, 429 or 5xx) and 2 when retrying will not help. Its output appears in Checkmk's notification log.

What EvoHub reads

EvoHub alert	Taken from
Title	Host, service and state, for example web01 / HTTP shop is CRIT, or web01 is DOWN for a host.
Description	The check output, followed by the long output.
Labels	`host`, `service`, `state`, `address`, `site`, `url`, `host_tags`, `time`.

Severity

The numeric state is used, so translated or abbreviated state names do not matter.

Checkmk state	EvoHub severity
CRIT (service), DOWN (host)	critical
UNREACH (host)	high
WARN	medium
UNKNOWN	low

Resolve and deduplication

An alert is identified by the host, or the host and service. A PROBLEM notification opens it (a repeat while it is open is recorded as Retriggered); a RECOVERY notification resolves it. Acknowledgement, downtime, flapping and custom notifications are accepted and change nothing.

#Set it up

#Create the integration

#Install the script

#Test the connection

#Create the notification rule

#Activate changes

#What EvoHub reads

#Severity

#Resolve and deduplication

#Related