Icinga

EvoHub reads Icinga in two ways, on the same URL:

Icinga Notifications (the notification module of Icinga DB Web) has a Webhook channel. Its default request body is read as it is.
Icinga 2 without Icinga Notifications runs a notification command. EvoHub provides the script and the configuration.

In both, a problem opens an EvoHub alert and the recovery resolves it.

Set it up

Create the integration

In EvoHub, go to On-Call → Integrations → + Add Integration, choose Icinga, pick an Escalation Policy and click Create Integration. Copy the Webhook URL.

Connect Icinga

Follow one of the two tabs below.

Icinga Notifications

In Icinga Web, go to Notifications → Configuration → Channels → New Channel and choose the type Webhook.
Set Method to POST, URL Template to your webhook URL, keep Request Body Template at its default {{json .}}, and set Response Status Codes to 200.
Add the channel to a contact and use that contact in your event rules.

Icinga 2 script

Save the script below as /etc/icinga2/scripts/evohub-notification and make it executable with chmod 755.
Test the connection. EvoHub answers with success and opens no alert: EVOHUB_URL='https://evohub.io/ingest/icinga?key=YOUR_INTEGRATION_KEY' /etc/icinga2/scripts/evohub-notification --test
Save the configuration below as /etc/icinga2/conf.d/evohub.conf, with your webhook URL in both EVOHUB_URL lines.
Check and reload: icinga2 daemon -C && systemctl reload icinga2.

Script (/etc/icinga2/scripts/evohub-notification, Python 3 standard library only):

#!/usr/bin/env python3
# EvoHub On-Call — Icinga 2 notification script.
# Install as /etc/icinga2/scripts/evohub-notification (chmod 755).
# Reads the EVOHUB_* environment the NotificationCommand sets.
# Test by hand: EVOHUB_URL='<URL>' ./evohub-notification --test
import json
import os
import sys
import urllib.error
import urllib.request

FIELDS = {
    "notification_type": "EVOHUB_NOTIFICATION_TYPE",
    "host": "EVOHUB_HOST",
    "host_display_name": "EVOHUB_HOST_DISPLAY_NAME",
    "service": "EVOHUB_SERVICE",
    "service_display_name": "EVOHUB_SERVICE_DISPLAY_NAME",
    "state": "EVOHUB_STATE",
    "output": "EVOHUB_OUTPUT",
    "address": "EVOHUB_ADDRESS",
    "time": "EVOHUB_TIME",
    "author": "EVOHUB_AUTHOR",
    "comment": "EVOHUB_COMMENT",
}


def main(argv):
    url = os.environ.get("EVOHUB_URL", "").strip()
    if not url:
        sys.stderr.write("evohub: EVOHUB_URL is not set\n")
        return 2
    if len(argv) == 2 and argv[1] == "--test":
        payload = {"evohub_test": True}
    else:
        payload = {k: os.environ[v] for k, v in FIELDS.items() if os.environ.get(v)}
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode("utf-8"),
        method="POST",
        headers={"Content-Type": "application/json", "User-Agent": "evohub-icinga2/1"},
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            resp.read()
        return 0
    except urllib.error.HTTPError as e:
        sys.stderr.write("evohub: EvoHub answered HTTP %d\n" % e.code)
    except (urllib.error.URLError, OSError) as e:
        sys.stderr.write("evohub: could not reach EvoHub: %s\n" % getattr(e, "reason", e))
    return 1


if __name__ == "__main__":
    sys.exit(main(sys.argv))

Configuration (/etc/icinga2/conf.d/evohub.conf):

object User "evohub" {
  display_name = "EvoHub On-Call"
}

object NotificationCommand "evohub-host-notification" {
  command = [ ConfigDir + "/scripts/evohub-notification" ]
  env = {
    EVOHUB_URL = "https://evohub.io/ingest/icinga?key=YOUR_INTEGRATION_KEY"
    EVOHUB_NOTIFICATION_TYPE = "$notification.type$"
    EVOHUB_HOST = "$host.name$"
    EVOHUB_HOST_DISPLAY_NAME = "$host.display_name$"
    EVOHUB_STATE = "$host.state$"
    EVOHUB_OUTPUT = "$host.output$"
    EVOHUB_ADDRESS = "$host.address$"
    EVOHUB_TIME = "$icinga.long_date_time$"
  }
}

object NotificationCommand "evohub-service-notification" {
  command = [ ConfigDir + "/scripts/evohub-notification" ]
  env = {
    EVOHUB_URL = "https://evohub.io/ingest/icinga?key=YOUR_INTEGRATION_KEY"
    EVOHUB_NOTIFICATION_TYPE = "$notification.type$"
    EVOHUB_HOST = "$host.name$"
    EVOHUB_HOST_DISPLAY_NAME = "$host.display_name$"
    EVOHUB_SERVICE = "$service.name$"
    EVOHUB_SERVICE_DISPLAY_NAME = "$service.display_name$"
    EVOHUB_STATE = "$service.state$"
    EVOHUB_OUTPUT = "$service.output$"
    EVOHUB_ADDRESS = "$host.address$"
    EVOHUB_TIME = "$icinga.long_date_time$"
  }
}

apply Notification "evohub" to Host {
  command = "evohub-host-notification"
  users = [ "evohub" ]
  types = [ Problem, Recovery ]
  states = [ Up, Down ]
  interval = 0
  assign where true
}

apply Notification "evohub" to Service {
  command = "evohub-service-notification"
  users = [ "evohub" ]
  types = [ Problem, Recovery ]
  states = [ OK, Warning, Critical, Unknown ]
  interval = 0
  assign where true
}

The configuration notifies a single evohub user, once per problem (interval = 0) and once on recovery, for every host and service. Narrow assign where to send only some of them. Acknowledgement, downtime and flapping notifications, if you add those types, change nothing in EvoHub.

What EvoHub reads

EvoHub alert	Icinga Notifications	Icinga 2 script
Title	The object name.	Host, service and state, for example db-02 / PostgreSQL connections is CRITICAL.
Description	The event message.	The check output.
Labels	The object's tags (such as `host` and `service`), `severity`, `url`, `incident_id`.	`host`, `service`, `state`, `address`, `time`.

Severity

Icinga Notifications severity	Icinga 2 state	EvoHub severity
emerg, alert, crit	CRITICAL, DOWN	critical
err	—	high
warning	WARNING	medium
notice	UNKNOWN	low
info, debug	—	info

Resolve and deduplication

An alert is identified by the object (Icinga Notifications) or by the host and service (Icinga 2). Icinga sends one notification per notified contact; any repeats while the alert is open are recorded as Retriggered and nobody is paged again. The alert resolves when the incident recovers (Icinga Notifications) or on the RECOVERY notification (Icinga 2).

#Set it up

#Create the integration

#Connect Icinga

#What EvoHub reads

#Severity

#Resolve and deduplication

#Related