# Icinga

EvoHub reads Icinga in two ways, on the same URL:

- **Icinga Notifications** (the notification module of Icinga DB Web) has a **Webhook** channel. Its default request body is read as it is.
- **Icinga 2** without Icinga Notifications runs a notification command. EvoHub provides the script and the configuration.

In both, a problem opens an EvoHub alert and the recovery resolves it.

## Set it up

:::steps
### Create the integration
In EvoHub, go to **On-Call → Integrations → + Add Integration**, choose **Icinga**, pick an **Escalation Policy** and click **Create Integration**. Copy the **Webhook URL**.
### Connect Icinga
Follow one of the two tabs below.
:::

:::tabs
::tab{title="Icinga Notifications"}
1. In Icinga Web, go to **Notifications → Configuration → Channels → New Channel** and choose the type **Webhook**.
2. Set **Method** to `POST`, **URL Template** to your webhook URL, keep **Request Body Template** at its default `{{json .}}`, and set **Response Status Codes** to `200`.
3. Add the channel to a contact and use that contact in your event rules.
::tab{title="Icinga 2 script"}
1. Save the script below as `/etc/icinga2/scripts/evohub-notification` and make it executable with `chmod 755`.
2. Test the connection. EvoHub answers with success and opens no alert:
   `EVOHUB_URL='https://evohub.io/ingest/icinga?key=YOUR_INTEGRATION_KEY' /etc/icinga2/scripts/evohub-notification --test`
3. Save the configuration below as `/etc/icinga2/conf.d/evohub.conf`, with your webhook URL in both `EVOHUB_URL` lines.
4. Check and reload: `icinga2 daemon -C && systemctl reload icinga2`.
:::

Script (`/etc/icinga2/scripts/evohub-notification`, Python 3 standard library only):

```python
#!/usr/bin/env python3
# EvoHub On-Call — Icinga 2 notification script.
# Install as /etc/icinga2/scripts/evohub-notification (chmod 755).
# Reads the EVOHUB_* environment the NotificationCommand sets.
# Test by hand: EVOHUB_URL='<URL>' ./evohub-notification --test
import json
import os
import sys
import urllib.error
import urllib.request

FIELDS = {
    "notification_type": "EVOHUB_NOTIFICATION_TYPE",
    "host": "EVOHUB_HOST",
    "host_display_name": "EVOHUB_HOST_DISPLAY_NAME",
    "service": "EVOHUB_SERVICE",
    "service_display_name": "EVOHUB_SERVICE_DISPLAY_NAME",
    "state": "EVOHUB_STATE",
    "output": "EVOHUB_OUTPUT",
    "address": "EVOHUB_ADDRESS",
    "time": "EVOHUB_TIME",
    "author": "EVOHUB_AUTHOR",
    "comment": "EVOHUB_COMMENT",
}


def main(argv):
    url = os.environ.get("EVOHUB_URL", "").strip()
    if not url:
        sys.stderr.write("evohub: EVOHUB_URL is not set\n")
        return 2
    if len(argv) == 2 and argv[1] == "--test":
        payload = {"evohub_test": True}
    else:
        payload = {k: os.environ[v] for k, v in FIELDS.items() if os.environ.get(v)}
    req = urllib.request.Request(
        url,
        data=json.dumps(payload).encode("utf-8"),
        method="POST",
        headers={"Content-Type": "application/json", "User-Agent": "evohub-icinga2/1"},
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            resp.read()
        return 0
    except urllib.error.HTTPError as e:
        sys.stderr.write("evohub: EvoHub answered HTTP %d\n" % e.code)
    except (urllib.error.URLError, OSError) as e:
        sys.stderr.write("evohub: could not reach EvoHub: %s\n" % getattr(e, "reason", e))
    return 1


if __name__ == "__main__":
    sys.exit(main(sys.argv))
```

Configuration (`/etc/icinga2/conf.d/evohub.conf`):

```text
object User "evohub" {
  display_name = "EvoHub On-Call"
}

object NotificationCommand "evohub-host-notification" {
  command = [ ConfigDir + "/scripts/evohub-notification" ]
  env = {
    EVOHUB_URL = "https://evohub.io/ingest/icinga?key=YOUR_INTEGRATION_KEY"
    EVOHUB_NOTIFICATION_TYPE = "$notification.type$"
    EVOHUB_HOST = "$host.name$"
    EVOHUB_HOST_DISPLAY_NAME = "$host.display_name$"
    EVOHUB_STATE = "$host.state$"
    EVOHUB_OUTPUT = "$host.output$"
    EVOHUB_ADDRESS = "$host.address$"
    EVOHUB_TIME = "$icinga.long_date_time$"
  }
}

object NotificationCommand "evohub-service-notification" {
  command = [ ConfigDir + "/scripts/evohub-notification" ]
  env = {
    EVOHUB_URL = "https://evohub.io/ingest/icinga?key=YOUR_INTEGRATION_KEY"
    EVOHUB_NOTIFICATION_TYPE = "$notification.type$"
    EVOHUB_HOST = "$host.name$"
    EVOHUB_HOST_DISPLAY_NAME = "$host.display_name$"
    EVOHUB_SERVICE = "$service.name$"
    EVOHUB_SERVICE_DISPLAY_NAME = "$service.display_name$"
    EVOHUB_STATE = "$service.state$"
    EVOHUB_OUTPUT = "$service.output$"
    EVOHUB_ADDRESS = "$host.address$"
    EVOHUB_TIME = "$icinga.long_date_time$"
  }
}

apply Notification "evohub" to Host {
  command = "evohub-host-notification"
  users = [ "evohub" ]
  types = [ Problem, Recovery ]
  states = [ Up, Down ]
  interval = 0
  assign where true
}

apply Notification "evohub" to Service {
  command = "evohub-service-notification"
  users = [ "evohub" ]
  types = [ Problem, Recovery ]
  states = [ OK, Warning, Critical, Unknown ]
  interval = 0
  assign where true
}
```

The configuration notifies a single `evohub` user, once per problem (`interval = 0`) and once on recovery, for every host and service. Narrow `assign where` to send only some of them. Acknowledgement, downtime and flapping notifications, if you add those types, change nothing in EvoHub.

## What EvoHub reads

| EvoHub alert | Icinga Notifications | Icinga 2 script |
| --- | --- | --- |
| Title | The object name. | Host, service and state, for example *db-02 / PostgreSQL connections is CRITICAL*. |
| Description | The event message. | The check output. |
| Labels | The object's tags (such as `host` and `service`), `severity`, `url`, `incident_id`. | `host`, `service`, `state`, `address`, `time`. |

### Severity

| Icinga Notifications severity | Icinga 2 state | EvoHub severity |
| --- | --- | --- |
| emerg, alert, crit | CRITICAL, DOWN | critical |
| err | — | high |
| warning | WARNING | medium |
| notice | UNKNOWN | low |
| info, debug | — | info |

## Resolve and deduplication

An alert is identified by the object (Icinga Notifications) or by the host and service (Icinga 2). Icinga sends one notification per notified contact; any repeats while the alert is open are recorded as **Retriggered** and nobody is paged again. The alert resolves when the incident recovers (Icinga Notifications) or on the **RECOVERY** notification (Icinga 2).

## Related

- [Integrations overview](https://docs-dev.evohub.io/integrations-overview.md)
- [Checkmk](https://docs-dev.evohub.io/checkmk.md)