Prometheus Alertmanager

EvoHub receives alerts from Prometheus through Alertmanager's webhook receiver. Each firing alert opens an EvoHub alert, and when Alertmanager reports it resolved, EvoHub resolves it too.

Set it up

Create the integration

In EvoHub, go to On-Call → Integrations → + Add Integration, choose Prometheus, pick an Escalation Policy and click Create Integration. Copy the Webhook URL.

Add a receiver to Alertmanager

Open your Alertmanager configuration (alertmanager.yml) and add a receiver that uses the URL. Keep send_resolved: true so EvoHub can resolve alerts automatically.

Route alerts to it

Point a route at the evohub receiver — the top-level route, or a sub-route for the alerts that should page — and reload Alertmanager.

route:
  receiver: evohub
  # Or keep your current default receiver and add a sub-route:
  # routes:
  #   - receiver: evohub
  #     matchers:
  #       - severity=~"critical|warning"

receivers:
  - name: evohub
    webhook_configs:
      - url: "https://evohub.io/ingest/prometheus?key=YOUR_INTEGRATION_KEY"
        send_resolved: true

To keep another destination during a migration, add a second route or receiver for it; Alertmanager can notify several receivers.

What EvoHub reads

Alertmanager sends a group of alerts in one request. EvoHub handles each alert in the group separately:

EvoHub alert	Taken from
Title	The `alertname` label (or "Prometheus Alert" if it is missing).
Description	The `summary` annotation, or the `description` annotation if there is no summary.
Severity	The `severity` label — see below.
Labels and annotations	All of the alert's labels and annotations, shown on the alert and usable in the voice template.
Fingerprint	Alertmanager's fingerprint for the alert (or one computed from its labels).

Severity

`severity` label	EvoHub severity
`critical`	critical
`warning`	high
`info`, `informational`	info
anything else, or missing	medium

Add a severity label to your alerting rules to control how urgent each alert is.

Resolve and deduplication

An alert with status firing opens an EvoHub alert. If the same alert (same fingerprint) is still open in EvoHub, Alertmanager's repeat notifications are recorded as Retriggered and nobody is paged again.
An alert with status resolved resolves the matching EvoHub alert. This needs send_resolved: true.
Because the fingerprint comes from the alert's labels, an alert whose labels change (for example a label with a changing value) is treated as a new alert. Keep frequently changing values in annotations, not labels.

Test it

Send a test alert to Alertmanager with amtool:

amtool alert add alertname=EvoHubTest severity=critical \
  --annotation=summary="Test alert from Alertmanager" \
  --alertmanager.url=http://localhost:9093

After Alertmanager's group wait, the alert appears in On-Call → Alerts and the integration's Last Event updates.

Troubleshooting

Nothing arrives: check Alertmanager's logs for webhook errors. A 401 means the key in the URL is wrong or the integration is disabled.
Alerts never resolve: make sure send_resolved: true is set on the webhook config.
Alert arrives but nobody is paged: check that the integration has an escalation policy and that someone is on call on the schedule it targets.

Prometheus Alertmanager

#Set it up

#Create the integration

#Add a receiver to Alertmanager

#Route alerts to it

#What EvoHub reads

#Severity

#Resolve and deduplication

#Test it

#Troubleshooting

#Related