Escalation policies

An escalation policy is the chain EvoHub follows when an alert arrives: who to notify first, how long to wait for a response, and who to try next. Every integration points at one escalation policy. This page explains each setting and how the chain runs.

#How a policy runs

1. An alert arrives through an integration and the integration's policy starts.
2. Step 1 runs after its delay (which can be 0 (now)). It notifies its target.
3. If nobody acknowledges, takes over or resolves the alert, the next step runs after its own delay. Each step's delay counts from the step before it.
4. After the last step, the policy either stops or, if Repeat escalation is on, waits and starts again from step 1.

The escalation stops as soon as someone:

• acknowledges the alert (unless the policy has an acknowledgement timeout, see below),
• takes over the alert,
• resolves the alert, or the source resolves it,
• suppresses it from a voice call.

Redirect on an alert stops the current escalation and starts the selected policy from the top. See Alerts and incidents.

An alert that arrives through an integration with no escalation policy is recorded, but nobody is paged.

#Create a policy

The New Policy dialog offers the common settings. The policy's own page (Edit) adds per-step notification methods, retries, webhook targets, Slack channels and the acknowledgement timeout.

#Step settings

#Targets

• Schedule pages whoever is on call on that schedule when the step runs. Pick a specific layer, or Any layer (round-robin) to let the step's position choose among the layers on duty. See Layers and escalation. If nobody is on call at that moment, the step reaches nobody and the alert's timeline says "No one was on call — this step reached nobody".

• User pages one person directly. Only people who can respond to alerts can be selected.

• Webhook sends an HTTP POST to the URL you enter, so another system can react. The body is JSON:

  {
    "alert_id": "6f1c2a4e-1b7d-4c55-9a0e-2f8b7d1c9e10",
    "title": "Database primary is down",
    "severity": "critical",
    "summary": "Replication lag over 30s, primary not answering on 5432",
    "step": 1
  }
  

  summary carries the alert's description. The request has a 10-second timeout and is sent with Content-Type: application/json and User-Agent: EvoHub-OnCall/1.0.

#Notification method and personal schedules

When a step forces a method, that channel is used even if the person has turned it off in their own preferences. A person's Notification schedule still applies, though: if their schedule does not allow the forced channel at that time, EvoHub uses the channels their schedule does allow instead, and if none are allowed, nothing is sent to them and the escalation carries on. See Notifications.

SMS is not available as a method. A step saved with SMS in the past is shown as SMS (no longer available) and is reset to Default when you edit the policy.

#Policy settings

#Repeat escalation

Turn on Repeat escalation to run the whole chain again when the last step has run and nobody responded. Choose 1 to 5 times and how many minutes to wait between cycles (1, 2, 3, 5, 10, 15, 30 or 60).

#If acknowledged but not resolved

By default, acknowledging an alert ends its escalation. That can leave an alert open with nobody watching it if the person who acknowledged gets pulled away. To guard against that, set If acknowledged but not resolved in to a time (5 minutes to 4 hours). If the alert is still unresolved when the time runs out, the escalation resumes and the timeline records it. Choose escalate again, up to 1 to 5 times so it cannot ring all night, and choose and call:

• Someone else — skips whoever acknowledged and looks for another pair of hands.
• Whoever acknowledged, then someone else — reminds them once. If that goes unanswered, the next repeat escalates past them.

Never (the default) turns this off. The setting is read each time, so turning it off also stops the reminders for alerts that are already acknowledged.

#Responding by phone

During a voice call, the person called can press 4 to acknowledge, 3 to escalate to the next step right away, or 6 to suppress the alert. See Notifications.

#Delete a policy

Open the policy and delete it; the Delete Policy dialog asks you to confirm. Escalations of the policy that are still running stop.

#Related

• Schedules and rotations
• Notifications
• Alerts and incidents
• Integrations overview