- What is alert escalation?
How an alert moves from a monitoring tool to a person, and on to the next person when nobody acknowledges.
- How to design an escalation policy
Levels, waits, targets and repeats in an escalation policy, with example policies, common mistakes and a checklist.
- On-call rotation patterns
Weekly, daily, primary and secondary, and follow the sun: the common on-call rotation patterns and when each fits.
- Incident severity levels
How to define severity levels by user impact, what each level should trigger, and how severity connects to paging.
- What to alert on
Page on symptoms, not causes: the four golden signals, RED and USE, checks from outside, and what makes an alert worth a page.
- Alert fatigue: causes and fixes
Why teams stop trusting their alerts, how to measure the noise, and the fixes: delete, wait, group, inhibit, route by urgency and review.
- Heartbeat and cron job monitoring
How heartbeat monitoring catches cron jobs that silently stop, and how to set the period and grace time.
- SLOs, error budgets and burn-rate alerts
SLIs, SLOs and error budgets, and how multiwindow burn-rate alerts page on real risk to users.
- Incident response basics
What happens after someone acknowledges: declaring an incident, the roles, communication, and the blameless postmortem.
- MTTA, MTTR and other incident metrics
What MTTD, MTTA, MTTR and MTBF measure, where each clock starts and stops, and how to use them without gaming them.
- How to choose an on-call tool
The questions that separate on-call tools: notification methods, schedules, alert sources, pricing, self-hosting, and a short trial plan.
- Alertmanager routing, grouping and inhibition explained
Learn how Alertmanager deduplicates, groups, and routes alerts using matchers, timers, and inhibition rules.
- Alert deduplication and grouping: dedup keys, grouping and flapping
Learn how dedup keys, grouping, and flap handling reduce noise in PagerDuty, Opsgenie, and Alertmanager with concrete field examples.
- Silences and maintenance windows: muting alerts without missing outages
Compare silences, maintenance windows, inhibition and snooze. Learn Alertmanager commands and a checklist to mute alerts without missing outages.
- On-call handover: a checklist and template
On-call handover checklist and template for SREs to pass pager context, open incidents, and risks to the next shift.
- On-call overrides and shift swaps
Learn when to use overrides, swaps, or rotation edits. Compare PagerDuty, Opsgenie, and Grafana IRM. Get clear rules for sick days and approvals.
- How to test an escalation policy before a real outage
How to test an escalation policy before a real outage. Check contacts, send a test alert, and walk every level to verify your on-call setup.
- Alert lifecycle: triggered, acknowledged, resolved
Learn how PagerDuty, Opsgenie, Grafana IRM and Prometheus Alertmanager handle alert lifecycle states: triggered, acknowledged and resolved.
- The PagerDuty Events API v2 format, explained
Learn the PagerDuty Events API v2 format. See required fields, dedup_key logic, and a curl example for triggers and resolves.
- Choosing alert notification channels: push, SMS, voice, email and chat
Choose alert notification channels: push, SMS, voice, email and chat. Learn delivery, cost, and DND rules for on-call reliability.
- Writing alert messages people can act on
Rules for writing alert titles and bodies that tell engineers what broke, who owns it, and what to do first.
- Incident roles: incident commander, communications lead and scribe
How to split incident work across the commander, communications lead and scribe roles, with PagerDuty and Google practice, handoffs and role cards.
- Writing runbooks that on-call engineers actually use
A clear runbook structure, linking from Prometheus or Grafana, and practical habits to keep documentation current and useful for on-call engineers.
- Blameless postmortems: a template and how to run the review
Learn to write blameless postmortems with a template. Covers triggers, timeline building, and action items to fix systems and prevent repeat incidents.
- What to write on a status page during an incident
How to write clear status page updates during an incident, with templates for each state, a default cadence and component statuses.