Incident Management and On-Call: from Pagers to AI agents

On-call meant a pager clipped to a belt and a monitoring server in the corner of the data center. Nagios, first released as NetSaint in 1999 and renamed in 2002, ran plugin checks against hosts and services and sent email or pager notifications when a check failed. HP OpenView, IBM Tivoli, and BMC Patrol did the same job for enterprises with bigger budgets.

Larger companies staffed a NOC: a room of operators watching wall screens, running scripted triage, and calling engineers off a printed phone tree. Escalation was a laminated sheet. If the first name did not answer, the operator dialed the next one.

The tools could tell you a check had failed. They could not tell you why. Diagnosis happened in a war room or a conference bridge, with people tailing logs over SSH on individual servers. Knowledge lived in heads, so the incident lasted as long as it took to find the one person who knew the system.

What broke was signal quality. Thresholds were static, every disk at 90 percent paged someone, and teams learned to ignore alerts. The postmortem, when there was one, often ended with a name instead of a fix.

PagerDuty, founded in 2009, turned on-call into a cloud service. Monitoring tools sent events to an API, and the service handled schedules, escalation policies, and delivery by push, SMS, and phone call. Opsgenie and VictorOps followed with the same model, and per-user pricing became the norm.

The bigger change was cultural. Cloud and continuous deployment put developers on call for their own services, the you-build-it-you-run-it model. Google published its Site Reliability Engineering book in 2016, which gave the industry a shared vocabulary: SLOs, error budgets, toil, and blameless postmortems.

Communication became a product too. Statuspage, bought by Atlassian in 2016, gave companies a public place to say what was broken. Consolidation followed: Splunk bought VictorOps in 2018, and Atlassian bought Opsgenie that same year.

What broke was volume. Microservices multiplied the number of things that could alert, and every team wired its monitors straight into the pager. Waking the right person got easy. Waking them for the right reason did not. I wrote about why the same incidents keep recurring in why incidents keep happening: the cause is usually people and process, rarely just the server.

The first AI in incident management was aimed at noise. AIOps platforms such as BigPanda and Moogsoft ingested alerts from every monitor, clustered related events into one incident, and suppressed duplicates. PagerDuty, by then the category leader, went public in April 2019. Dell bought Moogsoft in 2023 to fold AIOps into its operations portfolio.

Response moved into chat. incident.io, Rootly, and FireHydrant built Slack-native incident bots that open a channel, page responders, assign roles, keep a timeline, and draft the postmortem from the conversation. The bot did the paperwork; humans did the thinking.

Generative AI arrived late in this era as a copilot: summarize this incident, draft a status update, explain this log line. Useful, but the human still drove every step of the investigation.

This was also the period when the platform side matured. At LoginRadius, we moved from informal on-call rotations to an SRE practice with SLOs per service, error budgets, and post-incident reviews that produced structural fixes, which I described in our 2022 resilience review. The lesson I took: tooling helps, but the practice has to exist first.

The AI SRE agent is the new unit of incident tooling. When an alert fires, the agent starts investigating before anyone acknowledges the page. It pulls recent deploys, queries metrics, logs, and traces, checks past incidents, tests several hypotheses in parallel, and posts a ranked root cause with its evidence into the incident channel.

The products are real and shipping. PagerDuty's SRE Agent reached general availability in October 2025 and classifies incidents, surfaces context from past incidents, and recommends remediation. Datadog launched Bits AI SRE in December 2025. AWS made its DevOps Agent generally available in March 2026. incident.io, Rootly, Resolve AI, Traversal, and Cleric all sell investigation agents, and the open-source HolmesGPT is a CNCF sandbox project.

The limit is action. Almost every vendor stops at a recommendation or a pull request. incident.io says the only change its agent can make is a pull request you review and merge. Rootly says every change requires explicit human sign-off. Cleric and HolmesGPT are read-only by default. A few, such as Resolve AI and Traversal, describe automated mitigation, but customers gate it carefully.

The second limit is context. An agent is only as good as the telemetry and access it gets, which is the same lesson as everyone buying AI observability while nobody owns agent behavior. Meanwhile Atlassian stopped selling Opsgenie in June 2025, which pushed a wave of teams to re-platform right as agents arrived.

My bet is that incident management splits in two. Known failure classes, the ones with a runbook, a clear signal, and a reversible fix, get handled end to end by agents: a bad deploy rolled back, a saturated pool scaled, a stuck job restarted, a certificate renewed. The human sees a summary in the morning, not a page at 3 AM.

Novel failures stay human. Cascading outages, data corruption, security incidents, and anything that needs a judgment about customer impact will still pull in an incident commander. The agent becomes the fastest investigator in the room, not the decision maker.

The pattern mirrors the autonomous SOC: agents absorb the tier-1 queue, and people move up to judgment, design, and prevention. On-call rotations get smaller and quieter, but they do not disappear.

Three things gate this. Rollback and mitigation have to be safe by construction, agents need scoped identities with audit trails, and teams need evidence that agent actions are right often enough to remove the approval step for a given failure class.