TELEICU/Incident management: Difference between revisions
From OHC Network Wiki
More languages
More actions
Content deleted Content added
Create TELEICU incident management guide page (via create-page on MediaWiki MCP Server) |
Automated edit (via update-page on MediaWiki MCP Server) |
||
| Line 55: | Line 55: | ||
* Runbook updates if the procedure was incomplete |
* Runbook updates if the procedure was incomplete |
||
{{Navbox TELEICU}} |
|||
== Related == |
|||
* [[TELEICU/Monitoring & observability|Monitoring & observability]] |
|||
* [[TELEICU/Power loss runbook|Power loss runbook]] |
|||
* [[TELEICU/Spoke unreachable runbook|Spoke unreachable runbook]] |
|||
{{Related}} |
{{Related}} |
||
Revision as of 08:40, 6 July 2026
How downtime events are detected, classified, and escalated across the TELEICU fleet.
Detection
Every node runs an on-device agent that performs two functions:
- Heartbeat: reports node health to the central monitoring system at a regular interval
- Downtime classification: when the heartbeat stops, the agent (or a co-located watchdog) records the last known state — power loss, network disconnection, or other failure
Classification logic:
| Node stops responding entirely, no network activity, BMC/IPMI reports power state (if available) | | Node is running (CPU/memory metrics were reporting) but network path to CP is down | | Node is running, network is up, but middleware service is degraded or down | |
Notification chain
Severity levels:
| Severity | |||
|---|---|---|---|
| Hub admin assigned to that site | Phone call + Slack/PagerDuty | 15 min | | Hub admin assigned to that site | Slack/PagerDuty | 30 min | | Hub admin assigned to that site | Slack notification | 4 hours | | Ops team | Dashboard alert | Next business day | |
Escalation
If the primary hub admin does not acknowledge within the SLA:
- Secondary contact for the same hub
- Regional ops lead
- Central infrastructure team
Post-incident
Every P1 event produces a postmortem:
- Root cause (power, network, hardware, software, human)
- Downtime classification accuracy (did the on-device agent correctly identify the cause?)
- Action items for preventing recurrence
- Runbook updates if the procedure was incomplete