TELEICU/Incident management: Difference between revisions
From OHC Network Wiki
More languages
More actions
Content deleted Content added
Create TELEICU incident management guide page (via create-page on MediaWiki MCP Server) |
Fix both tables: proper MediaWiki || cell separators, add missing headers/labels (via update-page on MediaWiki MCP Server) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 19: | Line 19: | ||
{| class="wikitable" |
{| class="wikitable" |
||
|- |
|- |
||
! Classification |
|||
| ⚫ | |||
! Description |
|||
| ⚫ | |||
|- |
|||
| ⚫ | |||
| Power outage |
|||
| ⚫ | |||
|- |
|||
| Network outage |
|||
| ⚫ | |||
|- |
|||
| Other |
|||
| ⚫ | |||
|} |
|} |
||
| Line 31: | Line 39: | ||
|- |
|- |
||
! Severity |
! Severity |
||
! Condition |
|||
! Target |
|||
! Channel |
|||
! SLA |
|||
|- |
|- |
||
| P1 |
|||
| P1 — Site offline > 30 min | Hub admin assigned to that site | Phone call + Slack/PagerDuty | 15 min | |
|||
| |
| Site offline > 30 min |
||
| Hub admin assigned to that site |
|||
| Phone call + Slack/PagerDuty |
|||
| P4 — Informational | Ops team | Dashboard alert | Next business day | |
|||
| 15 min |
|||
|- |
|||
| P2 |
|||
| Site offline > 10 min |
|||
| Hub admin assigned to that site |
|||
| Slack/PagerDuty |
|||
| 30 min |
|||
|- |
|||
| P3 |
|||
| Service degraded |
|||
| Hub admin assigned to that site |
|||
| Slack notification |
|||
| 4 hours |
|||
|- |
|||
| P4 |
|||
| Informational |
|||
| Ops team |
|||
| Dashboard alert |
|||
| Next business day |
|||
|} |
|} |
||
| Line 55: | Line 86: | ||
* Runbook updates if the procedure was incomplete |
* Runbook updates if the procedure was incomplete |
||
{{Navbox TELEICU}} |
|||
== Related == |
|||
* [[TELEICU/Monitoring & observability|Monitoring & observability]] |
|||
* [[TELEICU/Power loss runbook|Power loss runbook]] |
|||
* [[TELEICU/Spoke unreachable runbook|Spoke unreachable runbook]] |
|||
{{Related}} |
{{Related}} |
||
Latest revision as of 08:51, 6 July 2026
How downtime events are detected, classified, and escalated across the TELEICU fleet.
Detection
Every node runs an on-device agent that performs two functions:
- Heartbeat: reports node health to the central monitoring system at a regular interval
- Downtime classification: when the heartbeat stops, the agent (or a co-located watchdog) records the last known state — power loss, network disconnection, or other failure
Classification logic:
| Classification | Description |
|---|---|
| Power outage | Node stops responding entirely, no network activity, BMC/IPMI reports power state (if available) |
| Network outage | Node is running (CPU/memory metrics were reporting) but network path to CP is down |
| Other | Node is running, network is up, but middleware service is degraded or down |
Notification chain
Severity levels:
| Severity | Condition | Target | Channel | SLA |
|---|---|---|---|---|
| P1 | Site offline > 30 min | Hub admin assigned to that site | Phone call + Slack/PagerDuty | 15 min |
| P2 | Site offline > 10 min | Hub admin assigned to that site | Slack/PagerDuty | 30 min |
| P3 | Service degraded | Hub admin assigned to that site | Slack notification | 4 hours |
| P4 | Informational | Ops team | Dashboard alert | Next business day |
Escalation
If the primary hub admin does not acknowledge within the SLA:
- Secondary contact for the same hub
- Regional ops lead
- Central infrastructure team
Post-incident
Every P1 event produces a postmortem:
- Root cause (power, network, hardware, software, human)
- Downtime classification accuracy (did the on-device agent correctly identify the cause?)
- Action items for preventing recurrence
- Runbook updates if the procedure was incomplete