TELEICU/Incident management
From OHC Network Wiki
More languages
More actions
How downtime events are detected, classified, and escalated across the TELEICU fleet.
Detection
Every node runs an on-device agent that performs two functions:
- Heartbeat: reports node health to the central monitoring system at a regular interval
- Downtime classification: when the heartbeat stops, the agent (or a co-located watchdog) records the last known state — power loss, network disconnection, or other failure
Classification logic:
| Classification | Description |
|---|---|
| Power outage | Node stops responding entirely, no network activity, BMC/IPMI reports power state (if available) |
| Network outage | Node is running (CPU/memory metrics were reporting) but network path to CP is down |
| Other | Node is running, network is up, but middleware service is degraded or down |
Notification chain
Severity levels:
| Severity | Condition | Target | Channel | SLA |
|---|---|---|---|---|
| P1 | Site offline > 30 min | Hub admin assigned to that site | Phone call + Slack/PagerDuty | 15 min |
| P2 | Site offline > 10 min | Hub admin assigned to that site | Slack/PagerDuty | 30 min |
| P3 | Service degraded | Hub admin assigned to that site | Slack notification | 4 hours |
| P4 | Informational | Ops team | Dashboard alert | Next business day |
Escalation
If the primary hub admin does not acknowledge within the SLA:
- Secondary contact for the same hub
- Regional ops lead
- Central infrastructure team
Post-incident
Every P1 event produces a postmortem:
- Root cause (power, network, hardware, software, human)
- Downtime classification accuracy (did the on-device agent correctly identify the cause?)
- Action items for preventing recurrence
- Runbook updates if the procedure was incomplete