TELEICU/Incident management
From OHC Network Wiki
More languages
More actions
How downtime events are detected, classified, and escalated across the TELEICU fleet.
Detection
Every node runs an on-device agent that performs two functions:
- Heartbeat: reports node health to the central monitoring system at a regular interval
- Downtime classification: when the heartbeat stops, the agent (or a co-located watchdog) records the last known state — power loss, network disconnection, or other failure
Classification logic:
| Node stops responding entirely, no network activity, BMC/IPMI reports power state (if available) | | Node is running (CPU/memory metrics were reporting) but network path to CP is down | | Node is running, network is up, but middleware service is degraded or down | |
Notification chain
Severity levels:
| Severity | |||
|---|---|---|---|
| Hub admin assigned to that site | Phone call + Slack/PagerDuty | 15 min | | Hub admin assigned to that site | Slack/PagerDuty | 30 min | | Hub admin assigned to that site | Slack notification | 4 hours | | Ops team | Dashboard alert | Next business day | |
Escalation
If the primary hub admin does not acknowledge within the SLA:
- Secondary contact for the same hub
- Regional ops lead
- Central infrastructure team
Post-incident
Every P1 event produces a postmortem:
- Root cause (power, network, hardware, software, human)
- Downtime classification accuracy (did the on-device agent correctly identify the cause?)
- Action items for preventing recurrence
- Runbook updates if the procedure was incomplete