TELEICU/Power loss runbook
From OHC Network Wiki
More languages
More actions
Procedure for handling a site power loss event.
Detection
Node heartbeat stops. On-device agent (if still running on UPS) reports power loss classification. If UPS is also depleted, the node goes completely silent.
Immediate actions (first 10 min)
- Confirm classification — check VictoriaMetrics for last recorded power metrics
- Notify hub admin assigned to the site per Incident management
- If UPS runtime data is available (from site inventory), estimate remaining uptime
During outage
- Node is offline. K3s control plane marks it as unreachable. etcd tolerates the absence.
- Workloads on that node will be rescheduled only if they have inter-pod anti-affinity and replicas elsewhere. By design, spoke workloads are typically singletons.
Recovery
When power is restored:
- Node boots automatically (BIOS set to "power on after AC loss")
- K3s agent re-registers with the control plane
- Tailscale tunnel re-establishes
- Cloudflare tunnel service starts
- Middleware containers come up in order
- Monitoring agent reconnects and catches up on buffered metrics
- On-device agent reports "recovered from power loss"
Post-recovery verification
- [ ] K3s node status: kubectl get nodes shows Ready
- [ ] Middleware health endpoint returns 200
- [ ] Metrics flowing to VictoriaMetrics
- [ ] Hub admin acknowledges site is operational
Prevention
- Site inventory should capture UPS capacity for each node (see site template)
- Consider solar + battery for high-priority sites with frequent power loss
- Graceful shutdown scripts on UPS trigger can be added via ansible or NixOS module