TELEICU/Power loss runbook: Difference between revisions

Content deleted Content added

Inline

Latest revision as of 08:40, 6 July 2026

runbookteleicuCARE 3.0+

Procedure for handling a site power loss event.

Detection

Node heartbeat stops. On-device agent (if still running on UPS) reports power loss classification. If UPS is also depleted, the node goes completely silent.

Immediate actions (first 10 min)

Confirm classification — check VictoriaMetrics for last recorded power metrics
Notify hub admin assigned to the site per Incident management
If UPS runtime data is available (from site inventory), estimate remaining uptime

During outage

Node is offline. K3s control plane marks it as unreachable. etcd tolerates the absence.
Workloads on that node will be rescheduled only if they have inter-pod anti-affinity and replicas elsewhere. By design, spoke workloads are typically singletons.

Recovery

When power is restored:

Node boots automatically (BIOS set to "power on after AC loss")
K3s agent re-registers with the control plane
Tailscale tunnel re-establishes
Cloudflare tunnel service starts
Middleware containers come up in order
Monitoring agent reconnects and catches up on buffered metrics
On-device agent reports "recovered from power loss"

Post-recovery verification

- [ ] K3s node status: kubectl get nodes shows Ready - [ ] Middleware health endpoint returns 200 - [ ] Metrics flowing to VictoriaMetrics - [ ] Hub admin acknowledges site is operational

Prevention

Site inventory should capture UPS capacity for each node (see site template)
Consider solar + battery for high-priority sites with frequent power loss
Graceful shutdown scripts on UPS trigger can be added via ansible or NixOS module

@@ Line 47: / Line 47: @@
 * Consider solar + battery for high-priority sites with frequent power loss
 * Graceful shutdown scripts on UPS trigger can be added via ansible or NixOS module
+{{Navbox TELEICU}}
 {{Related}}