https://ohcnwiki.tellmey.fyi/index.php?action=history&feed=atom&title=TELEICU%2FPower_loss_runbook 2026-07-06T07:15:09Z Revision history for this page on the wiki MediaWiki 1.45.4 https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Power_loss_runbook&diff=289&oldid=prev 2026-07-06T03:10:12Z

<p>Automated edit (via update-page on MediaWiki MCP Server)</p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 08:40, 6 July 2026</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 47:</td> <td colspan="2" class="diff-lineno">Line 47:</td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Consider solar + battery for high-priority sites with frequent power loss</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Consider solar + battery for high-priority sites with frequent power loss</div></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Graceful shutdown scripts on UPS trigger can be added via ansible or NixOS module</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Graceful shutdown scripts on UPS trigger can be added via ansible or NixOS module</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>{{Navbox TELEICU}}</div></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td> </tr> <!-- diff cache key mediawiki:diff:1.41:old-251:rev-289:wikidiff2=table:1.14.2:bc2a06be --> </table>

Admin https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Power_loss_runbook&diff=251&oldid=prev 2026-07-06T02:54:06Z

<p>Create TELEICU power loss runbook (via create-page on MediaWiki MCP Server)</p> <p><b>New page</b></p><div>{{Doc header<br /> |type=runbook<br /> |domain=teleicu<br /> |title=Power loss runbook<br /> |order=12<br /> }}<br /> <br /> Procedure for handling a site power loss event.<br /> <br /> == Detection ==<br /> <br /> Node heartbeat stops. On-device agent (if still running on UPS) reports power loss classification. If UPS is also depleted, the node goes completely silent.<br /> <br /> == Immediate actions (first 10 min) ==<br /> <br /> # Confirm classification — check VictoriaMetrics for last recorded power metrics<br /> # Notify hub admin assigned to the site per [[TELEICU/Incident management|Incident management]]<br /> # If UPS runtime data is available (from site inventory), estimate remaining uptime<br /> <br /> == During outage ==<br /> <br /> * Node is offline. K3s control plane marks it as unreachable. etcd tolerates the absence.<br /> * Workloads on that node will be rescheduled only if they have inter-pod anti-affinity and replicas elsewhere. By design, spoke workloads are typically singletons.<br /> <br /> == Recovery ==<br /> <br /> When power is restored:<br /> <br /> # Node boots automatically (BIOS set to &quot;power on after AC loss&quot;)<br /> # K3s agent re-registers with the control plane<br /> # Tailscale tunnel re-establishes<br /> # Cloudflare tunnel service starts<br /> # Middleware containers come up in order<br /> # Monitoring agent reconnects and catches up on buffered metrics<br /> # On-device agent reports &quot;recovered from power loss&quot;<br /> <br /> == Post-recovery verification ==<br /> <br /> - [ ] K3s node status: &lt;code&gt;kubectl get nodes&lt;/code&gt; shows Ready<br /> - [ ] Middleware health endpoint returns 200<br /> - [ ] Metrics flowing to VictoriaMetrics<br /> - [ ] Hub admin acknowledges site is operational<br /> <br /> == Prevention ==<br /> <br /> * Site inventory should capture UPS capacity for each node (see site template)<br /> * Consider solar + battery for high-priority sites with frequent power loss<br /> * Graceful shutdown scripts on UPS trigger can be added via ansible or NixOS module<br /> <br /> {{Related}}</div>

Admin