https://ohcnwiki.tellmey.fyi/index.php?action=history&feed=atom&title=TELEICU%2FNode_recovery_runbook 2026-07-06T09:03:14Z Revision history for this page on the wiki MediaWiki 1.45.4 https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Node_recovery_runbook&diff=290&oldid=prev 2026-07-06T03:10:14Z

<p>Automated edit (via update-page on MediaWiki MCP Server)</p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 08:40, 6 July 2026</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 42:</td> <td colspan="2" class="diff-lineno">Line 42:</td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>- [ ] Confirm monitoring data flowing</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>- [ ] Confirm monitoring data flowing</div></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>- [ ] Run canary validation before marking fully operational</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>- [ ] Run canary validation before marking fully operational</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>{{Navbox TELEICU}}</div></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td> </tr> <!-- diff cache key mediawiki:diff:1.41:old-252:rev-290:wikidiff2=table:1.14.2:bc2a06be --> </table>

Admin https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Node_recovery_runbook&diff=252&oldid=prev 2026-07-06T02:54:12Z

<p>Create TELEICU node recovery runbook (via create-page on MediaWiki MCP Server)</p> <p><b>New page</b></p><div>{{Doc header<br /> |type=runbook<br /> |domain=teleicu<br /> |title=Node recovery runbook<br /> |order=13<br /> }}<br /> <br /> Full recovery procedure for a node with disk failure, OS corruption, or hardware replacement.<br /> <br /> == Path A: Ansible ==<br /> <br /> # Replace hardware or disk<br /> # Install minimal Ubuntu + Tailscale (manual or PXE)<br /> # Node connects to Tailscale and reaches the ansible-pull repo<br /> # ansible-pull runs on next timer, applies full configuration<br /> # K3s agent rejoins the cluster<br /> # Middleware workloads are rescheduled onto the node<br /> <br /> &#039;&#039;&#039;Considerations:&#039;&#039;&#039;<br /> - No ZFS — no filesystem-level snapshot recovery<br /> - Restore middleware data from off-site backup (restic/borg to S3)<br /> - Longhorn will rebuild volume replicas from remaining nodes if the deployment uses replication<br /> <br /> == Path B: NixOS ==<br /> <br /> # Replace hardware or disk<br /> # Boot from the nixos-generator bootstrap ISO (pre-configured tunnel)<br /> # nixos-anywhere partitions the disk (ZFS + disko) and installs NixOS<br /> # Node reboots into NixOS, K3s agent joins, comin starts polling<br /> # ZFS snapshots can be restored from sanoid/syncoid backups<br /> <br /> &#039;&#039;&#039;Considerations:&#039;&#039;&#039;<br /> - ZFS send/recv for fast data recovery if snapshot target is available<br /> - Full OS state restored from flake — no manual package installs<br /> <br /> == Shared steps (both paths) ==<br /> <br /> - [ ] Verify hardware is functional (RAM test, disk health)<br /> - [ ] Update [[Special:CargoTables/TeleicuSites|TeleicuSites]] entry with hardware changes<br /> - [ ] Confirm node rejoined K3s cluster<br /> - [ ] Verify all middleware endpoints<br /> - [ ] Confirm monitoring data flowing<br /> - [ ] Run canary validation before marking fully operational<br /> <br /> {{Related}}</div>

Admin