https://ohcnwiki.tellmey.fyi/index.php?action=history&feed=atom&title=TELEICU%2FDisaster_recovery_playbook 2026-07-06T07:15:48Z Revision history for this page on the wiki MediaWiki 1.45.4 https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Disaster_recovery_playbook&diff=302&oldid=prev 2026-07-06T03:10:57Z

<p>Automated edit (via update-page on MediaWiki MCP Server)</p> <table style="background-color: #fff; color: #202122;" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 08:40, 6 July 2026</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 78:</td> <td colspan="2" class="diff-lineno">Line 78:</td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Key advantage of NixOS: every node has the previous generation in its Nix store. Rollback does not depend on network access.</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>Key advantage of NixOS: every node has the previous generation in its Nix store. Rollback does not depend on network access.</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>{{Navbox TELEICU}}</div></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td> <td class="diff-marker"></td> <td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td> </tr> <!-- diff cache key mediawiki:diff:1.41:old-269:rev-302:wikidiff2=table:1.14.2:bc2a06be --> </table>

Admin https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Disaster_recovery_playbook&diff=269&oldid=prev 2026-07-06T02:56:22Z

<p>Create TELEICU disaster recovery playbook (via create-page on MediaWiki MCP Server)</p> <p><b>New page</b></p><div>{{Doc header<br /> |type=playbook<br /> |domain=teleicu<br /> |title=Disaster recovery playbook<br /> |order=25<br /> }}<br /> <br /> Full recovery procedure for a major disaster affecting a region or the entire fleet.<br /> <br /> == Scope ==<br /> <br /> This playbook covers:<br /> <br /> * Regional disaster (e.g., natural disaster taking out all sites in one state)<br /> * Control plane failure (all 3 CP VMs lost)<br /> * Fleet-wide configuration corruption<br /> <br /> == Scenario 1: Regional outage ==<br /> <br /> All sites in one state are offline simultaneously.<br /> <br /> === Detection ===<br /> <br /> * VictoriaMetrics shows all nodes in the state as offline<br /> * Incident management system triggers P1 for the region<br /> * Hub admin for that state is contacted<br /> <br /> === Response ===<br /> <br /> # Confirm scope — is it just that state or wider?<br /> # Contact regional ISP/power authority for status<br /> # For each site: follow [[TELEICU/Spoke unreachable runbook|Spoke unreachable runbook]]<br /> # After recovery, verify all nodes rejoin the cluster<br /> # Update TeleicuSites with current status<br /> <br /> == Scenario 2: Full control plane loss ==<br /> <br /> All 3 CP VMs are destroyed.<br /> <br /> === Impact ===<br /> <br /> * K3s API server is down — no new deployments or pod scheduling<br /> * Existing agent nodes continue running existing workloads<br /> * No metrics or monitoring updates<br /> <br /> === Recovery ===<br /> <br /> # Provision new CP VMs in cloud (same or different region)<br /> # Restore etcd from the latest backup ([[TELEICU/Backup &amp; restore|Backup &amp; restore]])<br /> # Start K3s server on restored etcd<br /> # Agents re-register as they come online (Tailscale mesh brings them back)<br /> # Verify full cluster health<br /> # Restore monitoring stack (VictoriaMetrics + Grafana from backup)<br /> <br /> === Prevention ===<br /> <br /> * etcd backups to S3 in a separate geography<br /> * CP VMs in different availability zones if possible<br /> * Document CP provisioning in Infrastructure as Code (Terraform/OpenTofu)<br /> <br /> == Scenario 3: Fleet-wide config corruption ==<br /> <br /> A bad configuration is applied across all nodes.<br /> <br /> === Ansible path ===<br /> <br /> # Halt ansible-pull to prevent further convergence (disable cron/timer)<br /> # Revert the offending commit in the git repo<br /> # Carefully push the revert (may need per-wave rollout if the corruption is severe)<br /> # For nodes that are broken: follow [[TELEICU/Node recovery runbook|Node recovery runbook]]<br /> <br /> === NixOS path ===<br /> <br /> # Each node can be rolled back individually: &lt;code&gt;nixos-rebuild switch --rollback&lt;/code&gt;<br /> # Or at boot: select a previous generation<br /> # Fix the flake and push a corrected generation<br /> # comin will apply the fix on next poll<br /> <br /> Key advantage of NixOS: every node has the previous generation in its Nix store. Rollback does not depend on network access.<br /> <br /> {{Related}}</div>

Admin