TELEICU/Disaster recovery playbook
From OHC Network Wiki
More languages
More actions
Full recovery procedure for a major disaster affecting a region or the entire fleet.
Scope
This playbook covers:
- Regional disaster (e.g., natural disaster taking out all sites in one state)
- Control plane failure (all 3 CP VMs lost)
- Fleet-wide configuration corruption
Scenario 1: Regional outage
All sites in one state are offline simultaneously.
Detection
- VictoriaMetrics shows all nodes in the state as offline
- Incident management system triggers P1 for the region
- Hub admin for that state is contacted
Response
- Confirm scope — is it just that state or wider?
- Contact regional ISP/power authority for status
- For each site: follow Spoke unreachable runbook
- After recovery, verify all nodes rejoin the cluster
- Update TeleicuSites with current status
Scenario 2: Full control plane loss
All 3 CP VMs are destroyed.
Impact
- K3s API server is down — no new deployments or pod scheduling
- Existing agent nodes continue running existing workloads
- No metrics or monitoring updates
Recovery
- Provision new CP VMs in cloud (same or different region)
- Restore etcd from the latest backup (Backup & restore)
- Start K3s server on restored etcd
- Agents re-register as they come online (Tailscale mesh brings them back)
- Verify full cluster health
- Restore monitoring stack (VictoriaMetrics + Grafana from backup)
Prevention
- etcd backups to S3 in a separate geography
- CP VMs in different availability zones if possible
- Document CP provisioning in Infrastructure as Code (Terraform/OpenTofu)
Scenario 3: Fleet-wide config corruption
A bad configuration is applied across all nodes.
Ansible path
- Halt ansible-pull to prevent further convergence (disable cron/timer)
- Revert the offending commit in the git repo
- Carefully push the revert (may need per-wave rollout if the corruption is severe)
- For nodes that are broken: follow Node recovery runbook
NixOS path
- Each node can be rolled back individually:
nixos-rebuild switch --rollback - Or at boot: select a previous generation
- Fix the flake and push a corrected generation
- comin will apply the fix on next poll
Key advantage of NixOS: every node has the previous generation in its Nix store. Rollback does not depend on network access.