TELEICU/Node recovery runbook
More languages
More actions
Full recovery procedure for a node with disk failure, OS corruption, or hardware replacement.
Path A: Ansible
- Replace hardware or disk
- Install minimal Ubuntu + Tailscale (manual or PXE)
- Node connects to Tailscale and reaches the ansible-pull repo
- ansible-pull runs on next timer, applies full configuration
- K3s agent rejoins the cluster
- Middleware workloads are rescheduled onto the node
Considerations: - No ZFS — no filesystem-level snapshot recovery - Restore middleware data from off-site backup (restic/borg to S3) - Longhorn will rebuild volume replicas from remaining nodes if the deployment uses replication
Path B: NixOS
- Replace hardware or disk
- Boot from the nixos-generator bootstrap ISO (pre-configured tunnel)
- nixos-anywhere partitions the disk (ZFS + disko) and installs NixOS
- Node reboots into NixOS, K3s agent joins, comin starts polling
- ZFS snapshots can be restored from sanoid/syncoid backups
Considerations: - ZFS send/recv for fast data recovery if snapshot target is available - Full OS state restored from flake — no manual package installs
Shared steps (both paths)
- [ ] Verify hardware is functional (RAM test, disk health) - [ ] Update TeleicuSites entry with hardware changes - [ ] Confirm node rejoined K3s cluster - [ ] Verify all middleware endpoints - [ ] Confirm monitoring data flowing - [ ] Run canary validation before marking fully operational