TELEICU/Control plane failure runbook
From OHC Network Wiki
More languages
More actions
Procedure for handling loss of one or more control plane VMs.
Architecture
3 CP VMs with embedded etcd. etcd tolerates loss of 1 node (minority) and remains operational. Loss of 2 nodes (majority) causes loss of quorum.
Single CP VM failure
Impact: None — etcd maintains quorum. K3s operations continue uninterrupted.
Procedure:
- Identify the failed VM and root cause
- Rebuild or replace the VM
- Re-join it to the etcd cluster
- Verify:
etcdctl endpoint healthon all 3 nodes
Two CP VM failure (quorum loss)
Impact: K3s API server is down. Agent nodes continue running existing workloads but cannot accept new deployments or schedule changes.
Procedure:
- Identify the surviving VM — it has the latest etcd data
- Rebuild or restore the two failed VMs
- Manually restore etcd quorum using the surviving node's data
- Re-join restored nodes to the cluster
- Verify all agent nodes reconnected
- Verify all middleware workloads are healthy
Data safety:
- etcd state is backed up as part of the Backup & restore SOP - Application data is on Longhorn (Ansible) or ZFS (NixOS) volumes — independent of etcd - Restoring etcd from backup is the last resort if no CP node survives
Prevention
- Deploy CP VMs across different cloud availability zones if possible
- Monitor CP VM resource usage (CPU, memory, disk)
- Set alerts on etcd leader changes and db size growth
- Regular etcd backup (automated in the Backup & restore SOP)