TELEICU/Control plane failure runbook: Difference between revisions

Content deleted Content added

Inline

Latest revision as of 08:40, 6 July 2026

runbookteleicuCARE 3.0+

Procedure for handling loss of one or more control plane VMs.

Architecture

3 CP VMs with embedded etcd. etcd tolerates loss of 1 node (minority) and remains operational. Loss of 2 nodes (majority) causes loss of quorum.

Single CP VM failure

Impact: None — etcd maintains quorum. K3s operations continue uninterrupted.

Procedure:

Identify the failed VM and root cause
Rebuild or replace the VM
Re-join it to the etcd cluster
Verify: etcdctl endpoint health on all 3 nodes

Two CP VM failure (quorum loss)

Impact: K3s API server is down. Agent nodes continue running existing workloads but cannot accept new deployments or schedule changes.

Procedure:

Identify the surviving VM — it has the latest etcd data
Rebuild or restore the two failed VMs
Manually restore etcd quorum using the surviving node's data
Re-join restored nodes to the cluster
Verify all agent nodes reconnected
Verify all middleware workloads are healthy

Data safety:

- etcd state is backed up as part of the Backup & restore SOP - Application data is on Longhorn (Ansible) or ZFS (NixOS) volumes — independent of etcd - Restoring etcd from backup is the last resort if no CP node survives

Prevention

Deploy CP VMs across different cloud availability zones if possible
Monitor CP VM resource usage (CPU, memory, disk)
Set alerts on etcd leader changes and db size growth
Regular etcd backup (automated in the Backup & restore SOP)

@@ Line 48: / Line 48: @@
 * Set alerts on etcd leader changes and db size growth
 * Regular etcd backup (automated in the Backup & restore SOP)
+{{Navbox TELEICU}}
 {{Related}}