https://ohcnwiki.tellmey.fyi/index.php?action=history&feed=atom&title=TELEICU%2FControl_plane_failure_runbook
TELEICU/Control plane failure runbook - Revision history
2026-07-06T08:43:54Z
Revision history for this page on the wiki
MediaWiki 1.45.4
https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Control_plane_failure_runbook&diff=292&oldid=prev
Admin: Automated edit (via update-page on MediaWiki MCP Server)
2026-07-06T03:10:20Z
<p>Automated edit (via update-page on MediaWiki MCP Server)</p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 08:40, 6 July 2026</td>
</tr><tr>
<td colspan="2" class="diff-lineno">Line 48:</td>
<td colspan="2" class="diff-lineno">Line 48:</td>
</tr>
<tr>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Set alerts on etcd leader changes and db size growth</div></td>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Set alerts on etcd leader changes and db size growth</div></td>
</tr>
<tr>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Regular etcd backup (automated in the Backup & restore SOP)</div></td>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Regular etcd backup (automated in the Backup & restore SOP)</div></td>
</tr>
<tr>
<td colspan="2" class="diff-empty diff-side-deleted"></td>
<td class="diff-marker" data-marker="+"></td>
<td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><br /></td>
</tr>
<tr>
<td colspan="2" class="diff-empty diff-side-deleted"></td>
<td class="diff-marker" data-marker="+"></td>
<td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>{{Navbox TELEICU}}</div></td>
</tr>
<tr>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br /></td>
</tr>
<tr>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td>
<td class="diff-marker"></td>
<td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>{{Related}}</div></td>
</tr>
<!-- diff cache key mediawiki:diff:1.41:old-254:rev-292:wikidiff2=table:1.14.2:bc2a06be -->
</table>
Admin
https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Control_plane_failure_runbook&diff=254&oldid=prev
Admin: Create TELEICU control plane failure runbook (via create-page on MediaWiki MCP Server)
2026-07-06T02:54:28Z
<p>Create TELEICU control plane failure runbook (via create-page on MediaWiki MCP Server)</p>
<p><b>New page</b></p><div>{{Doc header<br />
|type=runbook<br />
|domain=teleicu<br />
|title=Control plane failure runbook<br />
|order=15<br />
}}<br />
<br />
Procedure for handling loss of one or more control plane VMs.<br />
<br />
== Architecture ==<br />
<br />
3 CP VMs with embedded etcd. etcd tolerates loss of 1 node (minority) and remains operational. Loss of 2 nodes (majority) causes loss of quorum.<br />
<br />
== Single CP VM failure ==<br />
<br />
'''Impact:''' None — etcd maintains quorum. K3s operations continue uninterrupted.<br />
<br />
'''Procedure:'''<br />
<br />
# Identify the failed VM and root cause<br />
# Rebuild or replace the VM<br />
# Re-join it to the etcd cluster<br />
# Verify: <code>etcdctl endpoint health</code> on all 3 nodes<br />
<br />
== Two CP VM failure (quorum loss) ==<br />
<br />
'''Impact:''' K3s API server is down. Agent nodes continue running existing workloads but cannot accept new deployments or schedule changes.<br />
<br />
'''Procedure:'''<br />
<br />
# Identify the surviving VM — it has the latest etcd data<br />
# Rebuild or restore the two failed VMs<br />
# Manually restore etcd quorum using the surviving node's data<br />
# Re-join restored nodes to the cluster<br />
# Verify all agent nodes reconnected<br />
# Verify all middleware workloads are healthy<br />
<br />
'''Data safety:'''<br />
<br />
- etcd state is backed up as part of the [[TELEICU/Backup & restore|Backup & restore]] SOP<br />
- Application data is on Longhorn (Ansible) or ZFS (NixOS) volumes — independent of etcd<br />
- Restoring etcd from backup is the last resort if no CP node survives<br />
<br />
== Prevention ==<br />
<br />
* Deploy CP VMs across different cloud availability zones if possible<br />
* Monitor CP VM resource usage (CPU, memory, disk)<br />
* Set alerts on etcd leader changes and db size growth<br />
* Regular etcd backup (automated in the Backup & restore SOP)<br />
<br />
{{Related}}</div>
Admin