TELEICU/Spoke unreachable runbook: Difference between revisions

← Older edit Newer edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 08:51, 6 July 2026

runbookteleicuCARE 3.0+

Procedure when a spoke/hub node stops reporting to the control plane.

Step 1: Classify the downtime

Check VictoriaMetrics for the node's last reported data:

Classification	Indicators
Power loss	Power metrics stopped before all other metrics; UPS status (if available) showed battery drain
Network loss	OS metrics were reporting but network path to CP/Tailscale went silent
OS/hardware crash	All metrics stopped simultaneously, no preceding degradation
Tunnel issue	Node is running but Cloudflare Tunnel disconnected

The on-device agent's last classification should indicate the cause.

Step 2: Route to the right responder

Downtime type	Responder	Channel
Power loss	Site admin / facilities	Phone
Network loss	ISP / site IT contact	Phone
OS/hardware crash	Hub admin	Slack/PagerDuty
Tunnel issue	Ops team	Slack

Step 3: Remediation

Power loss

Follow Power loss runbook.

Network loss

- Have site personnel check router/modem status - Check if the node has local connectivity (LAN) - If the node is up but isolated, Tailscale will reconnect when WAN is restored

OS/hardware crash

- Request site personnel to perform a hard reset - If the node does not boot, proceed to Node recovery runbook

Tunnel issue

- Check Cloudflare Tunnel logs via the cloudflared service - Verify tunnel credentials are still valid and not expired - Restart the cloudflared service - If credentials were tampered, re-deploy from vault/sops

Step 4: Post-recovery

- [ ] Node shows Ready in K3s - [ ] Tunnel endpoint responds - [ ] All middleware health checks pass - [ ] Metrics flowing to VictoriaMetrics - [ ] Downtime classification verified for accuracy

@@ Line 14: / Line 14: @@
 {| class="wikitable"
 |-
+! Classification
-| Power loss | Power metrics stopped before all other metrics; UPS status (if available) showed battery drain |
+! Indicators
-| Network loss | OS metrics were reporting but network path to CP/Tailscale went silent |
+|-
-| OS/hardware crash | All metrics stopped simultaneously, no preceding degradation |
+| Power loss
-| Tunnel issue | Node is running but Cloudflare Tunnel disconnected |
+| Power metrics stopped before all other metrics; UPS status (if available) showed battery drain
+|-
+| Network loss
+| OS metrics were reporting but network path to CP/Tailscale went silent
+|-
+| OS/hardware crash
+| All metrics stopped simultaneously, no preceding degradation
+|-
+| Tunnel issue
+| Node is running but Cloudflare Tunnel disconnected
 |}
@@ Line 24: / Line 34: @@
 == Step 2: Route to the right responder ==
+{| class="wikitable"
-| Downtime type | Responder | Channel |
+|-
-| Power loss | Site admin / facilities | Phone |
+! Downtime type
-| Network loss | ISP / site IT contact | Phone |
+! Responder
-| OS/hardware crash | Hub admin | Slack/PagerDuty |
+! Channel
-| Tunnel issue | Ops team | Slack |
+|-
+| Power loss
+| Site admin / facilities
+| Phone
+|-
+| Network loss
+| ISP / site IT contact
+| Phone
+|-
+| OS/hardware crash
+| Hub admin
+| Slack/PagerDuty
+|-
+| Tunnel issue
+| Ops team
+| Slack
+|}
 == Step 3: Remediation ==