TELEICU/Spoke unreachable runbook: Difference between revisions

Content deleted Content added

VisualWikitext

Inline

@@ Line 14: / Line 14: @@
 {| class="wikitable"
 |-
+! Classification
-| Power loss | Power metrics stopped before all other metrics; UPS status (if available) showed battery drain |
+! Indicators
-| Network loss | OS metrics were reporting but network path to CP/Tailscale went silent |
+|-
-| OS/hardware crash | All metrics stopped simultaneously, no preceding degradation |
+| Power loss
-| Tunnel issue | Node is running but Cloudflare Tunnel disconnected |
+| Power metrics stopped before all other metrics; UPS status (if available) showed battery drain
+|-
+| Network loss
+| OS metrics were reporting but network path to CP/Tailscale went silent
+|-
+| OS/hardware crash
+| All metrics stopped simultaneously, no preceding degradation
+|-
+| Tunnel issue
+| Node is running but Cloudflare Tunnel disconnected
 |}
@@ Line 24: / Line 34: @@
 == Step 2: Route to the right responder ==
+{| class="wikitable"
-| Downtime type | Responder | Channel |
+|-
-| Power loss | Site admin / facilities | Phone |
+! Downtime type
-| Network loss | ISP / site IT contact | Phone |
+! Responder
-| OS/hardware crash | Hub admin | Slack/PagerDuty |
+! Channel
-| Tunnel issue | Ops team | Slack |
+|-
+| Power loss
+| Site admin / facilities
+| Phone
+|-
+| Network loss
+| ISP / site IT contact
+| Phone
+|-
+| OS/hardware crash
+| Hub admin
+| Slack/PagerDuty
+|-
+| Tunnel issue
+| Ops team
+| Slack
+|}
 == Step 3: Remediation ==
@@ Line 36: / Line 63: @@
 === Network loss ===
-- Have site personnel check router/modem status
+* Have site personnel check router/modem status
-- Check if the node has local connectivity (LAN)
+* Check if the node has local connectivity (LAN)
-- If the node is up but isolated, Tailscale will reconnect when WAN is restored
+* If the node is up but isolated, Tailscale will reconnect when WAN is restored
 === OS/hardware crash ===
-- Request site personnel to perform a hard reset
+* Request site personnel to perform a hard reset
-- If the node does not boot, proceed to [[TELEICU/Node recovery runbook|Node recovery runbook]]
+* If the node does not boot, proceed to [[TELEICU/Node recovery runbook|Node recovery runbook]]
 === Tunnel issue ===
-- Check Cloudflare Tunnel logs via the cloudflared service
+* Check Cloudflare Tunnel logs via the cloudflared service
-- Verify tunnel credentials are still valid and not expired
+* Verify tunnel credentials are still valid and not expired
-- Restart the cloudflared service
+* Restart the cloudflared service
-- If credentials were tampered, re-deploy from vault/sops
+* If credentials were tampered, re-deploy from vault/sops
 == Step 4: Post-recovery ==
@@ Line 57: / Line 84: @@
 - [ ] Metrics flowing to VictoriaMetrics
 - [ ] Downtime classification verified for accuracy
+{{Navbox TELEICU}}
 {{Related}}