TELEICU/Spoke unreachable runbook: Difference between revisions
From OHC Network Wiki
More languages
More actions
Content deleted Content added
Fix both tables: proper MediaWiki syntax with row/cell separators (via update-page on MediaWiki MCP Server) |
Fix bullet lists: convert - to * (keep - [ ] checkboxes intact) (via update-page on MediaWiki MCP Server) |
||
| Line 63: | Line 63: | ||
=== Network loss === |
=== Network loss === |
||
* Have site personnel check router/modem status |
|||
* Check if the node has local connectivity (LAN) |
|||
* If the node is up but isolated, Tailscale will reconnect when WAN is restored |
|||
=== OS/hardware crash === |
=== OS/hardware crash === |
||
* Request site personnel to perform a hard reset |
|||
* If the node does not boot, proceed to [[TELEICU/Node recovery runbook|Node recovery runbook]] |
|||
=== Tunnel issue === |
=== Tunnel issue === |
||
* Check Cloudflare Tunnel logs via the cloudflared service |
|||
* Verify tunnel credentials are still valid and not expired |
|||
* Restart the cloudflared service |
|||
* If credentials were tampered, re-deploy from vault/sops |
|||
== Step 4: Post-recovery == |
== Step 4: Post-recovery == |
||
Latest revision as of 09:05, 6 July 2026
Procedure when a spoke/hub node stops reporting to the control plane.
Step 1: Classify the downtime
Check VictoriaMetrics for the node's last reported data:
| Classification | Indicators |
|---|---|
| Power loss | Power metrics stopped before all other metrics; UPS status (if available) showed battery drain |
| Network loss | OS metrics were reporting but network path to CP/Tailscale went silent |
| OS/hardware crash | All metrics stopped simultaneously, no preceding degradation |
| Tunnel issue | Node is running but Cloudflare Tunnel disconnected |
The on-device agent's last classification should indicate the cause.
Step 2: Route to the right responder
| Downtime type | Responder | Channel |
|---|---|---|
| Power loss | Site admin / facilities | Phone |
| Network loss | ISP / site IT contact | Phone |
| OS/hardware crash | Hub admin | Slack/PagerDuty |
| Tunnel issue | Ops team | Slack |
Step 3: Remediation
Power loss
Follow Power loss runbook.
Network loss
- Have site personnel check router/modem status
- Check if the node has local connectivity (LAN)
- If the node is up but isolated, Tailscale will reconnect when WAN is restored
OS/hardware crash
- Request site personnel to perform a hard reset
- If the node does not boot, proceed to Node recovery runbook
Tunnel issue
- Check Cloudflare Tunnel logs via the cloudflared service
- Verify tunnel credentials are still valid and not expired
- Restart the cloudflared service
- If credentials were tampered, re-deploy from vault/sops
Step 4: Post-recovery
- [ ] Node shows Ready in K3s - [ ] Tunnel endpoint responds - [ ] All middleware health checks pass - [ ] Metrics flowing to VictoriaMetrics - [ ] Downtime classification verified for accuracy