TELEICU/Spoke unreachable runbook
From OHC Network Wiki
More languages
More actions
Procedure when a spoke/hub node stops reporting to the control plane.
Step 1: Classify the downtime
Check VictoriaMetrics for the node's last reported data:
| Classification | Indicators |
|---|---|
| Power loss | Power metrics stopped before all other metrics; UPS status (if available) showed battery drain |
| Network loss | OS metrics were reporting but network path to CP/Tailscale went silent |
| OS/hardware crash | All metrics stopped simultaneously, no preceding degradation |
| Tunnel issue | Node is running but Cloudflare Tunnel disconnected |
The on-device agent's last classification should indicate the cause.
Step 2: Route to the right responder
| Downtime type | Responder | Channel |
|---|---|---|
| Power loss | Site admin / facilities | Phone |
| Network loss | ISP / site IT contact | Phone |
| OS/hardware crash | Hub admin | Slack/PagerDuty |
| Tunnel issue | Ops team | Slack |
Step 3: Remediation
Power loss
Follow Power loss runbook.
Network loss
- Have site personnel check router/modem status
- Check if the node has local connectivity (LAN)
- If the node is up but isolated, Tailscale will reconnect when WAN is restored
OS/hardware crash
- Request site personnel to perform a hard reset
- If the node does not boot, proceed to Node recovery runbook
Tunnel issue
- Check Cloudflare Tunnel logs via the cloudflared service
- Verify tunnel credentials are still valid and not expired
- Restart the cloudflared service
- If credentials were tampered, re-deploy from vault/sops
Step 4: Post-recovery
- [ ] Node shows Ready in K3s - [ ] Tunnel endpoint responds - [ ] All middleware health checks pass - [ ] Metrics flowing to VictoriaMetrics - [ ] Downtime classification verified for accuracy