TELEICU/Spoke unreachable runbook
More languages
More actions
Procedure when a spoke/hub node stops reporting to the control plane.
Step 1: Classify the downtime
Check VictoriaMetrics for the node's last reported data:
| Power metrics stopped before all other metrics; UPS status (if available) showed battery drain | | OS metrics were reporting but network path to CP/Tailscale went silent | | All metrics stopped simultaneously, no preceding degradation | | Node is running but Cloudflare Tunnel disconnected | |
The on-device agent's last classification should indicate the cause.
Step 2: Route to the right responder
| Downtime type | Responder | Channel | | Power loss | Site admin / facilities | Phone | | Network loss | ISP / site IT contact | Phone | | OS/hardware crash | Hub admin | Slack/PagerDuty | | Tunnel issue | Ops team | Slack |
Step 3: Remediation
Power loss
Follow Power loss runbook.
Network loss
- Have site personnel check router/modem status - Check if the node has local connectivity (LAN) - If the node is up but isolated, Tailscale will reconnect when WAN is restored
OS/hardware crash
- Request site personnel to perform a hard reset - If the node does not boot, proceed to Node recovery runbook
Tunnel issue
- Check Cloudflare Tunnel logs via the cloudflared service - Verify tunnel credentials are still valid and not expired - Restart the cloudflared service - If credentials were tampered, re-deploy from vault/sops
Step 4: Post-recovery
- [ ] Node shows Ready in K3s - [ ] Tunnel endpoint responds - [ ] All middleware health checks pass - [ ] Metrics flowing to VictoriaMetrics - [ ] Downtime classification verified for accuracy