Toggle search
Toggle menu
189
1
3
496
OHC Network Wiki
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

TELEICU/Spoke unreachable runbook: Difference between revisions

From OHC Network Wiki
More languages
More actions
Admin (talk | contribs)
Bureaucrats, Interface administrators, Administrators
483 edits
Content deleted Content added
VisualWikitext
Create TELEICU spoke unreachable runbook (via create-page on MediaWiki MCP Server)
 
Fix bullet lists: convert - to * (keep - [ ] checkboxes intact) (via update-page on MediaWiki MCP Server)
 
(2 intermediate revisions by the same user not shown)
Line 14: Line 14:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Classification
⚫
| Power loss | Power metrics stopped before all other metrics; UPS status (if available) showed battery drain |
! Indicators
⚫
| Network loss | OS metrics were reporting but network path to CP/Tailscale went silent |
|-
⚫
| OS/hardware crash | All metrics stopped simultaneously, no preceding degradation |
| Power loss
⚫
| Tunnel issue | Node is running but Cloudflare Tunnel disconnected |
⚫
| Power metrics stopped before all other metrics; UPS status (if available) showed battery drain
|-
| Network loss
⚫
| OS metrics were reporting but network path to CP/Tailscale went silent
|-
| OS/hardware crash
⚫
| All metrics stopped simultaneously, no preceding degradation
|-
| Tunnel issue
⚫
| Node is running but Cloudflare Tunnel disconnected
|}
|}


Line 24: Line 34:
== Step 2: Route to the right responder ==
== Step 2: Route to the right responder ==


{| class="wikitable"
| Downtime type | Responder | Channel |
|-
⚫
| Power loss | Site admin / facilities | Phone |
! Downtime type
⚫
| Network loss | ISP / site IT contact | Phone |
! Responder
| OS/hardware crash | Hub admin | Slack/PagerDuty |
! Channel
| Tunnel issue | Ops team | Slack |
|-
| Power loss
⚫
| Site admin / facilities
| Phone
|-
| Network loss
⚫
| ISP / site IT contact
| Phone
|-
| OS/hardware crash
| Hub admin
| Slack/PagerDuty
|-
| Tunnel issue
| Ops team
| Slack
|}


== Step 3: Remediation ==
== Step 3: Remediation ==
Line 36: Line 63:


=== Network loss ===
=== Network loss ===
- Have site personnel check router/modem status
* Have site personnel check router/modem status
- Check if the node has local connectivity (LAN)
* Check if the node has local connectivity (LAN)
- If the node is up but isolated, Tailscale will reconnect when WAN is restored
* If the node is up but isolated, Tailscale will reconnect when WAN is restored


=== OS/hardware crash ===
=== OS/hardware crash ===
- Request site personnel to perform a hard reset
* Request site personnel to perform a hard reset
- If the node does not boot, proceed to [[TELEICU/Node recovery runbook|Node recovery runbook]]
* If the node does not boot, proceed to [[TELEICU/Node recovery runbook|Node recovery runbook]]


=== Tunnel issue ===
=== Tunnel issue ===
- Check Cloudflare Tunnel logs via the cloudflared service
* Check Cloudflare Tunnel logs via the cloudflared service
- Verify tunnel credentials are still valid and not expired
* Verify tunnel credentials are still valid and not expired
- Restart the cloudflared service
* Restart the cloudflared service
- If credentials were tampered, re-deploy from vault/sops
* If credentials were tampered, re-deploy from vault/sops


== Step 4: Post-recovery ==
== Step 4: Post-recovery ==
Line 57: Line 84:
- [ ] Metrics flowing to VictoriaMetrics
- [ ] Metrics flowing to VictoriaMetrics
- [ ] Downtime classification verified for accuracy
- [ ] Downtime classification verified for accuracy

{{Navbox TELEICU}}


{{Related}}
{{Related}}

Latest revision as of 09:05, 6 July 2026

runbookteleicuCARE 3.0+

Procedure when a spoke/hub node stops reporting to the control plane.

Step 1: Classify the downtime

Check VictoriaMetrics for the node's last reported data:

Classification Indicators
Power loss Power metrics stopped before all other metrics; UPS status (if available) showed battery drain
Network loss OS metrics were reporting but network path to CP/Tailscale went silent
OS/hardware crash All metrics stopped simultaneously, no preceding degradation
Tunnel issue Node is running but Cloudflare Tunnel disconnected

The on-device agent's last classification should indicate the cause.

Step 2: Route to the right responder

Downtime type Responder Channel
Power loss Site admin / facilities Phone
Network loss ISP / site IT contact Phone
OS/hardware crash Hub admin Slack/PagerDuty
Tunnel issue Ops team Slack

Step 3: Remediation

Power loss

Follow Power loss runbook.

Network loss

  • Have site personnel check router/modem status
  • Check if the node has local connectivity (LAN)
  • If the node is up but isolated, Tailscale will reconnect when WAN is restored

OS/hardware crash

  • Request site personnel to perform a hard reset
  • If the node does not boot, proceed to Node recovery runbook

Tunnel issue

  • Check Cloudflare Tunnel logs via the cloudflared service
  • Verify tunnel credentials are still valid and not expired
  • Restart the cloudflared service
  • If credentials were tampered, re-deploy from vault/sops

Step 4: Post-recovery

- [ ] Node shows Ready in K3s - [ ] Tunnel endpoint responds - [ ] All middleware health checks pass - [ ] Metrics flowing to VictoriaMetrics - [ ] Downtime classification verified for accuracy

Open Healthcare Network's flagship product: an open-source, FHIR-native electronic medical record (EMR) and healthcare operating system.

Retrieved from "https://ohcnwiki.tellmey.fyi/index.php?title=TELEICU/Spoke_unreachable_runbook&oldid=323"