TELEICU/Architecture overview: Difference between revisions

Content deleted Content added

Inline

Latest revision as of 08:48, 6 July 2026

conceptteleicuCARE 3.0+

The TELEICU middleware layer connects 200+ hospitals (spokes) to central specialist hubs across 7 states. This page describes the current architecture, the reliability problems it faces, and the upgrade strategy being undertaken.

Scope

200+ hospital sites (spokes) providing tele-ICU services
Designated hub hospitals with specialist coverage
7 states across the network
Currently running on heterogeneous hardware with mixed Ubuntu versions (20.04, 22.04, 24.04)

Current pain points

Three reliability factors degrade the current deployment:

Frequent power and internet disruption — sites in semi-urban and rural areas experience intermittent connectivity. Nodes go offline unpredictably.
Tampering by unauthorized personnel — physical access to site hardware has led to configuration drift: Cloudflare tunnel credentials altered, docker-compose files modified, system settings changed without tracking.
Gap in OS security and maintenance updates — 200 devices are out of sync across Ubuntu 20.04, 22.04, and 24.04. No consistent patch cadence.

Upgrade strategy

The goal is to consolidate all nodes into a single K3s cluster with consistent, reproducible system state. Two paths are under evaluation:

Path A — Ansible Pull: Manage the OS layer and k3s agent configuration through a pull-based ansible workflow. Minimal new tooling, existing ops team skills apply.
Path B — NixOS: Full OS immutability via NixOS, provisioned with nixos-anywhere, updated via comin (pull) with deploy-rs as backup.

Both paths share the same target topology: all hospital PCs become K3s agent nodes, managed from 1–3 cloud-based control plane VMs. Inter-node communication runs over Tailscale.

Target topology

Key design decisions

Storage — NixOS path uses ZFS + OpenEBS for localPV; Ansible path uses Longhorn atop existing filesystems (no repartitioning possible).
Updates — Both paths are pull-based (comin / ansible-pull) to handle unreliable connectivity.
Monitoring — VictoriaMetrics for metrics, with on-device agents that classify downtime by cause (power vs network vs other).
Incident management — Hub-admin notification chain with escalation based on downtime classification.
Secrets — sops-nix for NixOS, Ansible Vault + sops for Ansible.
Networking — Tailscale mesh with ACL-based tagging for all inter-node traffic.

See the Migration strategy for the full decision framework.

@@ Line 31: / Line 31: @@
 Both paths share the same target topology: all hospital PCs become K3s agent nodes, managed from 1–3 cloud-based control plane VMs. Inter-node communication runs over Tailscale.
+== Target topology ==
+{{#mermaid:flowchart TB
+    subgraph CP["Cloud Control Plane (1-3 VMs)"]
+        CP1["CP VM 1<br/>(etcd member)"]
+        CP2["CP VM 2<br/>(etcd member)"]
+        CP3["CP VM 3<br/>(etcd member)"]
+    end
+    subgraph TS["Tailscale Mesh"]
+        CP1 --- TS
+        CP2 --- TS
+        CP3 --- TS
+    end
+    subgraph Sites["Hospital Sites"]
+        direction TB
+        HUB["Hub Site<br/>K3s agent + middleware"]
+        S1["Spoke Site<br/>K3s agent + middleware"]
+        S2["Spoke Site<br/>K3s agent + middleware"]
+        S3["Spoke Site<br/>K3s agent + middleware"]
+    end
+    TS --- HUB
+    TS --- S1
+    TS --- S2
+    TS --- S3
+}}
 == Key design decisions ==
@@ Line 42: / Line 71: @@
 See the [[TELEICU/Migration strategy|Migration strategy]] for the full decision framework.
+{{Navbox TELEICU}}
 {{Related}}