TELEICU/Architecture overview

conceptteleicuCARE 3.0+

TeleICU

Architecture

Architecture overview

Topology Current state

Middleware

Strategy

Migration strategy

Decision matrix Path A: Ansible Path B: NixOS

Canary rollout

Operations

Security baseline

Incident management Monitoring

Backup & restore

Runbooks

Power loss

Node recovery Config drift CP failure Spoke unreachable

OS update SOP

Change & recovery

Change management

Rollback strategy New hospital setup Bulk migration

DR playbook

Flows

Site onboarding

The TELEICU middleware layer connects 200+ hospitals (spokes) to central specialist hubs across 7 states. This page describes the current architecture, the reliability problems it faces, and the upgrade strategy being undertaken.

Scope

• 200+ hospital sites (spokes) providing tele-ICU services
• Designated hub hospitals with specialist coverage
• 7 states across the network
• Currently running on heterogeneous hardware with mixed Ubuntu versions (20.04, 22.04, 24.04)

Current pain points

Three reliability factors degrade the current deployment:

1. Frequent power and internet disruption — sites in semi-urban and rural areas experience intermittent connectivity. Nodes go offline unpredictably.
2. Tampering by unauthorized personnel — physical access to site hardware has led to configuration drift: Cloudflare tunnel credentials altered, docker-compose files modified, system settings changed without tracking.
3. Gap in OS security and maintenance updates — 200 devices are out of sync across Ubuntu 20.04, 22.04, and 24.04. No consistent patch cadence.

Upgrade strategy

The goal is to consolidate all nodes into a single K3s cluster with consistent, reproducible system state. Two paths are under evaluation:

• Path A — Ansible Pull: Manage the OS layer and k3s agent configuration through a pull-based ansible workflow. Minimal new tooling, existing ops team skills apply.
• Path B — NixOS: Full OS immutability via NixOS, provisioned with nixos-anywhere, updated via comin (pull) with deploy-rs as backup.

Both paths share the same target topology: all hospital PCs become K3s agent nodes, managed from 1–3 cloud-based control plane VMs. Inter-node communication runs over Tailscale.

Target topology

{{#mermaid:flowchart TB
    subgraph CP["Cloud Control Plane (1-3 VMs)"]
        CP1["CP VM 1<br/>(etcd member)"]
        CP2["CP VM 2<br/>(etcd member)"]
        CP3["CP VM 3<br/>(etcd member)"]
    end

    subgraph TS["Tailscale Mesh"]
        CP1 --- TS
        CP2 --- TS
        CP3 --- TS
    end

    subgraph Sites["Hospital Sites"]
        direction TB
        HUB["Hub Site<br/>K3s agent + middleware"]
        S1["Spoke Site<br/>K3s agent + middleware"]
        S2["Spoke Site<br/>K3s agent + middleware"]
        S3["Spoke Site<br/>K3s agent + middleware"]
    end

    TS --- HUB
    TS --- S1
    TS --- S2
    TS --- S3
}}

Key design decisions

• Storage — NixOS path uses ZFS + OpenEBS for localPV; Ansible path uses Longhorn atop existing filesystems (no repartitioning possible).
• Updates — Both paths are pull-based (comin / ansible-pull) to handle unreliable connectivity.
• Monitoring — VictoriaMetrics for metrics, with on-device agents that classify downtime by cause (power vs network vs other).
• Incident management — Hub-admin notification chain with escalation based on downtime classification.
• Secrets — sops-nix for NixOS, Ansible Vault + sops for Ansible.
• Networking — Tailscale mesh with ACL-based tagging for all inter-node traffic.

See the Migration strategy for the full decision framework.

Related

Open Healthcare Network's flagship product: an open-source, FHIR-native electronic medical record (EMR) and healthcare operating system.