TELEICU/Path B: NixOS

guideteleicuCARE 3.0+

TeleICU

Architecture

Architecture overview

Topology Current state

Middleware

Strategy

Migration strategy

Decision matrix Path A: Ansible Path B: NixOS

Canary rollout

Operations

Security baseline

Incident management Monitoring

Backup & restore

Runbooks

Power loss

Node recovery Config drift CP failure Spoke unreachable

OS update SOP

Change & recovery

Change management

Rollback strategy New hospital setup Bulk migration

DR playbook

Flows

Site onboarding

Full architecture and workflow for the NixOS migration path.

Architecture

Every hospital PC runs NixOS with identical, pinned system configuration. A flake repository defines the full OS state — packages, services, users, firewall, K3s agent — and nodes converge to the declared state via comin (pull-based, primary) or deploy-rs (push-based, backup).

{{#mermaid:flowchart TB
    subgraph Repo["Flake Repository"]
        FLAKE["flake.nix<br/>all node configs"]
    end

    subgraph Primary["Primary Path"]
        COMIN["comin (pull)<br/>each node polls<br/>for new generation"]
    end

    subgraph Backup["Backup Path"]
        DEPLOY["deploy-rs (push)<br/>triggered from CI<br/>for urgent deploys"]
    end

    FLAKE --> COMIN
    FLAKE --> DEPLOY

    COMIN --> N1["Spoke Node<br/>NixOS + K3s agent"]
    COMIN --> N2["Hub Node<br/>NixOS + K3s agent"]
    COMIN --> NM["... (200 more)"]

    DEPLOY --> N1
    DEPLOY --> N2
}}

Provisioning (zero-touch)

New sites are provisioned using a custom ISO generated with nixos-generator:

1. The ISO boots with a pre-configured Cloudflare Tunnel or Tailscale connection already active
2. A bootstrap script on the ISO initiates nixos-anywhere over the tunnel
3. nixos-anywhere partitions the disk (ZFS via disko) and installs NixOS
4. The node reboots into its NixOS generation and registers as a K3s agent
5. comin starts polling for configuration updates

What NixOS manages

• Every system package, kernel parameter, and service
• Tailscale configuration and bringup
• K3s agent installation, config, and auto-join
• Cloudflare tunnel service (cloudflared)
• ZFS dataset layout and mount points
• OpenEBS localPV provisioner for K3s
• Monitoring agent, log shipper, downtime classifier
• Firewall rules (strict — only Tailscale + K3s + tunnel outbound)
• sops-nix secret decryption at build time

What K3s manages

• Middleware container deployments
• Service discovery, ingress, workload scaling
• Identity as a Kubernetes node (identical to Path A at this layer)

Storage

ZFS + OpenEBS localPV:

• Root pool: mirrored (if multiple disks) or single-disk with regular snapshots
• LocalPV datasets for stateful middleware workloads
• ZFS snapshots via sanoid/syncoid for fast backup
• Off-site backup via restic or zfs send to S3

Rollback

• Instant: nixos-rebuild switch --rollback reverts to the previous generation
• At boot: hold Shift/space during systemd-boot to select a previous generation
• Every NixOS generation is an atomic, bootable unit — rollback is always available even if the OS is partially broken

Update mechanism

comin is the primary path (pull-based, suited for unreliable connectivity):

- Each node polls the flake repository on a configurable interval - When a new generation is available, it builds and activates it atomically - If the build fails, the node stays on its current generation — no partial state

deploy-rs is the backup path:

- Push-based, triggered from CI for urgent/canary deploys - Requires the node to be reachable at deploy time - Used for staging and canary nodes where immediate rollout is needed

Repository structure

{{#mermaid:flowchart LR
    subgraph Flake["teleicu-nixos/"]
        FLAKE_NIX["flake.nix<br/>flake.lock"]
        HOSTS["hosts/<br/>cp-1/<br/>cp-2/<br/>cp-3/<br/>spoke-template/<br/>hub-template/"]
        MODULES["modules/<br/>tailscale.nix<br/>k3s-agent.nix<br/>cloudflared.nix<br/>zfs.nix<br/>openebs.nix<br/>monitoring.nix<br/>downtime-classifier.nix"]
        PROFILES["profiles/<br/>base.nix<br/>spoke.nix<br/>hub.nix"]
        SECRETS["secrets/"]
        ISO["iso/<br/>bootstrap.nix"]
        COMIN_CFG["comin/<br/>configuration.nix"]
    end
}}

Flake structure

flake.nix exposes:

- nixosConfigurations for each host (CP VMs + spoke/hub templates) - cominConfigurations for comin-based auto-updates - packages for the bootstrap ISO - devShells for development tooling

Related

Open Healthcare Network's flagship product: an open-source, FHIR-native electronic medical record (EMR) and healthcare operating system.