TELEICU/Configuration drift runbook

runbookteleicuCARE 3.0+

TeleICU

Architecture

Architecture overview

Topology Current state

Middleware

Strategy

Migration strategy

Decision matrix Path A: Ansible Path B: NixOS

Canary rollout

Operations

Security baseline

Incident management Monitoring

Backup & restore

Runbooks

Power loss

Node recovery Config drift CP failure Spoke unreachable

OS update SOP

Change & recovery

Change management

Rollback strategy New hospital setup Bulk migration

DR playbook

Flows

Site onboarding

Procedures for detecting and remediating configuration drift.

Detection

Ansible path

• ansible-pull runs on timer and reports changed/changed state
• Non-compliant files are reverted each cycle
• Any manual change is detected on the next pull and overwritten
• Audit: git diff in the ansible repo shows what changed between pulls

NixOS path

• Drift is structurally impossible at the OS level — /etc is in the Nix store (read-only)
• Any tamper attempt at runtime fails with "Read-only file system"
• At generation level: nixos-rebuild list-generations shows the active generation
• If someone boots a different generation, comin will apply the current one on next poll

Remediation

Ansible path

1. Identify the drift: check the ansible-pull log for changed tasks that should not have changed
2. If the drift is from an outdated playbook, update the playbook and push to repo
3. If the drift is from tampering, investigate physical security at the site
4. Force an immediate pull: ansible-pull -U <repo> on the affected node
5. Review site access logs to identify who tampered

NixOS path

1. Check if the node is on the expected generation: compare /run/current-system hash with the flake's output hash
2. If not, trigger comin to pull the latest generation
3. If comin is failing, use deploy-rs as backup to push the correct generation
4. Investigate physical access — a NixOS node cannot be tampered with at runtime, so any divergence means someone booted a different generation or replaced the disk

Prevention

• Ensure all nodes have the correct pull interval configured
• For NixOS: remove boot entries for old generations (nixos-rebuild with --max-generations)
• For Ansible: enable ansible-pull logging to a central location
• Physical: lock server cases, use tamper-evident seals, restrict USB boot

TeleICU infrastructure

Architecture

Overview · Topology · Current state · Middleware

Strategy

Migration · Decision matrix · Path A: Ansible · Path B: NixOS · Canary rollout

Operations

Security · Incidents · Monitoring · Backup & restore

Runbooks

Power loss · Node recovery · Config drift · CP failure · Spoke unreachable · OS update

Change & recovery

Change mgmt · Rollback · New hospital · Bulk migration · DR

Flows

Site onboarding

Related

Open Healthcare Network's flagship product: an open-source, FHIR-native electronic medical record (EMR) and healthcare operating system.