TELEICU/Canary rollout
From OHC Network Wiki
More languages
More actions
Both migration paths use the same staged rollout process to minimize fleet-wide risk.
Stages
- Staging — a small set of test VMs or spare hardware that mirrors production. Every change lands here first.
- Canary (1–2 sites) — real hospital PCs, selected from sites with tolerant operations. Changes soak here for a validation period.
- Regional rollout — one state or hub region at a time, with a gating check after each region.
- Fleet-wide — remaining sites after all regions have passed.
Gating criteria
A stage must pass all checks before promoting to the next:
- All nodes report healthy to K3s control plane
- VictoriaMetrics shows no anomaly in key metrics
- Downtime classifier reports expected status (no false positives)
- Hub admin confirms no degradation in clinical workflow
- Rollback tested and confirmed working
Per-path
Ansible path
- Push change to ansible repo
stagingbranch → staging nodes converge on next pull - Merge to
canarybranch → canary nodes pull and apply - Merge to
regional/<state>branch → that state's nodes converge - Merge to
main→ fleet-wide rollout
NixOS path
- Update flake → rebuild in staging
- For canary:
deploy-rs .#canary-node(push) or merge to canary branch for comin - For regional: tag a release → comin picks up the tag
- For fleet: update
mainbranch of flake → all comin agents pull new generation
Branch/tag structure
syntaxhighlight lang="text" main ────────────── fleet-wide
├── regional/kerala
├── regional/maharashtra
├── …
└── canary
└── staging
syntaxhighlight