TELEICU/Canary rollout
From OHC Network Wiki
More languages
More actions
Both migration paths use the same staged rollout process to minimize fleet-wide risk.
Stages
{{#mermaid:flowchart LR
STAGING["Staging<br/>(test VMs/hardware)"]
CANARY["Canary<br/>(1-2 real sites)"]
REGIONAL["Regional Rollout<br/>(one state at a time)"]
FLEET["Fleet-wide<br/>(remaining sites)"]
STAGING -->|"validate →"| CANARY
CANARY -->|"gate →"| REGIONAL
REGIONAL -->|"gate →"| FLEET
}}
Gating criteria
A stage must pass all checks before promoting to the next:
- All nodes report healthy to K3s control plane
- VictoriaMetrics shows no anomaly in key metrics
- Downtime classifier reports expected status (no false positives)
- Hub admin confirms no degradation in clinical workflow
- Rollback tested and confirmed working
Per-path
Ansible path
{{#mermaid:flowchart LR
STAGING["staging branch"]
CANARY["canary branch"]
REGIONAL["regional/<state> branch"]
MAIN["main branch"]
STAGING -->|"merge"| CANARY
CANARY -->|"merge"| REGIONAL
REGIONAL -->|"merge"| MAIN
}}
NixOS path
{{#mermaid:flowchart LR
STAGING["Update flake → rebuild in staging"]
CANARY["deploy-rs .#canary-node<br/>or merge to canary branch"]
REGIONAL["Tag release → comin picks up tag"]
FLEET["Merge to main → comin pulls"]
STAGING -->|"validate →"| CANARY
CANARY -->|"gate →"| REGIONAL
REGIONAL -->|"gate →"| FLEET
}}