TELEICU/OS update SOP: Difference between revisions
From OHC Network Wiki
More languages
More actions
Content deleted Content added
Create TELEICU OS update SOP (via create-page on MediaWiki MCP Server) |
Automated edit (via update-page on MediaWiki MCP Server) |
||
| Line 53: | Line 53: | ||
# Ansible: push a hotfix playbook to a dedicated <code>emergency</code> branch and trigger forced pulls via deploy-rs or a webhook |
# Ansible: push a hotfix playbook to a dedicated <code>emergency</code> branch and trigger forced pulls via deploy-rs or a webhook |
||
# NixOS: update flake and use deploy-rs to push the new generation immediately (bypass comin's poll cadence) |
# NixOS: update flake and use deploy-rs to push the new generation immediately (bypass comin's poll cadence) |
||
{{Navbox TELEICU}} |
|||
{{Related}} |
{{Related}} |
||
Latest revision as of 08:40, 6 July 2026
Scheduled procedure for keeping the fleet's operating system up to date with security patches.
Cadence
- Security patches: applied within 7 days of upstream release
- Non-security updates: bundled into quarterly releases
- Kernel/hardware enablement: as needed for new hardware deployments
Path A: Ansible
Process:
- Maintain an
unattended-upgradesoraptrole in the ansible repo that pins specific package versions - For security patches: update the pin list and push to repo — nodes apply on next ansible-pull cycle
- For quarterly updates: merge to
canaryfirst, validate for 7 days, then regional rollout - Monitor ansible-pull logs across the fleet for any nodes that failed to apply
Key difference: no OS immutability. Nodes apply updates in-place. Ansible re-enforces state on each pull.
Path B: NixOS
Process:
- Update the nixpkgs input in the flake to the latest patched revision
- Rebuild in staging, validate
- Merge flake update to canary branch — comin nodes pick up the new generation
- After validation, merge to main — fleet-wide rollout
- Each node activates the new generation atomically (no partial update state)
- If a patch breaks something, generation-based rollback is instant
Key difference: OS is immutably declared in the flake. Updates are atomic generations, not in-place package upgrades.
Verification
After a patch cycle:
- Sample every region — verify nodes are on the expected OS generation/package version
- Check VictoriaMetrics for any post-patch anomalies
- Review any nodes that failed to apply (network out of sync, disk full, etc.)
- Update TeleicuSites if OS baseline has changed
Emergency patch
For zero-day vulnerabilities:
- Ansible: push a hotfix playbook to a dedicated
emergencybranch and trigger forced pulls via deploy-rs or a webhook - NixOS: update flake and use deploy-rs to push the new generation immediately (bypass comin's poll cadence)