Admin: Automated edit (via update-page on MediaWiki MCP Server)

2026-07-06T03:10:26Z

Automated edit (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 08:40, 6 July 2026
Line 53:		Line 53:
	# Ansible: push a hotfix playbook to a dedicated <code>emergency</code> branch and trigger forced pulls via deploy-rs or a webhook		# Ansible: push a hotfix playbook to a dedicated <code>emergency</code> branch and trigger forced pulls via deploy-rs or a webhook
	# NixOS: update flake and use deploy-rs to push the new generation immediately (bypass comin's poll cadence)		# NixOS: update flake and use deploy-rs to push the new generation immediately (bypass comin's poll cadence)

			{{Navbox TELEICU}}

	{{Related}}		{{Related}}

Admin: Create TELEICU OS update SOP (via create-page on MediaWiki MCP Server)

2026-07-06T02:54:43Z

Create TELEICU OS update SOP (via create-page on MediaWiki MCP Server)

New page

{{Doc header
|type=sop
|domain=teleicu
|title=OS update SOP
|order=17
}}

Scheduled procedure for keeping the fleet's operating system up to date with security patches.

== Cadence ==

* '''Security patches''': applied within 7 days of upstream release
* '''Non-security updates''': bundled into quarterly releases
* '''Kernel/hardware enablement''': as needed for new hardware deployments

== Path A: Ansible ==

Process:

# Maintain an <code>unattended-upgrades</code> or <code>apt</code> role in the ansible repo that pins specific package versions
# For security patches: update the pin list and push to repo — nodes apply on next ansible-pull cycle
# For quarterly updates: merge to <code>canary</code> first, validate for 7 days, then regional rollout
# Monitor ansible-pull logs across the fleet for any nodes that failed to apply

Key difference: no OS immutability. Nodes apply updates in-place. Ansible re-enforces state on each pull.

== Path B: NixOS ==

Process:

# Update the nixpkgs input in the flake to the latest patched revision
# Rebuild in staging, validate
# Merge flake update to canary branch — comin nodes pick up the new generation
# After validation, merge to main — fleet-wide rollout
# Each node activates the new generation atomically (no partial update state)
# If a patch breaks something, generation-based rollback is instant

Key difference: OS is immutably declared in the flake. Updates are atomic generations, not in-place package upgrades.

== Verification ==

After a patch cycle:

* Sample every region — verify nodes are on the expected OS generation/package version
* Check VictoriaMetrics for any post-patch anomalies
* Review any nodes that failed to apply (network out of sync, disk full, etc.)
* Update [[Special:CargoTables/TeleicuSites|TeleicuSites]] if OS baseline has changed

== Emergency patch ==

For zero-day vulnerabilities:

# Ansible: push a hotfix playbook to a dedicated <code>emergency</code> branch and trigger forced pulls via deploy-rs or a webhook
# NixOS: update flake and use deploy-rs to push the new generation immediately (bypass comin's poll cadence)

{{Related}}

TELEICU/OS update SOP - Revision history

Admin: Automated edit (via update-page on MediaWiki MCP Server)

Admin: Create TELEICU OS update SOP (via create-page on MediaWiki MCP Server)