TELEICU/Configuration drift runbook
From OHC Network Wiki
More languages
More actions
Procedures for detecting and remediating configuration drift.
Detection
Ansible path
ansible-pullruns on timer and reports changed/changed state- Non-compliant files are reverted each cycle
- Any manual change is detected on the next pull and overwritten
- Audit:
git diffin the ansible repo shows what changed between pulls
NixOS path
- Drift is structurally impossible at the OS level — /etc is in the Nix store (read-only)
- Any tamper attempt at runtime fails with "Read-only file system"
- At generation level:
nixos-rebuild list-generationsshows the active generation - If someone boots a different generation, comin will apply the current one on next poll
Remediation
Ansible path
- Identify the drift: check the ansible-pull log for
changedtasks that should not have changed - If the drift is from an outdated playbook, update the playbook and push to repo
- If the drift is from tampering, investigate physical security at the site
- Force an immediate pull:
ansible-pull -U <repo>on the affected node - Review site access logs to identify who tampered
NixOS path
- Check if the node is on the expected generation: compare
/run/current-systemhash with the flake's output hash - If not, trigger comin to pull the latest generation
- If comin is failing, use deploy-rs as backup to push the correct generation
- Investigate physical access — a NixOS node cannot be tampered with at runtime, so any divergence means someone booted a different generation or replaced the disk
Prevention
- Ensure all nodes have the correct pull interval configured
- For NixOS: remove boot entries for old generations (nixos-rebuild with
--max-generations) - For Ansible: enable
ansible-pulllogging to a central location - Physical: lock server cases, use tamper-evident seals, restrict USB boot