TELEICU/Backup & restore
From OHC Network Wiki
More languages
More actions
Backup strategy for all stateful components across the TELEICU fleet.
What gets backed up
| Component | Path A: Ansible | Path B: NixOS |
|---|---|---|
| K3s etcd state | etcd snapshot to S3 (automated) | etcd snapshot to S3 (automated) |
| Middleware data | Longhorn volume backups to S3 | ZFS send/recv to S3 + OpenEBS snapshots |
| Node OS state | Not backed up (reprovision via ansible-pull) | Nix flake (OS state is fully declarative — no backup needed) |
| Application configs | Versioned in ansible repo + vault | Versioned in flake repo + sops-nix |
| Monitoring data | VictoriaMetrics (separate retention policy) | VictoriaMetrics (separate retention policy) |
Cadence
| Backup type | Frequency | Retention | |---|---|---| | etcd snapshot | Every 2 hours | 30 days | | Longhorn volume backup | Daily | 90 days | | ZFS snapshot | Every 6 hours | 7 days local, 90 days remote | | ZFS send to S3 | Daily | 90 days | | VictoriaMetrics data | Per retention policy | 45 days default |
Restore procedure
etcd restore
- Stop K3s server on all CP nodes
- Restore etcd from the latest snapshot on a single node
- Start K3s server on the restored node
- Add remaining CP nodes back to the cluster
Longhorn restore (Path A)
- Create a new PVC from the Longhorn backup in the S3 target
- Attach to the replacement pod
- Verify data integrity
ZFS restore (Path B)
- Receive the ZFS send stream from S3 or the backup target
- Rollback the dataset to the received snapshot
- Verify data integrity
Off-site backup
All backups target an S3-compatible object store in a separate geography from the CP VMs. This survives a full region failure.