TELEICU/Backup & restore: Difference between revisions
From OHC Network Wiki
More languages
More actions
Content deleted Content added
Automated edit (via update-page on MediaWiki MCP Server) |
Fix Cadence table: convert Markdown pipe syntax to proper MediaWiki wikitable (via update-page on MediaWiki MCP Server) |
||
| Line 39: | Line 39: | ||
== Cadence == |
== Cadence == |
||
{| class="wikitable" |
|||
| Backup type | Frequency | Retention | |
|||
|- |
|||
|---|---|---| |
|||
! Backup type |
|||
| etcd snapshot | Every 2 hours | 30 days | |
|||
! Frequency |
|||
| ⚫ | |||
! Retention |
|||
| ⚫ | |||
|- |
|||
| ⚫ | |||
| etcd snapshot |
|||
| VictoriaMetrics data | Per retention policy | 45 days default | |
|||
| Every 2 hours |
|||
| 30 days |
|||
|- |
|||
| ⚫ | |||
| Daily |
|||
| 90 days |
|||
|- |
|||
| ZFS snapshot |
|||
| Every 6 hours |
|||
| ⚫ | |||
|- |
|||
| ⚫ | |||
| Daily |
|||
| 90 days |
|||
|- |
|||
| VictoriaMetrics data |
|||
| Per retention policy |
|||
| 45 days default |
|||
|} |
|||
== Restore procedure == |
== Restore procedure == |
||
Latest revision as of 08:51, 6 July 2026
Backup strategy for all stateful components across the TELEICU fleet.
What gets backed up
| Component | Path A: Ansible | Path B: NixOS |
|---|---|---|
| K3s etcd state | etcd snapshot to S3 (automated) | etcd snapshot to S3 (automated) |
| Middleware data | Longhorn volume backups to S3 | ZFS send/recv to S3 + OpenEBS snapshots |
| Node OS state | Not backed up (reprovision via ansible-pull) | Nix flake (OS state is fully declarative — no backup needed) |
| Application configs | Versioned in ansible repo + vault | Versioned in flake repo + sops-nix |
| Monitoring data | VictoriaMetrics (separate retention policy) | VictoriaMetrics (separate retention policy) |
Cadence
| Backup type | Frequency | Retention |
|---|---|---|
| etcd snapshot | Every 2 hours | 30 days |
| Longhorn volume backup | Daily | 90 days |
| ZFS snapshot | Every 6 hours | 7 days local, 90 days remote |
| ZFS send to S3 | Daily | 90 days |
| VictoriaMetrics data | Per retention policy | 45 days default |
Restore procedure
etcd restore
- Stop K3s server on all CP nodes
- Restore etcd from the latest snapshot on a single node
- Start K3s server on the restored node
- Add remaining CP nodes back to the cluster
Longhorn restore (Path A)
- Create a new PVC from the Longhorn backup in the S3 target
- Attach to the replacement pod
- Verify data integrity
ZFS restore (Path B)
- Receive the ZFS send stream from S3 or the backup target
- Rollback the dataset to the received snapshot
- Verify data integrity
Off-site backup
All backups target an S3-compatible object store in a separate geography from the CP VMs. This survives a full region failure.