View source for TELEICU/Backup & restore

{{Doc header
|type=guide
|domain=teleicu
|title=Backup & restore
|order=18
}}

Backup strategy for all stateful components across the TELEICU fleet.

== What gets backed up ==

{| class="wikitable"
|-
! Component
! Path A: Ansible
! Path B: NixOS
|-
| K3s etcd state
| etcd snapshot to S3 (automated)
| etcd snapshot to S3 (automated)
|-
| Middleware data
| Longhorn volume backups to S3
| ZFS send/recv to S3 + OpenEBS snapshots
|-
| Node OS state
| Not backed up (reprovision via ansible-pull)
| Nix flake (OS state is fully declarative — no backup needed)
|-
| Application configs
| Versioned in ansible repo + vault
| Versioned in flake repo + sops-nix
|-
| Monitoring data
| VictoriaMetrics (separate retention policy)
| VictoriaMetrics (separate retention policy)
|}

== Cadence ==

| Backup type | Frequency | Retention |
|---|---|---|
| etcd snapshot | Every 2 hours | 30 days |
| Longhorn volume backup | Daily | 90 days |
| ZFS snapshot | Every 6 hours | 7 days local, 90 days remote |
| ZFS send to S3 | Daily | 90 days |
| VictoriaMetrics data | Per retention policy | 45 days default |

== Restore procedure ==

=== etcd restore ===

# Stop K3s server on all CP nodes
# Restore etcd from the latest snapshot on a single node
# Start K3s server on the restored node
# Add remaining CP nodes back to the cluster

=== Longhorn restore (Path A) ===

# Create a new PVC from the Longhorn backup in the S3 target
# Attach to the replacement pod
# Verify data integrity

=== ZFS restore (Path B) ===

# Receive the ZFS send stream from S3 or the backup target
# Rollback the dataset to the received snapshot
# Verify data integrity

== Off-site backup ==

All backups target an S3-compatible object store in a separate geography from the CP VMs. This survives a full region failure.

{{Related}}