TELEICU/Backup & restore - Revision history

Admin: Fix Cadence table: convert Markdown pipe syntax to proper MediaWiki wikitable (via update-page on MediaWiki MCP Server)

2026-07-06T03:21:48Z

Fix Cadence table: convert Markdown pipe syntax to proper MediaWiki wikitable (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 08:51, 6 July 2026
Line 39:		Line 39:
	== Cadence ==		== Cadence ==

			{\| class="wikitable"
	\| Backup type \| Frequency \| Retention \|
			\|-
	\|---\|---\|---\|
			! Backup type
	\| etcd snapshot \| Every 2 hours \| 30 days \|
			! Frequency
⚫	\| Longhorn volume backup ~~\| Daily \| 90 days \|~~
			! Retention
⚫	~~\| ZFS snapshot \| Every 6 hours~~ \| 7 days local, 90 days remote \|
			\|-
⚫	\| ZFS send to S3 ~~\| Daily \| 90 days \|~~
			\| etcd snapshot
	\| VictoriaMetrics data \| Per retention policy \| 45 days default \|
			\| Every 2 hours
			\| 30 days
			\|-
		⚫	\| Longhorn volume backup
			\| Daily
			\| 90 days
			\|-
			\| ZFS snapshot
			\| Every 6 hours
		⚫	\| 7 days local, 90 days remote
			\|-
		⚫	\| ZFS send to S3
			\| Daily
			\| 90 days
			\|-
			\| VictoriaMetrics data
			\| Per retention policy
			\| 45 days default
			\|}

	== Restore procedure ==		== Restore procedure ==

Admin: Automated edit (via update-page on MediaWiki MCP Server)

2026-07-06T03:10:33Z

Automated edit (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 08:40, 6 July 2026
Line 71:		Line 71:

	All backups target an S3-compatible object store in a separate geography from the CP VMs. This survives a full region failure.		All backups target an S3-compatible object store in a separate geography from the CP VMs. This survives a full region failure.

			{{Navbox TELEICU}}

	{{Related}}		{{Related}}

Admin: Create TELEICU backup and restore guide (via create-page on MediaWiki MCP Server)

2026-07-06T02:55:32Z

Create TELEICU backup and restore guide (via create-page on MediaWiki MCP Server)

New page

{{Doc header
|type=guide
|domain=teleicu
|title=Backup & restore
|order=18
}}

Backup strategy for all stateful components across the TELEICU fleet.

== What gets backed up ==

{| class="wikitable"
|-
! Component
! Path A: Ansible
! Path B: NixOS
|-
| K3s etcd state
| etcd snapshot to S3 (automated)
| etcd snapshot to S3 (automated)
|-
| Middleware data
| Longhorn volume backups to S3
| ZFS send/recv to S3 + OpenEBS snapshots
|-
| Node OS state
| Not backed up (reprovision via ansible-pull)
| Nix flake (OS state is fully declarative — no backup needed)
|-
| Application configs
| Versioned in ansible repo + vault
| Versioned in flake repo + sops-nix
|-
| Monitoring data
| VictoriaMetrics (separate retention policy)
| VictoriaMetrics (separate retention policy)
|}

== Cadence ==

| Backup type | Frequency | Retention |
|---|---|---|
| etcd snapshot | Every 2 hours | 30 days |
| Longhorn volume backup | Daily | 90 days |
| ZFS snapshot | Every 6 hours | 7 days local, 90 days remote |
| ZFS send to S3 | Daily | 90 days |
| VictoriaMetrics data | Per retention policy | 45 days default |

== Restore procedure ==

=== etcd restore ===

# Stop K3s server on all CP nodes
# Restore etcd from the latest snapshot on a single node
# Start K3s server on the restored node
# Add remaining CP nodes back to the cluster

=== Longhorn restore (Path A) ===

# Create a new PVC from the Longhorn backup in the S3 target
# Attach to the replacement pod
# Verify data integrity

=== ZFS restore (Path B) ===

# Receive the ZFS send stream from S3 or the backup target
# Rollback the dataset to the received snapshot
# Verify data integrity

== Off-site backup ==

All backups target an S3-compatible object store in a separate geography from the CP VMs. This survives a full region failure.

{{Related}}