Toggle menu
Toggle preferences menu
Toggle personal menu
Not logged in
Your IP address will be publicly visible if you make any edits.

Deployment/Kubernetes/Day-2 operations: Difference between revisions

From OHC Network Wiki
Content deleted Content added
OHC identity seed
 
Add Navbox deployment template before Related (via update-page on MediaWiki MCP Server)
 
Line 113: Line 113:


{{Warning|1=Restore is a design goal, not a finished, tested procedure. End-to-end restore is not yet fully automated or documented in this deployment, and several supporting pieces (such as extending and repopulating Longhorn volumes) are still being worked out. Do not treat restore as production-proven until it has been validated against your own backups. Practice and verify a full restore on a throwaway cluster before depending on it.}}
{{Warning|1=Restore is a design goal, not a finished, tested procedure. End-to-end restore is not yet fully automated or documented in this deployment, and several supporting pieces (such as extending and repopulating Longhorn volumes) are still being worked out. Do not treat restore as production-proven until it has been validated against your own backups. Practice and verify a full restore on a throwaway cluster before depending on it.}}

== Maturity: available today vs evolving ==


Use this split to set expectations. The "evolving / planned" column is drawn from the deployment's roadmap.
Use this split to set expectations. The "evolving / planned" column is drawn from the deployment's roadmap.
Line 143: Line 141:


{{Tip|1=Manual Grafana setup, manual secret retrieval, and unautomated restore are the rough edges to plan around. As the roadmap items land (operator-managed Postgres, Terraform-generated secrets, version-locked charts), these workflows will tighten up.}}
{{Tip|1=Manual Grafana setup, manual secret retrieval, and unautomated restore are the rough edges to plan around. As the roadmap items land (operator-managed Postgres, Terraform-generated secrets, version-locked charts), these workflows will tighten up.}}

{{Navbox deployment}}


{{Related}}
{{Related}}

Latest revision as of 09:34, 6 July 2026

guidedeploymentCARE 3.0+

Once Care is running on the cluster (see Deploying Care onto the cluster), day-2 operations cover keeping it observable and recoverable: monitoring, log aggregation, and backups for every stateful component.

This deployment is a work in progress. Some capabilities are running today; others are design goals that are partially built or not yet automated. This page is honest about which is which — treat anything marked "evolving" or "planned" as not-yet-tested, and verify against the cluster before relying on it.

Monitoring and logs

Observability is provided by a Prometheus + Grafana + Loki stack, deployed through the monitoring module:

python deployer.py apply-tofu monitoring

The pieces fit together like this:

Component Role
Prometheus Scrapes and stores metrics from cluster components.
Grafana Dashboards and visualization over metrics and logs.
Loki Log aggregation and storage, queried from Grafana.
Alloy Collects logs from pods/nodes and ships them into Loki.

After the stack is stable, finish the setup in Grafana:

  1. Log in to Grafana using the Kubernetes-autogenerated admin password (read it from the Grafana admin secret in the monitoring namespace — do not hardcode or commit it).
  2. Add Loki as a data source so logs become queryable alongside metrics.
  3. Import the dashboards you need.
📝 Note
Initial Grafana setup (data source + dashboards) is a manual step today. The admin password is generated by Kubernetes — retrieve it from the cluster secret rather than setting a known value.
⚠️ Warning
Alloy is known to emit a noisy fsnotifier-related warning in this environment. It is a known cosmetic issue and does not affect log shipping.

Backups

Each stateful component is backed up on a daily/configurable schedule to S3-compatible external storage. The mechanism differs by component.

PostgreSQL (Care, Odoo, Metabase databases)

PostgreSQL is backed up using the [https://cloudnative-pg.io/ CloudNativePG] operator together with the barman-cloud plugin, which performs continuous WAL archiving and base backups to S3-compatible storage. Install the plugin manifest from the CloudNativePG release:

kubectl apply -f \
  https://github.com/cloudnative-pg/plugin-barman-cloud/releases/download/v0.10.0/manifest.yaml

The plugin needs credentials and a target bucket. Provide an S3 credentials secret and point the archive at your bucket — keep all values in cluster secrets, never in committed YAML:

kubectl create secret generic s3-secret \
  --from-literal=access_key=<s3-access-key> \
  --from-literal=secret_key=<s3-secret-key>
ℹ️ Info
The same Postgres + barman-cloud backup pattern applies to every Postgres instance in the deployment: Care's database, Odoo's database, and Metabase's database. Configure each with its own bucket/prefix.

Volumes (Longhorn snapshots and backups)

Persistent volumes are managed by [https://longhorn.io/ Longhorn]. Longhorn takes volume snapshots and ships volume backups to an S3 target. The backup target is configured with an S3 credentials secret in the longhorn-system namespace — generalize the endpoint and keys to your own provider:

kubectl create secret generic <backup-secret-name> -n longhorn-system \
  --from-literal=AWS_ACCESS_KEY_ID=<s3-access-key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<s3-secret-key> \
  --from-literal=AWS_ENDPOINTS=<s3-endpoint> \
  --from-literal=AWS_REGION=<s3-region>

What gets backed up

Daily/configurable backups are intended for each stateful component:

Component What is protected Mechanism
PostgreSQL (Care) Application database CloudNativePG + barman-cloud → S3
PostgreSQL (Odoo) Odoo database CloudNativePG + barman-cloud → S3
PostgreSQL (Metabase) Metabase database CloudNativePG + barman-cloud → S3
OpenSearch / Elasticsearch Search index data Volume backup to external media
Object store (RustFS) Uploads, audit logs, public data Incremental sync to external store
Persistent volumes (Longhorn) All attached volumes Longhorn snapshots/backups → S3
📝 Note
Redis/Valkey runs in non-persistent mode by design — it holds no durable state and is not backed up.

Restore

A core design goal of this deployment is that a fresh cluster can be restored from backups — rebuild the Kubernetes cluster, then replay Postgres WAL archives, Longhorn volume backups, and the object-store sync to bring Care back up.

⚠️ Warning
Restore is a design goal, not a finished, tested procedure. End-to-end restore is not yet fully automated or documented in this deployment, and several supporting pieces (such as extending and repopulating Longhorn volumes) are still being worked out. Do not treat restore as production-proven until it has been validated against your own backups. Practice and verify a full restore on a throwaway cluster before depending on it.

Use this split to set expectations. The "evolving / planned" column is drawn from the deployment's roadmap.

Available today Evolving / planned
Prometheus + Grafana + Loki monitoring Move Postgres to a cluster operator–managed Helm chart
Alloy log shipping into Loki Generate secrets/passwords via Terraform and auto-configure
Postgres backups via CloudNativePG + barman-cloud → S3 Version-lock all Helm charts
Longhorn volume snapshots/backups → S3 TLS for database and other inter-pod communication
Per-component daily/configurable backups Documented, automated restore-from-backup procedure
Caching layer for the nginx-based frontend
💡 Tip
Manual Grafana setup, manual secret retrieval, and unautomated restore are the rough edges to plan around. As the roadmap items land (operator-managed Postgres, Terraform-generated secrets, version-locked charts), these workflows will tighten up.