TELEICU/Monitoring & observability: Difference between revisions

Content deleted Content added

Inline

Revision as of 08:40, 6 July 2026

referenceteleicuCARE 3.0+

The monitoring architecture for the TELEICU fleet.

Architecture

{{#mermaid:flowchart TB
    subgraph Agent["Per-Node (200+)"]
        VMA["vmagent<br/>(metrics scraper)"]
        DCA["Downtime<br/>Classifier"]
        LOGS["Vector / Alloy<br/>(log shipper)"]
    end

    subgraph Central["Central (CP VMs or separate)"]
        VM["VictoriaMetrics<br/>(time-series DB)"]
        GRAFANA["Grafana<br/>(dashboards)"]
        ALERT["Alertmanager"]
    end

    VMA -->|"remote-write<br/>(buffered on disk)"| VM
    DCA -->|"classification<br/>events"| VM
    LOGS -->|"buffered<br/>log stream"| VM
    VM --> GRAFANA
    VM --> ALERT
}}

VictoriaMetrics

Chosen over Prometheus for:

Efficiency — significantly lower resource usage per metric at 200+ node scale
Remote-write — native support for buffered writes from vmagent, critical for unreliable connectivity
Downsampling — automatic downsampling of older data to reduce storage
Single-node or cluster — can start small and scale

On-device downtime classifier

Each node runs a small agent that:

Monitors power state (AC status via /sys, UPS if available)
Monitors network connectivity (Tailscale status, upstream reachability)
Monitors K3s agent health
Classifies any outage into: power, network, or other
Reports classification to VictoriaMetrics as an event

This enables the incident management system to route alerts to the correct responder.

Key dashboards

| Dashboard | Purpose | |---|---| | Fleet overview | All nodes, status by state, online/offline counts | | Per-node view | CPU, memory, disk, network, power status, K3s health | | Downtime classification | Pie chart of outage causes across the fleet | | Patch compliance | OS version / NixOS generation distribution | | Middleware health | Per-service HTTP health check status |

Alerts

| Alert | Condition | Severity | |---|---|---| | Node offline | Heartbeat missing > 5 min | P3 → P2 → P1 escalation | | Downtime: power | Classified as power loss | P2 | | Downtime: network | Classified as network loss | P2 | | Downtime: other | Classified as unknown/other | P1 | | etcd leader changes | > 1 change in 10 min | P2 | | Disk space > 85% | On any node or CP VM | P3 | | Patch delta | Node > 30 days behind latest generation | P3 |

@@ Line 74: / Line 74: @@
 | Disk space > 85% | On any node or CP VM | P3 |
 | Patch delta | Node > 30 days behind latest generation | P3 |
+{{Navbox TELEICU}}
 {{Related}}