TELEICU/Monitoring & observability
More languages
More actions
The monitoring architecture for the TELEICU fleet.
Architecture
VictoriaMetrics
Chosen over Prometheus for:
- Efficiency — significantly lower resource usage per metric at 200+ node scale
- Remote-write — native support for buffered writes from vmagent, critical for unreliable connectivity
- Downsampling — automatic downsampling of older data to reduce storage
- Single-node or cluster — can start small and scale
On-device downtime classifier
Each node runs a small agent that:
- Monitors power state (AC status via /sys, UPS if available)
- Monitors network connectivity (Tailscale status, upstream reachability)
- Monitors K3s agent health
- Classifies any outage into: power, network, or other
- Reports classification to VictoriaMetrics as an event
This enables the incident management system to route alerts to the correct responder.
Key dashboards
| Dashboard | Purpose | |---|---| | Fleet overview | All nodes, status by state, online/offline counts | | Per-node view | CPU, memory, disk, network, power status, K3s health | | Downtime classification | Pie chart of outage causes across the fleet | | Patch compliance | OS version / NixOS generation distribution | | Middleware health | Per-service HTTP health check status |
Alerts
| Alert | Condition | Severity | |---|---|---| | Node offline | Heartbeat missing > 5 min | P3 → P2 → P1 escalation | | Downtime: power | Classified as power loss | P2 | | Downtime: network | Classified as network loss | P2 | | Downtime: other | Classified as unknown/other | P1 | | etcd leader changes | > 1 change in 10 min | P2 | | Disk space > 85% | On any node or CP VM | P3 | | Patch delta | Node > 30 days behind latest generation | P3 |