TELEICU/Monitoring & observability: Difference between revisions
More languages
More actions
Automated edit (via update-page on MediaWiki MCP Server) |
Fix Mermaid diagram: remove <pre> wrapping so parser function executes (via update-page on MediaWiki MCP Server) |
||
| Line 10: | Line 10: | ||
== Architecture == |
== Architecture == |
||
<pre> |
|||
{{#mermaid:flowchart TB |
{{#mermaid:flowchart TB |
||
subgraph Agent["Per-Node (200+)"] |
subgraph Agent["Per-Node (200+)"] |
||
| Line 30: | Line 29: | ||
VM --> ALERT |
VM --> ALERT |
||
}} |
}} |
||
</pre> |
|||
== VictoriaMetrics == |
== VictoriaMetrics == |
||
Latest revision as of 08:48, 6 July 2026
The monitoring architecture for the TELEICU fleet.
Architecture
VictoriaMetrics
Chosen over Prometheus for:
- Efficiency — significantly lower resource usage per metric at 200+ node scale
- Remote-write — native support for buffered writes from vmagent, critical for unreliable connectivity
- Downsampling — automatic downsampling of older data to reduce storage
- Single-node or cluster — can start small and scale
On-device downtime classifier
Each node runs a small agent that:
- Monitors power state (AC status via /sys, UPS if available)
- Monitors network connectivity (Tailscale status, upstream reachability)
- Monitors K3s agent health
- Classifies any outage into: power, network, or other
- Reports classification to VictoriaMetrics as an event
This enables the incident management system to route alerts to the correct responder.
Key dashboards
| Dashboard | Purpose | |---|---| | Fleet overview | All nodes, status by state, online/offline counts | | Per-node view | CPU, memory, disk, network, power status, K3s health | | Downtime classification | Pie chart of outage causes across the fleet | | Patch compliance | OS version / NixOS generation distribution | | Middleware health | Per-service HTTP health check status |
Alerts
| Alert | Condition | Severity | |---|---|---| | Node offline | Heartbeat missing > 5 min | P3 → P2 → P1 escalation | | Downtime: power | Classified as power loss | P2 | | Downtime: network | Classified as network loss | P2 | | Downtime: other | Classified as unknown/other | P1 | | etcd leader changes | > 1 change in 10 min | P2 | | Disk space > 85% | On any node or CP VM | P3 | | Patch delta | Node > 30 days behind latest generation | P3 |