Admin: Fix Mermaid diagram: remove

 wrapping so parser function executes (via update-page on MediaWiki MCP Server)

2026-07-06T03:18:30Z

Fix Mermaid diagram: remove <pre> wrapping so parser function executes (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 08:48, 6 July 2026
Line 10:		Line 10:
	== Architecture ==		== Architecture ==

	<pre>
	{{#mermaid:flowchart TB		{{#mermaid:flowchart TB
	subgraph Agent["Per-Node (200+)"]		subgraph Agent["Per-Node (200+)"]
Line 30:		Line 29:
	VM --> ALERT		VM --> ALERT
	}}		}}
	</pre>

	== VictoriaMetrics ==		== VictoriaMetrics ==

Admin: Automated edit (via update-page on MediaWiki MCP Server)

2026-07-06T03:10:37Z

Automated edit (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 08:40, 6 July 2026
Line 74:		Line 74:
	\| Disk space > 85% \| On any node or CP VM \| P3 \|		\| Disk space > 85% \| On any node or CP VM \| P3 \|
	\| Patch delta \| Node > 30 days behind latest generation \| P3 \|		\| Patch delta \| Node > 30 days behind latest generation \| P3 \|

			{{Navbox TELEICU}}

	{{Related}}		{{Related}}

Admin: Create TELEICU monitoring and observability reference (via create-page on MediaWiki MCP Server)

2026-07-06T02:55:41Z

Create TELEICU monitoring and observability reference (via create-page on MediaWiki MCP Server)

New page

{{Doc header
|type=reference
|domain=teleicu
|title=Monitoring & observability
|order=19
}}

The monitoring architecture for the TELEICU fleet.

== Architecture ==

<pre>
{{#mermaid:flowchart TB
subgraph Agent["Per-Node (200+)"]
VMA["vmagent (metrics scraper)"]
DCA["Downtime Classifier"]
LOGS["Vector / Alloy (log shipper)"]
end

subgraph Central["Central (CP VMs or separate)"]
VM["VictoriaMetrics (time-series DB)"]
GRAFANA["Grafana (dashboards)"]
ALERT["Alertmanager"]
end

VMA -->|"remote-write (buffered on disk)"| VM
DCA -->|"classification events"| VM
LOGS -->|"buffered log stream"| VM
VM --> GRAFANA
VM --> ALERT
}}
</pre>

== VictoriaMetrics ==

Chosen over Prometheus for:

* '''Efficiency''' — significantly lower resource usage per metric at 200+ node scale
* '''Remote-write''' — native support for buffered writes from vmagent, critical for unreliable connectivity
* '''Downsampling''' — automatic downsampling of older data to reduce storage
* '''Single-node or cluster''' — can start small and scale

== On-device downtime classifier ==

Each node runs a small agent that:

* Monitors power state (AC status via /sys, UPS if available)
* Monitors network connectivity (Tailscale status, upstream reachability)
* Monitors K3s agent health
* Classifies any outage into: power, network, or other
* Reports classification to VictoriaMetrics as an event

This enables the incident management system to route alerts to the correct responder.

== Key dashboards ==

| Dashboard | Purpose |
|---|---|
| Fleet overview | All nodes, status by state, online/offline counts |
| Per-node view | CPU, memory, disk, network, power status, K3s health |
| Downtime classification | Pie chart of outage causes across the fleet |
| Patch compliance | OS version / NixOS generation distribution |
| Middleware health | Per-service HTTP health check status |

== Alerts ==

| Alert | Condition | Severity |
|---|---|---|
| Node offline | Heartbeat missing > 5 min | P3 → P2 → P1 escalation |
| Downtime: power | Classified as power loss | P2 |
| Downtime: network | Classified as network loss | P2 |
| Downtime: other | Classified as unknown/other | P1 |
| etcd leader changes | > 1 change in 10 min | P2 |
| Disk space > 85% | On any node or CP VM | P3 |
| Patch delta | Node > 30 days behind latest generation | P3 |

{{Related}}

TELEICU/Monitoring & observability - Revision history

Admin: Fix Mermaid diagram: remove
wrapping so parser function executes (via update-page on MediaWiki MCP Server)

Admin: Automated edit (via update-page on MediaWiki MCP Server)

Admin: Create TELEICU monitoring and observability reference (via create-page on MediaWiki MCP Server)