Availability checks, notifications and metrics

Configure HTTP, TCP or external checks in Network & security → Availability checks. HTTP requires an expected status (200 by default) and optional response text; redirects are observed rather than followed. Content checks read at most 64 KiB. Deadlines are 100ms–10s and intervals are 15s–1h. At most 50 checks persist; two due probes run per scheduler cycle within the existing optional-collector concurrency/deadline budget. Port connectivity alone is not application health.

The 24-hour summary reports observed uptime plus coverage and unknown time. A sample covers at most 1.5 configured intervals or until the next sample. Before the first observation, after an overdue sample, during maintenance/disabled periods, and while the app is offline, coverage is unknown. Unknown periods never count as healthy. Down transitions open one persisted incident; repeated down results do not duplicate it; a subsequent up result closes it. Incidents can be acknowledged; one-hour maintenance is available in the UI, with bounded custom end times through the authenticated API.

Local checks cannot observe an outage of their own machine. Run node scripts/external-observer.js https://your-host/ready /secure/path/receipts.jsonl on an independent machine. The observer keeps at most 10,000 receipts locally, so results remain available while the console is offline. Its --once mode supports controlled verification. Create an external check and import chronological receipts using authenticated POST /api/v2/checks/ID/observations after recovery. The endpoint accepts only increasing timestamps within the last 24 hours. This is an integration mechanism; no production off-host observer or availability SLO has been deployed or measured here.

For webhook alerts set SYSTEM_OPS_WEBHOOK_URL and optional SYSTEM_OPS_WEBHOOK_TOKEN on the server, then restart. Settings → Notifications shows delivery status and provides a test-delivery action. Tokens/destination credentials are never returned by status APIs. A receiver's 2xx response is the delivery acknowledgement. Outage/recovery events use a persisted outbox and stable Idempotency-Key; retry delays start at 30s, cap at 5m and stop after 10 attempts with visible exhausted status. Receivers must deduplicate keys: delivery is at least once, not exactly once. Maintenance defers delivery; acknowledged/resolved unsent outage messages are suppressed. Notifications cannot send during complete console/host failure; use an off-host alerting service for that case.

Authenticated GET /metrics exports Prometheus text format 0.0.4, with stable seconds/bytes units and at most 32 collector labels. It does not expose URL, IP, path, user-agent or process labels. Scraping reads cached collector state and process self-metrics without launching commands. See observability/prometheus.yml and grafana-dashboard.json. Put the application API password in the protected token file for scraping; that credential currently grants broader API access, so keep it private and use HTTPS or a trusted local transport. The exporter follows the official Prometheus exposition contract.

Prometheus and Grafana are separate optional services with their own CPU, RAM and disk costs; they are not included in Node-only resource claims. CI's observability fixture checks a real authenticated scrape and imports the example dashboard into Grafana. It is not evidence of a production monitoring deployment or a 24-hour soak.