rays
dd91602a49
Reviewing what production's traces and metrics leave out turned up twelve gaps; each is an issue. Metrics. A Prometheus histogram for ipx and the directory both (ipx_directory::metrics::Histograms), which ipx's request histogram (#364) now uses too. - ipx: each statement's time once it has a connection, by its first word, from SeaORM's metric callback; and the wait for one, for hand-written SQL in db.rs, measured as the call less the statement (the callback runs in the poll that returns the result, so the time read after the await is that statement's) (#367). SeaORM's own queries are timed but their wait is not seen: a shortcut, noted. - ipx: counters of what the events say, counted in emit, which every event passes once: feed failures by error.type, new items, downloads saved and failed, DeadAir's seconds found (#368). - ipx: the page's list_shown, play_start and stall timings (#369). - directory: requests by method, route (a route layer puts the matched route on the response, or each feed's page was a row) and status; feeds checked and failed; snapshots taken (#370). - ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11 calls deprecated, because the browser tests' Caddy is 2.6 (#371). Traces. - A list's sync is list_sync with steps, an OPML upload opml_import (#372). Doing so showed sync_group reading every account and its subscription to the list once per listed feed; read once now (#377). - Each command the worker runs, from the socket, the CLI or the page, is a command span with its job inside (#373). - The daemon's start is a trace, from opening the database through the catalogue's repairs to "daemon started" (#374). - The directory's lines carry trace_id and span_id: WithTrace moves to ipx_directory::otel and both use it (#375). - A tap or key is its trace's root span (#376). The page names a root span id in each request's traceparent, where the parent was random and pointed at nothing, and reports the tap (ids, start, its last answer) when the next begins or the page is hidden. opentelemetry 0.33 gives a span builder no way to take ids, so the shared tracer provider has an id generator that takes given ids inside with_ids, on one thread. Alerts. grafana/alerts.py, Grafana alerting provisioning beside the dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals stopped, the directory's refresher stopped, Tower's load and memory, the monitoring down, Tempo not reading its blocks, iPX's lines not reaching Loki. No contact point yet: firing shows in Grafana. Prometheus now scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's config, not this repository's. The dashboard has database and page-timing panels; the prod check knows the new metrics, spans and alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
89 lines
6.1 KiB
Python
89 lines
6.1 KiB
Python
"""The iPX alert rules in Grafana (#365): what should be looked at now rather than at the next prod
|
|
check, written out as Grafana's alerting provisioning, beside the dashboard.
|
|
|
|
python3 grafana/alerts.py > /mnt/fast/arcane/projects/monitoring/grafana-provisioning/alerting/ipx.json
|
|
|
|
Grafana reads it at start, or at once from POST /api/admin/provisioning/alerting/reload. There is
|
|
no contact point yet: a rule that fires shows in Grafana's Alerting page, and adding one (email,
|
|
ntfy, a webhook) is a file in the same folder. Each rule is a query and a threshold on it; the
|
|
query's value is in the alert, so it says how bad as well as what.
|
|
"""
|
|
import json
|
|
|
|
# Prometheus was added in Grafana's settings, not provisioned, so its uid is Grafana's own.
|
|
PROM = "eenybu3ao2t4wb"
|
|
LOKI = "P8E80F9AEF21F6940"
|
|
|
|
|
|
def rule(uid, title, expr, op, value, for_, summary, *, ds=PROM, window=600, no_data="OK"):
|
|
"""Fires when `expr` is `op` (gt, lt) `value` for `for_`. No data is fine by default: most of
|
|
these metrics are absent until there is something to say (no job running, no scan yet)."""
|
|
return {
|
|
"uid": f"ipx-{uid}", "title": title, "condition": "C", "for": for_,
|
|
"data": [
|
|
{"refId": "A", "relativeTimeRange": {"from": window, "to": 0}, "datasourceUid": ds,
|
|
# Loki's instant query is a queryType; Prometheus's a flag.
|
|
"model": {"refId": "A", "expr": expr,
|
|
**({"queryType": "instant"} if ds == LOKI else {"instant": True})}},
|
|
{"refId": "C", "datasourceUid": "__expr__",
|
|
"model": {"refId": "C", "type": "threshold", "expression": "A",
|
|
"conditions": [{"evaluator": {"type": op, "params": [value]}}]}},
|
|
],
|
|
"noDataState": no_data, "execErrState": "Error",
|
|
"annotations": {"summary": summary},
|
|
"labels": {"service": "ipx"},
|
|
}
|
|
|
|
|
|
rules = [
|
|
# Down: Prometheus cannot reach it. A container gone is no data at all, so that fires too.
|
|
rule("down", "ipx is down", 'up{job="ipx"}', "lt", 1, "2m",
|
|
"Prometheus cannot reach ipx's /metrics: the iPX container is down or restarting.", no_data="Alerting"),
|
|
rule("directory-down", "ipx-directory is down", 'up{job="ipx-directory"}', "lt", 1, "5m",
|
|
"Prometheus cannot reach the directory's /metrics.", no_data="Alerting"),
|
|
rule("web-down", "ipx-web is down", 'up{job="ipx-web"}', "lt", 1, "5m",
|
|
"Prometheus cannot reach Caddy's /metrics: the page is likely down too.", no_data="Alerting"),
|
|
# The worker: the daemon answers `ipx status` while it is stuck, so only these say so.
|
|
rule("scan-stale", "ipx's scans have stopped", 'max(ipx_last_scan_age_seconds{job="ipx"})', "gt", 900, "5m",
|
|
"No scan has finished for over 15 minutes; scans run at least every 10. The worker is stuck: its job and trace say on what."),
|
|
rule("job-long", "ipx's worker is stuck on one job", 'max(ipx_worker_job_seconds{job="ipx"})', "gt", 1800, "1m",
|
|
"One job has run for over 30 minutes (ipx_worker_job_seconds says which kind)."),
|
|
rule("pool-full", "ipx's database connections are all in use",
|
|
'max_over_time(ipx_db_connections{job="ipx",state="idle"}[5m])', "lt", 1, "5m",
|
|
"No idle connection for five minutes: requests and scans are waiting for one (ipx_db_wait_seconds)."),
|
|
rule("5xx", "ipx is answering with server errors",
|
|
'sum(increase(ipx_http_request_duration_seconds_count{job="ipx",status=~"5.."}[10m]))', "gt", 5, "0s",
|
|
"More than five 5xx answers in ten minutes; ipx's log has a warning for each, with its trace."),
|
|
rule("502", "ipx-web cannot reach ipx",
|
|
'sum(increase(caddy_http_request_duration_seconds_count{job="ipx-web",code=~"502|504"}[10m]))', "gt", 5, "0s",
|
|
"Caddy answered 502 or 504 for ipx more than five times in ten minutes: ipx is not answering it."),
|
|
rule("jev", "ipx's calls to Jev are refused",
|
|
'sum(increase(ipx_jev_requests_total{job="ipx",result=~"rate_limited|malformed"}[15m]))', "gt", 0, "0s",
|
|
"TypeSafe refused calls as over its limits (#277) or malformed (#257): ipx is asking wrongly."),
|
|
rule("signals-stale", "Signals have stopped", 'max(ipx_signals_last_pass_age_seconds{job="ipx"})', "gt", 900, "5m",
|
|
"The Signals judge has not finished a pass for 15 minutes; its signals trace shows which Jev call holds it."),
|
|
rule("directory-refresh", "The directory has stopped checking feeds",
|
|
'sum(increase(directory_feed_checks_total{job="ipx-directory"}[13h]))', "lt", 1, "30m",
|
|
"No feed checked in 13 hours; each is due every 12.", window=46800),
|
|
# Tower, which everything shares: a stall here is every request slow (2026-10-18, load 160).
|
|
rule("load", "Tower is overloaded", 'max(node_load5{job="node"})', "gt", 48, "10m",
|
|
"Load over 48 on twelve threads for ten minutes: builds, or processes blocked on a disk."),
|
|
rule("memory", "Tower is out of memory", 'min(node_memory_MemAvailable_bytes{job="node"})', "lt", 2e9, "5m",
|
|
"Under 2 GB available, with no swap: next comes stalling on page faults, then the OOM killer."),
|
|
# The monitoring itself (#366): nothing else watches it.
|
|
rule("monitoring-down", "Part of the monitoring is down", 'min by (job) (up{job=~"prometheus|tempo|loki|alloy"})',
|
|
"lt", 1, "5m", "Prometheus cannot reach Tempo, Loki or Alloy."),
|
|
rule("tempo-blocks", "Tempo cannot read its stored traces",
|
|
'(max(tempodb_blocklist_tenant_index_age_seconds) > bool 900) + (max(tempodb_blocklist_length) < bool 1)',
|
|
"gt", 0, "10m",
|
|
"Tempo's look over its stored blocks is failing (2026-10-25: one empty block); searches see only the last 15 "
|
|
"minutes. Its log says which block: move it out of /mnt/user/logs/tempo/blocks/single-tenant."),
|
|
rule("logs-stopped", "No lines from iPX reach Loki", 'sum(count_over_time({container="iPX"}[3h]))', "lt", 1, "0s",
|
|
"Loki has nothing from iPX in three hours: Alloy is not shipping it, or the container is down.",
|
|
ds=LOKI, window=10800, no_data="Alerting"),
|
|
]
|
|
|
|
print(json.dumps({"apiVersion": 1, "groups": [
|
|
{"orgId": 1, "name": "ipx", "folder": "iPX", "interval": "1m", "rules": rules},
|
|
]}, indent=2))
|