rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

The directory's refresher alert waits out a restart (#365)

Browse Source 
Its rounds come in bursts about twelve hours apart and its count starts
again at each restart, so for hours after a deploy no check had been
counted and the rule would have fired. It judges only once the directory
has been up 13 hours.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-25 16:35:46 +00:00
parent dd91602a49
commit c4f0a362d8
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
grafana/alerts.py
View File

@@ -62,8 +62,11 @@ rules = [
"TypeSafe refused calls as over its limits (#277) or malformed (#257): ipx is asking wrongly."), "TypeSafe refused calls as over its limits (#277) or malformed (#257): ipx is asking wrongly."),
rule("signals-stale", "Signals have stopped", 'max(ipx_signals_last_pass_age_seconds{job="ipx"})', "gt", 900, "5m", rule("signals-stale", "Signals have stopped", 'max(ipx_signals_last_pass_age_seconds{job="ipx"})', "gt", 900, "5m",
"The Signals judge has not finished a pass for 15 minutes; its signals trace shows which Jev call holds it."), "The Signals judge has not finished a pass for 15 minutes; its signals trace shows which Jev call holds it."),
# Only once it has been up 13 hours: its rounds come in bursts about twelve hours apart, and the
# count starts again at each restart, so a deploy would read as stopped.
rule("directory-refresh", "The directory has stopped checking feeds", rule("directory-refresh", "The directory has stopped checking feeds",
'sum(increase(directory_feed_checks_total{job="ipx-directory"}[13h]))', "lt", 1, "30m", 'sum(increase(directory_feed_checks_total{job="ipx-directory"}[13h])) < 1'
' and on() max(directory_uptime_seconds{job="ipx-directory"}) > 46800', "lt", 1, "30m",
"No feed checked in 13 hours; each is due every 12.", window=46800), "No feed checked in 13 hours; each is due every 12.", window=46800),
# Tower, which everything shares: a stall here is every request slow (2026-10-18, load 160). # Tower, which everything shares: a stall here is every request slow (2026-10-18, load 160).
rule("load", "Tower is overloaded", 'max(node_load5{job="node"})', "gt", 48, "10m", rule("load", "Tower is overloaded", 'max(node_load5{job="node"})', "gt", 48, "10m",