rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Nothing alerts when ipx goes wrong #365

New Issue
Closed
opened 2026-10-25 08:56:23 -07:00 by rays · 1 comment
rays commented 2026-10-25 08:56:23 -07:00
Owner
Copy Link

Found reviewing what production's traces and metrics leave out, 2026-10-25. Prometheus has no rules and Grafana no alerting provisioned, so a stuck worker, a full pool or ipx down waits for someone to run the prod check; Tempo's search was broken for 15 hours on 2026-10-25 unnoticed. Wanted: Grafana alert rules from the repo, beside the dashboard: ipx or the directory down, the worker stuck (last scan over 15 min, a job over 30 min), the pool full, 5xx, Jev rate-limited or malformed, Tower's load and memory, Tempo's block poll failing. No contact point yet: they show as firing in Grafana's Alerting page until one is added.

Found reviewing what production's traces and metrics leave out, 2026-10-25. Prometheus has no rules and Grafana no alerting provisioned, so a stuck worker, a full pool or ipx down waits for someone to run the prod check; Tempo's search was broken for 15 hours on 2026-10-25 unnoticed. Wanted: Grafana alert rules from the repo, beside the dashboard: ipx or the directory down, the worker stuck (last scan over 15 min, a job over 30 min), the pool full, 5xx, Jev rate-limited or malformed, Tower's load and memory, Tempo's block poll failing. No contact point yet: they show as firing in Grafana's Alerting page until one is added.
rays added the enhancement label 2026-10-25 08:56:23 -07:00
rays commented 2026-10-25 09:35:48 -07:00
Author
Owner
Copy Link

Fixed in dd91602 and 34-line follow-up c4f0a36: grafana/alerts.py writes 16 Grafana alert rules to the monitoring project's grafana-provisioning/alerting/ipx.json (ipx, directory or ipx-web down; scans stopped; a job over 30 min; the pool full; 5xx; 502s; Jev refused; Signals stopped; the directory's refresher; Tower's load and memory; monitoring down; Tempo's blocks; iPX's lines not reaching Loki). No contact point yet: firing shows in Grafana's Alerting page. All healthy after the deploy. Deployed 2026-10-25.

Fixed in dd91602 and 34-line follow-up c4f0a36: grafana/alerts.py writes 16 Grafana alert rules to the monitoring project's grafana-provisioning/alerting/ipx.json (ipx, directory or ipx-web down; scans stopped; a job over 30 min; the pool full; 5xx; 502s; Jev refused; Signals stopped; the directory's refresher; Tower's load and memory; monitoring down; Tempo's blocks; iPX's lines not reaching Loki). No contact point yet: firing shows in Grafana's Alerting page. All healthy after the deploy. Deployed 2026-10-25.
rays closed this issue 2026-10-25 09:35:48 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
rays
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rays/ipx#365
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?