rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Prometheus does not scrape the monitoring itself #366

New Issue
Closed
opened 2026-10-25 08:56:24 -07:00 by rays · 1 comment
rays commented 2026-10-25 08:56:24 -07:00
Owner
Copy Link

Found reviewing what production's traces and metrics leave out, 2026-10-25. Prometheus scrapes ipx, the directory and node, not Tempo, Loki, Alloy or itself. Tempo counts the failure it had on 2026-10-25 (tempodb_blocklist_poll_errors_total) and Alloy what it ships; neither was looked at. Wanted: scrape jobs for all four, and an alert on Tempo's poll errors.

Found reviewing what production's traces and metrics leave out, 2026-10-25. Prometheus scrapes ipx, the directory and node, not Tempo, Loki, Alloy or itself. Tempo counts the failure it had on 2026-10-25 (tempodb_blocklist_poll_errors_total) and Alloy what it ships; neither was looked at. Wanted: scrape jobs for all four, and an alert on Tempo's poll errors.
rays added the enhancement label 2026-10-25 08:56:24 -07:00
rays commented 2026-10-25 09:35:49 -07:00
Author
Owner
Copy Link

Fixed: Prometheus scrapes prometheus, tempo, loki and alloy (prometheus.yml in /mnt/fast/appdata/prometheus/etc, backup .pre-monitoring); Alloy's command sets --server.http.listen-addr=0.0.0.0:12345 so it can be reached. All up. The Tempo alert reads tempodb_blocklist_tenant_index_age_seconds and tempodb_blocklist_length. Deployed 2026-10-25.

Fixed: Prometheus scrapes prometheus, tempo, loki and alloy (prometheus.yml in /mnt/fast/appdata/prometheus/etc, backup .pre-monitoring); Alloy's command sets --server.http.listen-addr=0.0.0.0:12345 so it can be reached. All up. The Tempo alert reads tempodb_blocklist_tenant_index_age_seconds and tempodb_blocklist_length. Deployed 2026-10-25.
rays closed this issue 2026-10-25 09:35:49 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
rays
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rays/ipx#366
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?