rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Metrics for the worker's job, the queues and the work in flight (#195)

Browse Source 
The healthcheck answers while the worker is stuck on one job, and nothing
showed which job or for how long. STATS, one static of atomics, is kept as
the work happens: the worker's job and since when (a guard, so a failed or
stopped job is counted too), jobs and their seconds by kind, when a scan
last finished, web requests, feed fetches and downloads in progress, bytes
downloaded, live updates dropped for a listener that fell behind, DeadAir's
backlog and how long it has been on its file, and the artwork kept. With
the event queue, the DNS retries and the healthcheck's last counts, /metrics
still reads memory only. The dashboard's Daemon row and the prod-check
skill take them up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-11 15:14:54 +00:00
parent a985cfafef
commit 29f1cbce29
9 changed files with 254 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
grafana/dashboard.py
View File

@@ -163,6 +163,33 @@ y += 8
ts("Memory", [prom('ipx_process_resident_bytes{job="ipx"}', legend="resident")], 0, w=12, unit="bytes")
ts("Open files and sockets", [prom('ipx_process_open_fds{job="ipx"}', legend="open")], 12, w=12)
y += 8
# Jobs and queues (#195)
ts("Worker", [prom('max by (kind) (ipx_worker_job_seconds{job="ipx"})', legend="on a {{kind}} for"),
prom('max(ipx_last_scan_age_seconds{job="ipx"})', ref="B", legend="since the last scan finished")],
0, w=8, unit="s", desc="A job's time that only climbs, or no scan for over 10 minutes, means the worker is stuck; the healthcheck still passes.")
ts("Jobs done", [prom('sum by (kind) (increase(ipx_jobs_total{job="ipx"}[$__rate_interval]))', legend="{{kind}}")],
8, w=8, bars=True, stack=True)
ts("Time on jobs", [prom('sum by (kind) (increase(ipx_job_seconds_total{job="ipx"}[$__rate_interval]))', legend="{{kind}}")],
16, w=8, unit="s", bars=True, stack=True, desc="Seconds the worker spent on each kind of job in each interval.")
y += 8
ts("In flight", [prom('max(ipx_http_requests_in_flight{job="ipx"})', legend="web requests"),
prom('max(ipx_scan_feeds_fetching{job="ipx"})', ref="B", legend="feeds fetching"),
prom('max(ipx_downloads_active{job="ipx"})', ref="C", legend="downloads"),
prom('2 - max(ipx_torrent_slots_free{job="ipx"})', ref="D", legend="torrents")],
0, w=8)
ts("Download speed", [prom('sum(rate(ipx_download_bytes_total{job="ipx"}[$__rate_interval]))', legend="downloading")],
8, w=8, unit="Bps")
ts("Backlogs", [prom('max(ipx_deadair_backlog{job="ipx"})', legend="DeadAir to look at"),
prom('max(ipx_feed_retries{job="ipx"})', ref="B", legend="feeds to retry a lookup"),
prom('max(ipx_event_queue{job="ipx"})', ref="C", legend="live updates unread"),
prom('sum(increase(ipx_events_missed_total{job="ipx"}[$__rate_interval]))', ref="D", legend="live updates missed")],
16, w=8)
y += 8
ts("DeadAir on a file", [prom('max(ipx_deadair_busy_seconds{job="ipx"})', legend="for")], 0, w=12, unit="s",
desc="How long DeadAir has been on the file it is analysing; a few seconds each.")
ts("Artwork kept", [prom('max(ipx_art_cache_bytes{job="ipx"})', legend="kept"),
prom('max(ipx_art_cache_limit_bytes{job="ipx"})', ref="B", legend="the most kept")], 12, w=12, unit="bytes")
y += 8
# ---- Traces
row("Traces")