rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Signals in /metrics and on the dashboard; failed Jev calls are failed spans (#270)

Browse Source 
The judge had spans but nothing in Prometheus: no count of items judged or
of how calls to Jev ended, nor whether the judge was still passing. The jev
span recorded an error field without the span's status, so Tempo's failed
traces left Jev out, and a timeout recorded nothing. Calls now send
traceparent like every other call out of ipx.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-17 02:55:37 +00:00
parent 3b48f51a56
commit 33eb1f4db2
5 changed files with 135 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -109,6 +109,9 @@ from 2026-10-11 15:15 UTC. All gauges:
| `ipx_feed_retries` | feeds waiting to retry a failed name lookup |
| `ipx_art_cache_bytes`, `ipx_art_cache_limit_bytes` | artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design |
| `ipx_art_evicted_total`, `ipx_art_requests_total{result="hit"\|"miss"}` | counters: images the sweep threw out to stay under the limit, and artwork served from disk or fetched from the publisher; many evicted at every sweep with a falling hit rate means the limit is too small |
| `ipx_signals_judged_total`, `ipx_signals_hits_total` | counters: items Jev judged for Signals, and Signals they fit (#254) |
| `ipx_jev_requests_total{result="ok"\|"key_refused"\|"malformed"\|"error"}`, `ipx_jev_seconds_total` | counters: calls to TypeSafe's Jev, Signals' and icons', by how they ended, and the seconds waited; `malformed` is ipx's request at fault (#257) |
| `ipx_signals_last_pass_age_seconds`, `ipx_signals_keys_refused` | since the Signals judge last finished a pass (at most 5 minutes when well; absent until one has); people whose key TypeSafe refused |
| `directory_process_resident_bytes`, `directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`, `directory_uptime_seconds`, `directory_build_info` | the directory's, `job="ipx-directory"`, from its `metrics_bind` (production `192.168.1.130:9188`, LAN only) |
| `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the healthcheck last counted it (every 30 s) |
@@ -194,6 +197,8 @@ a count says something happened, the lines and traces say why.
$Q prom 'max(max_over_time(ipx_http_requests_in_flight{job="ipx"}[$range]))' 24h
$Q prom 'sum(increase(ipx_events_missed_total{job="ipx"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h
$Q prom 'sum by (result) (increase(ipx_jev_requests_total{job="ipx"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_signals_last_pass_age_seconds{job="ipx"}[$range]))' 24h
```
`up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is
unset. The daemon keeps all 20 connections open on Postgres (#241), so `idle` is the pool's 20 less
@@ -210,6 +215,8 @@ a count says something happened, the lines and traces say why.
where the worker's time went (reap was most of it until #193). Web requests in flight near
the pool's max mean the pool is what they wait on. Live updates missed mean a page showed stale
counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on.
Any `malformed` Jev call is ipx asking wrongly (#257); `key_refused` is someone's key; the Signals
judge over 10 minutes since a pass is stuck on Jev: its `signals` trace shows which `jev` call.
7. **What the page saw.** The browser's own errors and timings, which no server line shows.
```