rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Signals in /metrics and on the dashboard; failed Jev calls are failed spans (#270)

Browse Source 
The judge had spans but nothing in Prometheus: no count of items judged or
of how calls to Jev ended, nor whether the judge was still passing. The jev
span recorded an error field without the span's status, so Tempo's failed
traces left Jev out, and a timeout recorded nothing. Calls now send
traceparent like every other call out of ipx.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-17 02:55:37 +00:00
parent 3b48f51a56
commit 33eb1f4db2
5 changed files with 135 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
grafana/dashboard.py
View File

@@ -254,6 +254,28 @@ table("Failed downloads", [loki(f'sum by (feed, msg) (count_over_time({EV} | ev=
12, rename={"feed": "Feed", "msg": "Error", "Value": "Failures"}, sort=[{"displayName": "Failures", "desc": True}])
y += 8
# ---- ipx: Signals, items put to TypeSafe's Jev on each person's own key (#254)
row("ipx: Signals")
ts("Items judged", [prom('sum(increase(ipx_signals_judged_total{job="ipx"}[$__rate_interval]))', legend="judged"),
prom('sum(increase(ipx_signals_hits_total{job="ipx"}[$__rate_interval]))', ref="B", legend="fit a Signal")],
0, w=8, bars=True, desc="Each item is judged once against every Signal of its subscribers that has not had it; a burst follows a scan or a new Signal.")
ts("Calls to Jev", [prom('sum by (result) (increase(ipx_jev_requests_total{job="ipx"}[$__rate_interval]))', legend="{{result}}"),
prom('sum(increase(ipx_jev_seconds_total{job="ipx"}[$__rate_interval])) / '
'sum(increase(ipx_jev_requests_total{job="ipx"}[$__rate_interval]))', ref="B", legend="seconds a call")],
8, w=8, bars=True, stack=True,
desc="key_refused stops that person's Signals until they save another key; malformed stops the pass and is ipx's to fix; "
"error is TypeSafe failing or slow, and the item is asked again next pass.")["fieldConfig"]["overrides"] = [
{"matcher": {"id": "byName", "options": "seconds a call"},
"properties": [{"id": "unit", "value": "s"}, {"id": "custom.axisPlacement", "value": "right"},
{"id": "custom.drawStyle", "value": "line"}, {"id": "custom.stacking", "value": {"mode": "none"}}]}]
ts("Signals judge", [prom('max(ipx_signals_last_pass_age_seconds{job="ipx"})', legend="since the last pass"),
prom('max(ipx_signals_keys_refused{job="ipx"})', ref="B", legend="keys refused")],
16, w=8, unit="s", desc="It passes at least every five minutes; much longer means it is stuck on Jev. Keys refused, on the right, "
"are people whose Signals wait for a new key.")["fieldConfig"]["overrides"] = [
{"matcher": {"id": "byName", "options": "keys refused"},
"properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]}]
y += 8
# ---- ipx: the process, what it holds between requests and jobs (#194)
row("ipx: process")
# Open connections stay open, idle, for ten minutes after a burst, so open at the max is not