rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Signals: calls to TypeSafe paced under its rate limits, a 429 honoured, and the wait traced, counted and drawn (#277)

Browse Source 
TypeSafe allows 80 requests and 100,000 input tokens a second. Six calls
in flight was a cap on concurrency, not rate: at 74 ms a call it is 81 a
second. Every call to Jev now takes a turn, a 60th of a second or its
guessed tokens' share of 80,000, so up to ten can be in flight. A 429
holds every call back for Retry-After and ends the pass. Each jev span
records paced_ms, /metrics ipx_jev_paced_seconds_total and a rate_limited
result, and the dashboard draws calls and tokens a second against the
limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-17 14:38:20 +00:00
parent 5130f754cc
commit ce78599ab4
6 changed files with 115 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
grafana/dashboard.py
View File

@@ -260,11 +260,12 @@ ts("Items judged", [prom('sum(increase(ipx_signals_judged_total{job="ipx"}[$__ra
prom('sum(increase(ipx_signals_hits_total{job="ipx"}[$__rate_interval]))', ref="B", legend="fit a Signal")],
0, w=8, bars=True, desc="Each item is judged once against every Signal of its subscribers that has not had it; a burst follows a scan or a new Signal.")
ts("Calls to Jev", [prom('sum by (result) (increase(ipx_jev_requests_total{job="ipx"}[$__rate_interval]))', legend="{{result}}"),
prom('sum(increase(ipx_jev_seconds_total{job="ipx"}[$__rate_interval])) / '
prom('(sum(increase(ipx_jev_seconds_total{job="ipx"}[$__rate_interval])) - sum(increase(ipx_jev_paced_seconds_total{job="ipx"}[$__rate_interval]))) / '
'sum(increase(ipx_jev_requests_total{job="ipx"}[$__rate_interval]))', ref="B", legend="seconds a call")],
8, w=8, bars=True, stack=True,
desc="key_refused stops that person's Signals until they save another key; malformed stops the pass and is ipx's to fix; "
"error is TypeSafe failing or slow, and the item is asked again next pass.")["fieldConfig"]["overrides"] = [
"rate_limited is TypeSafe saying slow down (#277), and every call waits; "
"error is TypeSafe failing or slow, and the item is asked again next pass. Seconds a call is TypeSafe's, without the wait for a turn.")["fieldConfig"]["overrides"] = [
{"matcher": {"id": "byName", "options": "seconds a call"},
"properties": [{"id": "unit", "value": "s"}, {"id": "custom.axisPlacement", "value": "right"},
{"id": "custom.drawStyle", "value": "line"}, {"id": "custom.stacking", "value": {"mode": "none"}}]}]
@@ -276,10 +277,19 @@ ts("Signals judge", [prom('max(ipx_signals_last_pass_age_seconds{job="ipx"})', l
"properties": [{"id": "unit", "value": "short"}, {"id": "custom.axisPlacement", "value": "right"}]}]
y += 8
ts("Tokens", [prom('sum by (kind) (increase(ipx_jev_tokens_total{job="ipx"}[$__rate_interval]))', legend="{{kind}}")],
0, w=12, bars=True, stack=True, desc="What TypeSafe counted for each call, as its answers say: what Signals, icons and key checks cost.")
0, w=8, bars=True, stack=True, desc="What TypeSafe counted for each call, as its answers say: what Signals, icons and key checks cost.")
ts("Tokens a call", [prom('sum by (kind) (increase(ipx_jev_tokens_total{job="ipx"}[$__rate_interval])) / ignoring(kind) group_left '
'sum(increase(ipx_jev_requests_total{job="ipx", result="ok"}[$__rate_interval]))', legend="{{kind}}")],
12, w=12, desc="An item's text is most of the input; on the server's key, one call carries everyone's Signals on that item.")
8, w=8, desc="Most of the input is the questions, one for each Signal, then the item's text, cut at 1,000 characters.")
# TypeSafe's default limits (#277): ipx paces itself to 60 calls and 80,000 input tokens a second.
LIMIT = lambda n: [{"color": "transparent", "value": None}, {"color": "red", "value": n}]
p = ts("Against TypeSafe's limits", [prom('sum(rate(ipx_jev_requests_total{job="ipx"}[$__rate_interval]))', legend="calls a second"),
prom('sum(rate(ipx_jev_tokens_total{job="ipx", kind="input"}[$__rate_interval])) / 1000', ref="B", legend="thousand input tokens a second"),
prom('sum(rate(ipx_jev_paced_seconds_total{job="ipx"}[$__rate_interval]))', ref="C", legend="seconds waited for a turn, a second")],
16, w=8, desc="TypeSafe allows 80 calls and 100,000 input tokens a second; the red line is 80 on both scales (calls, and thousands of tokens). "
"Waiting for a turn means ipx is holding itself under them, and is the time a new Signal's backfill takes longer for it.")
p["fieldConfig"]["defaults"]["custom"]["thresholdsStyle"] = {"mode": "line"}
p["fieldConfig"]["defaults"]["thresholds"] = {"mode": "absolute", "steps": LIMIT(80)}
y += 8
# ---- ipx: the process, what it holds between requests and jobs (#194)