rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

ipx-directory traces, and ipx's trace follows its calls there (#213)

Browse Source 
The exporter setup and the traceparent parser move from ipx into ipx_directory::otel, which ipx
already depends on, so both binaries share them. The directory makes a span a request, a child of
the caller's traceparent, and a trace a refresher round with a span a feed; ipx sends the current
span's traceparent on every call to the directory and wraps each share in a span. The dashboard's
Traces row and the prod-check skill cover all three services.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-12 16:19:32 +00:00
parent a3708769a4
commit 44e75df99e
12 changed files with 188 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
grafana/dashboard.py
View File

@@ -25,7 +25,9 @@ EV = SEL + ' |= "\\"ev\\":\\"" | json | __error__="" | ev != ""'
HTTP = SEL + ' |= "\\"target\\":\\"ipx::http\\"" | json | __error__="" | path != "/api/events"'
BAD = SEL + ' | json | __error__="" | level =~ "WARN|ERROR"'
TEXT = ' | line_format "{{.level}} {{.target}}: {{.message}}"'
TRACES = '{resource.service.name="ipx" && resource.deployment.environment.name="production"'
# Every production span: ipx-web's, ipx's and the directory's, one trace from the page to the
# directory where a request goes that far (#213). A daemon run by hand has no such tag.
TRACES = '{resource.deployment.environment.name="production"'
# The directory's JSON log (#204): a line a request (method, path, status, duration_ms, token), a
# line a snapshot (instance, accepted, refused_*), a line a refresher round (checked) and one a
# failed feed check (feed, error).
@@ -294,11 +296,12 @@ table("Snapshots by server", [loki(f'sum by (instance) (count_over_time({DIR} |
12, rename={"instance": "Token", "Value": "Snapshots"}, sort=[{"displayName": "Snapshots", "desc": True}])
y += 8
# ---- Traces: ipx's, to Tempo; ipx-web and the directory send none
row("ipx: traces")
for x, title, q in [(0, "Recent traces", TRACES + "}"),
(12, "Slow traces (over 2s)", TRACES + " && duration > 2s}")]:
panel("table", title, 12, 10, x,
# ---- Traces: all three send them to Tempo (#213)
row("Traces")
for x, title, q in [(0, "Recent", TRACES + "}"),
(8, "Slow (over 2s)", TRACES + " && duration > 2s}"),
(16, "Failed", TRACES + " && status = error}")]:
panel("table", title, 8, 10, x,
[{"refId": "A", "datasource": TEMPO, "queryType": "traceql", "query": q, "limit": 50,
"tableType": "traces"}],
fieldConfig={"defaults": {}, "overrides": []})