The zone is in Cloudflare and the rays-unraid tunnel routes the name to ipx-directory, as it
does ipx-dir.sdf1.net, which keeps answering until every server using it has moved: a redirect
to another host would drop the Authorization header. Production ipx shares with the new name.
CLAUDE.md, docs/directory.md, the prod-check skill and the dashboard name it; the admin page's
placeholder and the settings' hint use directory.example.net.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exporter setup and the traceparent parser move from ipx into ipx_directory::otel, which ipx
already depends on, so both binaries share them. The directory makes a span a request, a child of
the caller's traceparent, and a trace a refresher round with a span a feed; ipx sends the current
span's traceparent on every call to the directory and wraps each share in a span. The dashboard's
Traces row and the prod-check skill cover all three services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx-web logs every request now (#210) and the directory a line a refresher round, so each gets
its own queries; Tower's node metrics and the monitoring's own health get sections after
2026-10-12, when load passed 200 and Prometheus did not come back after Docker restarted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new crate and service, crates/directory, live at ipx-dir.sdf1.net. ipx
servers send the public feeds their people subscribe to (POST /v1/snapshot,
replacing what that token sent before); the directory fetches each feed's
title, artwork, category, description and latest twenty items every six
hours with ipx-feed, and lists them A to Z and by subscribers summed over
every server. It holds metadata only, never a file or a file's address.
Tokens are in its config file as SHA-256 hashes, each ro, rw or rwa, and
the file is read again within a minute of a change, so adding and revoking
one is an edit and nothing else; a revoked token takes its server's counts.
At first they were to come from environment variables, which would have
meant recreating the container to add one. A file that does not load keeps
the tokens as they were.
Every address sent is checked with ipx_feed::url_looks_private, the check
ipx makes before sending, so one server's miss does not publish a paid feed
to every other; an rwa token can remove and block what both miss.
Deployed: ipx-directory in the ipx Arcane project, database ipx_directory
(its own login) on the databases project's Postgres, which db-backup
already covers, and a tunnel route with no Access in front. The store's
tests run on Postgres as well as SQLite when IPX_TEST_DATABASE_URL is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Database connections panel drew idle connections as their own line, which
sits on the max for ten minutes after any burst and read as the pool maxing out,
while at most two were ever in use. It now draws in use against the max, with
open dashed behind them. The pool itself goes from sqlx's ten to twenty: the
server allows 100 and had about 13 open across every app.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kept bytes against the limit was a flat line once the cache filled. Now a
percentage of the limit beside the share served from disk, and the images each
sweep evicts. The counter shipped minutes ago as ipx_art_trimmed_total is
renamed ipx_art_evicted_total before anything came to read it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep trims the cache to its limit, so the kept bytes it reports are the
limit every time once the cache has filled: the panel was two lines flat at
500 MiB. A counter of images thrown out says whether the limit is too small.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seventh check groups the browser's errors and lists play starts and
stalls (ipx::client, #190); the trace check names the steps of an item
list, the feed list, a reap and DeadAir's per-file trace (#187), and says
a trace with no name began in the page (#192), which query.py now labels;
the events list has deadair_done and DeadAir's own lines; and "since the
last deploy" is worked out from the container's start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The healthcheck answers while the worker is stuck on one job, and nothing
showed which job or for how long. STATS, one static of atomics, is kept as
the work happens: the worker's job and since when (a guard, so a failed or
stopped job is counted too), jobs and their seconds by kind, when a scan
last finished, web requests, feed fetches and downloads in progress, bytes
downloaded, live updates dropped for a listener that fell behind, DeadAir's
backlog and how long it has been on its file, and the artwork kept. With
the event queue, the DNS retries and the healthcheck's last counts, /metrics
still reads memory only. The dashboard's Daemon row and the prod-check
skill take them up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
query.py prom asks Prometheus as it asks Loki, and a sixth check looks for
a pool at its max, a worker or runtime queue that only grows, memory, open
files or tasks that only climb, and restarts by the uptime's resets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing the page did in the browser reached the logs: a DeadAir playback
failure on the phone was a toast and nothing more. POST /api/client takes
page errors, unhandled rejections, hls.js and media errors, the time from
play to sound and the length of a stall, and logs them as ev lines under
ipx::client, inside the request's span. Only those kinds are taken, the
text cut to 500 characters, and the page sends each at most once a minute.
The page's API requests send traceparent, a new trace id for each tap or
key, and the server's request span takes it as its parent, so one action's
requests are one trace in Tempo, the report included. The stub DOM the
smoke and native-bridge tests run in gains crypto and performance, which
every browser the page runs in has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed moved to its new address was only logged by follow_move. It is now an event, feed_moved
with the feed and its old and new addresses, so it goes where every other event goes: the log,
with from and to as fields, the admin page's Scans view, `ipx fetch`, and the page, which gets
the feed's new row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon ticked every 60 s and ran a scan pass each time: the sweep, then a check-state query
per feed (about 180) to find which were due. In the six hours before, 293 of 362 passes found
nothing due. Now, after each pass, it works out when the earliest feed is due (due_at, shared
with the scan's own check, over Db::http_states, one query) and sleeps until then: at least
30 s, so a feed that never gets a check time cannot spin it, and at most 10 minutes, so what no
command announces, ipx add or a shorter schedule, is picked up. Commands still wake it at once,
and the first pass after starting runs straight away, as the tick's did. The scan reads every
feed's state in one query too.
The prod-check skill says what to expect now: tens of scans in six hours, and pending as the
real queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From Dash0's structured logging guide, what applies here:
- Each JSON line inside a traced span ends with its trace_id and span_id, so a line in Loki leads
to its trace in Tempo; the access log is written inside its request's span so it has one too.
The JSON formatter takes no extra fields, so WithTrace appends them to the object it writes.
- A feed or download failure carries error.type (the HTTP status, or dns, redirect_loop,
timeout, ...) and http.response.status_code, from failure_kind beside explain_failure, so
failures group by kind without a regex over msg.
- Each event was logged twice: words under ipx::scan and fields under ipx::io. It is now one
line under ipx::scan with both; the wire copy is at debug, for the admin page's Daemon I/O tab,
and out of production's log. The healthcheck's status reply stays under ipx::io.
- The access log's ms is duration_ms. The dashboard and the prod-check skill follow.
- error fields are Display with the anyhow chain everywhere, not a mix of Debug and Display.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.claude/skills/ipx-prod-check: what production's JSON log and traces
carry, the queries that find trouble (warnings grouped, failing feeds and
downloads, 5xx and slow routes, whether the worker keeps up, slow and
failed traces), how to tell a publisher's dead feed from an ipx bug, and
filing what is found as issues per CLAUDE.md. query.py beside it runs the
LogQL and TraceQL through a throwaway container on the monitoring
network, since Loki and Tempo publish no query port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
