ba18b44 had the JSON layer find a line's span context through the
dispatch it was registered with, and production's logging wrapped that
layer in an Option, whose Layer impl does not pass on_register_dispatch
on: the layer never got its dispatch and every line went out without
trace_id and span_id. The log test built a subscriber of its own,
without the Option, and passed.
Both services now box their stderr layer instead of an Option of each
kind (a Box passes registration on), the formatter falls back to the
default dispatcher when it was never registered, which a global
subscriber's lines find, and the log test builds the subscriber with
the function init uses, writing to its buffer, so a wrapper like this
one is under test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqkrVDqdiQEMWAkNGQ2tYA
WithTrace found a line's span with Span::current(), which asks the
thread's default dispatcher; inside an event under a scoped subscriber
(with_default), tracing's re-entrancy guard answers with none, so every
line went out without trace_id and span_id. Production, on the global
dispatcher, was not affected, but the log test had to go global for it.
The JSON layer now carries a small layer of its own that keeps the
subscriber it is registered with (on_register_dispatch, a weak handle),
and the formatter takes the registry's current span from its own
context and asks tracing-opentelemetry for its context through that
handle (get_otel_context), with no dispatcher lookup. The log test runs
scoped again; with the old formatter it fails for want of the ids.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqkrVDqdiQEMWAkNGQ2tYA
Three tests for what the dashboard and the alert rules read and nothing
checked: every ipx_ metric named in grafana/dashboard.py and alerts.py
(read out of them at compile time) is served by render, with the ones
written only once known (counts, the last scan, a job under way) set
first; ipx's JSON log lines for a scan's events carry ev, feed, new,
bytes, msg and error.type, and inside a span trace_id and span_id; the
directory's snapshot line carries instance, accepted and the refused_
counts.
The JSON layer was built the same way in both services' setup; it is
otel::json_layer now, so the tests run the layer production does.
The log tests set the global subscriber, as production does: under a
scoped one WithTrace finds no current span (#408).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.
Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
word, from SeaORM's metric callback; and the wait for one, for
hand-written SQL in db.rs, measured as the call less the statement
(the callback runs in the poll that returns the result, so the time
read after the await is that statement's) (#367). SeaORM's own queries
are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
event passes once: feed failures by error.type, new items, downloads
saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
route on the response, or each feed's page was a row) and status;
feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
calls deprecated, because the browser tests' Caddy is 2.6 (#371).
Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
(#372). Doing so showed sync_group reading every account and its
subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
span id in each request's traceparent, where the parent was random and
pointed at nothing, and reports the tap (ids, start, its last answer)
when the next begins or the page is hidden. opentelemetry 0.33 gives a
span builder no way to take ids, so the shared tracer provider has an
id generator that takes given ids inside with_ids, on one thread.
Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.
The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The style guide imports types and uses them unqualified. 222 places
wrote a path such as std::collections::HashMap or std::time::Duration
inline; each file now imports the type once, one use a module, and
names it bare. Done with a script, file by file, only where the name is
unambiguous there:
- one path for the name throughout the file, and not a name the file
already binds otherwise (its own types, other imports, a function's
local use);
- not a name generic enough to mislead bare (Error, Result, Value,
Mutex, Client, Request, Response, Command, Event, Context, Map, Ok);
- only code a normal build compiles, so no import is unused outside
tests or names a crate only tests depend on.
193 paths stay, those cases above. No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured against /src/styleguide/rustguide.md:
- Each use statement imports from one module, and free functions are
called through theirs (db::step, db::now, jsonwebtoken::decode,
crate::feed::redact, unix::signal) rather than imported bare.
- unsafe_code is forbidden workspace-wide. The one unsafe was a test's
set_var, which is unsound while other tests' threads read the
environment; the test now reads PATH, which every run has.
- Locks say why they may panic: expect("a panic elsewhere poisoned this
lock") in place of 25 unwraps, and unique_slug's endless search says
it cannot end.
- No combinator chains in for headers: each is a named variable first.
- #[must_use] on the 27 functions clippy's must_use_candidate names.
- A literal bool passed where its meaning is not plain gets its name
(/* keep_open */ true, /* piece */ false, ...).
No behaviour changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The code was formatted by hand, and `cargo fmt --check` differed in 866
places. rustfmt.toml keeps the house's long lines (120) and short
expressions on one line, which halves what changes; this commit is only
what rustfmt does with it, and is named in .git-blame-ignore-revs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exporter setup and the traceparent parser move from ipx into ipx_directory::otel, which ipx
already depends on, so both binaries share them. The directory makes a span a request, a child of
the caller's traceparent, and a trace a refresher round with a span a feed; ipx sends the current
span's traceparent on every call to the directory and wraps each share in a span. The dashboard's
Traces row and the prod-check skill cover all three services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
