lib.rs held 2,000 lines: fetching, parsing, OPML, Patreon, discovery,
artwork, categories, Jev and redaction, and their tests. It is now the
module list and the re-exports, so ipx and the directory call
feed::fetch, feed::parse and the rest as before:
fetch.rs fetch and its limits parse.rs RSS and Atom
outlines.rs OPML (not opml.rs, which would shadow the opml crate)
patreon.rs a creator's shows discover.rs finding a feed
wordpress.rs WordPress.com's API failure.rs failures in words
privacy.rs redact, looks_private artwork.rs https art, icons
category.rs Apple's categories jev.rs Jev's category
Moved verbatim, private items pub(crate) where a sibling calls them,
calls across modules module::function. Every definition and doc comment
line of the old lib.rs is in exactly one of the new files; each test
went with what it tests, and all 40 run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The style guide imports types and uses them unqualified. 222 places
wrote a path such as std::collections::HashMap or std::time::Duration
inline; each file now imports the type once, one use a module, and
names it bare. Done with a script, file by file, only where the name is
unambiguous there:
- one path for the name throughout the file, and not a name the file
already binds otherwise (its own types, other imports, a function's
local use);
- not a name generic enough to mislead bare (Error, Result, Value,
Mutex, Client, Request, Response, Command, Event, Context, Map, Ok);
- only code a normal build compiles, so no import is unused outside
tests or names a crate only tests depend on.
193 paths stay, those cases above. No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The style guide asks for doc comments on a library's public API. Every
type, function, constant and static the two libraries export now has
one, saying what it is for or what it does that its name does not:
fetch follows redirects itself so credentials stay on the feed's host,
Config::parse refuses repeated token names, Store::due_one is the
admin's Refresh. Fields and variants whose names say it all have none,
as the guide allows.
category_path's comment had slid above categories(), which then had
two; each has its own again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured against /src/styleguide/rustguide.md:
- Each use statement imports from one module, and free functions are
called through theirs (db::step, db::now, jsonwebtoken::decode,
crate::feed::redact, unix::signal) rather than imported bare.
- unsafe_code is forbidden workspace-wide. The one unsafe was a test's
set_var, which is unsound while other tests' threads read the
environment; the test now reads PATH, which every run has.
- Locks say why they may panic: expect("a panic elsewhere poisoned this
lock") in place of 25 unwraps, and unique_slug's endless search says
it cannot end.
- No combinator chains in for headers: each is a named variable first.
- #[must_use] on the 27 functions clippy's must_use_candidate names.
- A literal bool passed where its meaning is not plain gets its name
(/* keep_open */ true, /* piece */ false, ...).
No behaviour changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both run on the daemon's single worker, so every other feed's check
and download waited behind them.
#339: prefer_https asked each new artwork host whether it served the
image on https, up to 20 s each, one after another, and the per-scan
cache is keyed by the authority, which ports and user names multiply
for one server. It now asks at most eight hosts a read; past that an
http image stays http, for /api/art to fetch, as one that fails does.
#337: a list (OPML or Patreon) may bring in 2,000 feeds; past that
sync_group refuses it whole, so the feed shows an error and nothing
changes. davewiner's, the largest seen, had 922. sync_group also read
the whole feeds table and rebuilt an id map with a Feed cloned per
entry for every new feed; it now reads the table once and checks ids
against sets, and unique_slug takes a predicate rather than a map.
same_feed skips parsing two URLs unless both are Patreon's, since only
those can differ and match.
sync_group and sync_opml lose the parent argument, which nothing used
once promote_kept stopped taking it (#332).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx's clients had no destination policy and followed redirects
anywhere. Adding a feed returned the status or first line of whatever
answered at the address; an enclosure declared as audio was downloaded
whole from any URL and served back through /media; /api/art fetched
any image a feed named. Feeds come from publishers and anyone signed
in, so loopback, the Docker network and the LAN were all reachable.
ipx-feed's guard is a connector layer: after the name resolves and the
connection opens, before the request is written, it looks at the
peer's address and refuses anything off the public internet unless
config.toml's [general] fetch_internal lists it. Being on the
connection, it covers every request on ctx.client and feed_client,
each redirect hop, IP literals (which never reach a DNS resolver) and
a name that resolves differently the second time. reqwest's rustls
connection reports the peer address; a test run by hand checks HTTPS
gets through, since were it lost every https fetch would be refused.
The directory is on the server's own network by design
(http://ipx-directory:8100), so it moves to local_client, unguarded,
set by an admin. The browser suite and load tests serve their feeds on
loopback and list 127.0.0.1.
Not covered: librqbit's own connections (trackers, peers), which do
not go through these clients.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The opml crate's Outline reader (hard-xml's derive) recurses once per
<outline> level with no limit. A 2 MB upload holds tens of thousands of
levels, enough to overflow a tokio worker's 2 MiB stack, and a stack
overflow aborts the process, not the task. A fetched OPML (up to 50 MB)
reached the same parse on every scan, so a publisher could keep the
daemon in a crash loop.
feed::read_opml counts nesting with quick-xml's streaming reader, which
does not recurse, and refuses past 100 before handing the text to the
opml crate. Every place that parsed OPML goes through it: the upload,
`ipx opml`, a fetched OPML's feeds and its title. The limit also bounds
collect_outlines' recursion and the drop of the nested Vec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pelgranepress.com is on WordPress.com, whose CDN answers /feed/, /feed/atom/ and ?feed=rss2
with HTTP 403 and "Checking your browser...", a JavaScript challenge, whatever the User-Agent.
The checks that hit it failed for hours (8 in the day before this), and the publisher has
not changed anything.
The API at public-api.wordpress.com/wp/v2/sites/<host>/posts has the same posts with no check,
and each post's guid is the feed's own (https://host/?p=<id>), so the items read from it are
the ones already stored. fetch recognises the challenge by its /__challenge/ script, asks the
API instead, and parse reads WordPress's posts JSON as a feed. It sends no title or image for
the feed, which record_feed keeps as they were.
shortcut: the API's default ten posts, as many as WordPress puts in a feed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
resp.bytes() read any size whole: an episode's MP3 given as a feed was
held in memory at every check. Feeds here are a few MB at most.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed that sends <source:markdown> beside its HTML (Scripting News) has its
Markdown kept in entries.markdown, added to existing databases by an ALTER.
A stored item gets it when a scan finds it told differently, the Markdown
counting as a change, the way corrections arrive (#141). The API sends it as
written and made HTML, cleaned by ammonia like the description.
The item's page has a choice between its meta line and its buttons: HTML,
Markdown or Markdown source, remembered in the browser.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A description with Markdown and no HTML tag is rendered to HTML with
pulldown-cmark, and an item with no text but RSS's <source:markdown>
(https://source.scripting.com/, as Scripting News sends beside its HTML) uses
that. HTML is left alone, and plain text without Markdown in it stays as it
was, so a line like "1984. ..." is not made a list. ammonia cleans the result
as any feed's HTML. Stored items keep what they have: the scanner does not
rewrite an item it has.
Reference: http://scripting.com/2026/10/06/005559.html
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cargo clippy --workspace --all-targets had 24 warnings; its own fixes took
most, the rest by hand: two type aliases for long types, a struct built
whole, a checked division, a doc comment's stray blank line. Its fix for
the stale-feed fold was wrong: `any` stops at the first feed it removes,
where the fold removed every one, so that is a loop instead.
A pre-commit hook in .githooks refuses unformatted Rust; clippy, being a
build, is in CLAUDE.md's checks with -D warnings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The code was formatted by hand, and `cargo fmt --check` differed in 866
places. rustfmt.toml keeps the house's long lines (120) and short
expressions on one line, which halves what changes; this commit is only
what rustfmt does with it, and is named in .git-blame-ignore-revs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Signal is a name and keywords. A task beside DeadAir's asks TypeSafe's Jev, on the owner's own
key, about each item of their subscribed feeds from the last 30 days and every new one: one call
an item, one noul question for each of their Signals still to hear about it, 0.5 or more a fit.
Every verdict is kept, fitting or not, in signal_verdicts, so nothing is asked twice, and new
keywords drop a Signal's verdicts to be judged afresh. A key TypeSafe refuses (401/402/403) is
passed over until its owner saves another, so a bad key is not sent the whole queue.
A Signal's items come through entries_where with an EXISTS on its hits, so filters, search, sort
and read state are an ordinary list's, and reading an item in a Signal reads it in its feed.
The page shows Signals under the places, with New signal, and Settings takes the TypeSafe key,
which is never sent back. The browser suite stubs Jev in the fixture server (IPX_TYPESAFE_URL).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With apple_country set, the directory fetches Apple's top 100 podcasts and the top 50 of each of
its 19 categories for that country (itunes.apple.com/{cc}/rss/toppodcasts), looks up each
show's RSS address (/lookup, 100 ids a call, three seconds apart, as Apple asks for no more than
about twenty a minute), and stores them as the snapshot apple-podcasts with no subscribers. The
refresher fetches them like any feed; the most-subscribed chart is unchanged. Apple's primary
genre goes in as the snapshot's category, used only while a feed names none.
The same weekly run reads Apple's genre tree (ws/genres?id=26) and warns when it differs from
ipx_feed's CATEGORIES; on 2026-10-15 they were the same, all 19 and their subcategories.
A syncs table keeps when it last ran: at start if a week has passed, again weekly, an hour after
a failure. keep_instances spares apple-podcasts, which no token holds, and no token may take
the name. A trace a sync, a span each answer.
CLAUDE.md: builds one cargo command at a time, the load looked at before and after, after a
build chained with a test compile took Tower past 200.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fetch that failed logged reqwest's words, which name the whole address: Patreon's auth key
went to Loki on 2026-10-15, and, since scan events go to every open page, to everyone's
browser too. #225 had taken queries out of the traces' address fields only.
Every event is redacted where it is emitted, for the log, the live stream and a terminal
alike: address fields lose their query and password, and so does any address in an error's
words (feed::redact_text). The artwork proxy's warnings and the directory's failed checks
the same. The error kept on the feed, shown to its own subscribers, keeps the detail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the slow traces of 2026-10-13: the directory checked a round's 25 feeds one after
another, so two libsyn feeds timing out at 30 s made a round 70 s; a forced scan looked up
62 site icons in turn, 22 s of 62; mjg59.dreamwidth.org takes the connection and never
answers, so it ran the full 30 s in every scan it was in; and since #225 strips queries,
every YouTube fetch span read the same address, outside any span naming its feed.
The directory now checks five at a time (SQLite stays on one connection, so its writes
queue). A forced scan looks up the icon in the fetch-ahead task, which already runs six
deep, and hands it to scan_one with the body. Both feed clients give up after 15 s of
silence (no successful fetch took over 8 s in a day of traces); FEED_TIMEOUT stays 30 s
for a feed still sending. The directory's client had no connect timeout; it has ipx's 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fetch span and the download span recorded the whole address, so a private feed's
credential (Patreon's ?auth=) went to Tempo with every scan, readable by anyone with
Grafana. feed::redact drops the query, fragment and user:password; a token in the path
would still go, noted where it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five
times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's
allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident
memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its
upstream was archived in 2025.
Declared once in the directory crate, which ipx builds in, so both binaries get it.
What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart
from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size.
The feed crate's spans never reached Tempo either: both trace exporters filtered to their own
crate, so ipx_feed's fetch and parse spans were dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx asks ipx-directory on its people's behalf, with the token an admin set
in the new Admin, Directory tab, so the page never talks to it and never
sees the token or a feed's address. /api/directory, /api/popular and a
feed's Directory page answer from it; subscribing takes the address from
the directory and goes through Add feed's steps. With no directory set
up, the Directory says so. The local directory code is gone; its privacy
checks are the snapshot builder's now, and their tests moved with them.
With sharing on, ipx sends its snapshot after every scan, whenever the
config changes (set_cfg) and every half hour: each public feed and how
many here subscribe, never an OPML or Patreon creator itself or anything
private. A feed can be marked private in its settings, for a paid feed
whose address does not give it away; anyone subscribed may mark it, only
an admin unmark it. The admin's category for a feed that names none goes
with it, which the directory uses only while the feed names none, so the
category an admin gives a blog still reaches the Directory.
The directory lists only feeds it has read as a feed: the first snapshot
after a start went before the first scan, when an OPML was not yet known
to be one, and the directory listed the OPML. It now fetches what a
snapshot brings in at once instead of within the minute.
The browser suite runs its own ipx-directory on SQLite; the Directory's
tests match feeds by title, since the directory names them by address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first step of the split (#207). The daemon moves to crates/ipx unchanged;
feed.rs becomes crates/feed (ipx-feed), which the directory service (#204)
will share for its metadata refresher and its privacy check.
To stand alone, fetch takes the feed's address and its credentials instead
of ipx's config::Feed (config::Feed::auth gives them), collect_outlines
moves in with the OPML parsing, and the address half of looks_private is
feed::url_looks_private. ipx imports the crate as `feed` at its root, so
every crate::feed path is unchanged. The version is the workspace's, in the
root Cargo.toml.
Checked: cargo test on Postgres (118 + 29), tsc, page smoke, native bridge,
contrast and Playwright (72) pass; the image built from the new Dockerfile
is deployed and healthy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
