node tests/load/run.js --postgres puts the scratch daemon in IPX_TEST_DATABASE_URL, the database
the Rust tests already use on Postgres, ipodderx_test on the databases project's server, in a
schema of its own, ipx_load, dropped and made again each run as /tmp/ipx-load is wiped. The URL
is in /src/.envrc, taken from ipx.env. It is production's server and role, so the harness refuses
a database whose name does not end in _test, and hands psql the password in its environment,
where no other process on Tower can read it. psql comes from /src/install.sh.
After seeding it runs ANALYZE: a schema filled seconds before has no planner statistics until
autovacuum gets there, which it did partway through a test, and the search's p95 swung between
40ms and 166ms from run to run. Analyzed first, three runs gave 37-38ms.
Each budget now has a value for each database, about twice what it measures on Tower: Postgres
answers the feed list at p95 54ms to SQLite's 242, having no one connection to queue for (#136).
Running it found #140.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Running the load tests on Postgres (#139) found a scan reading 1.2 feeds a second: after ten
minutes the first of 1,500 had reached 1,136, where SQLite does them in 27s since #135. On the
databases project's server, production's, 200 single-row inserts took 2,565ms as commits of their
own, 12.8ms each waiting for the WAL to reach the disk (synchronous_commit=on), 56ms with
synchronous_commit=off, and 67ms in one transaction. A feed's entries and enclosures are written
a row at a time, about fifty commits for 20 new items, and production's scans paid the same.
ipx's own sessions now set synchronous_commit=off, through sqlx's connect options, so it holds
on top of whatever options a URL sets, the tests' search_path included. A crash of the Postgres
server can lose the last commits, about the last 0.6s, and never corrupts anything; a crash of ipx
loses nothing; other databases on the server keep their setting. The same first scan takes 20s,
and the Rust tests on Postgres 3.3s instead of 7.1s. The test of SQLite's setting checks this one
too when it runs on Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The browser tests drive one person against a handful of feeds, one request at a time, and cannot
show what production's load does. node tests/load/run.js builds a release binary, serves 1,500
generated feeds from itself, starts a daemon of its own under /tmp/ipx-load, seeds a hundred
listeners with thirty feeds each and four downloaded files, and runs four k6 scripts, with
`ipx status` -- the healthcheck -- run every second and failing the run at its 5s timeout:
browse 25 people at once: feed list, All Subscriptions, a feed, a search, the Directory,
a feed's page in it
listening 100 players saving positions every second and marking items read while scans write;
each reads its own state back, which must be as it left it
media 50 listeners seeking: every range checked byte for byte against the served file
signin a flood of wrong passwords while others browse, and the gap between refusing a
known name and an unknown one
Each budget is about twice what it measures now. Getting here found #135 (SQLite waiting for the
disk after every write, a first scan of 1,500 feeds estimated at 50 minutes, now 27s), #136
(SQLite's single connection, left open), #137 and #138 (fixed in the commit before this). k6 is
installed from its own signed apt repository by /src/install.sh, outside this repository.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The load tests (#133) sent forty clients' wrong passwords to /api/login, 54 attempts a second,
which takes no account. Everyone else's requests took 4s (median 3.96s for /api/feeds, about 20ms
otherwise) and `ipx status`, the healthcheck, 1.3s (#137): each attempt was an Argon2id check,
tens of milliseconds of CPU, run inside the handler on one of the runtime's workers, so a handful
at once held every worker the rest of the server answers on. And a wrong password for an
account's name was refused a median 31ms later than one for a made-up name (#138), since only a
name with a hash was checked: the answer read the same, the time said which names are accounts.
auth::check_password runs the check on the blocking pool, at most half the cores at once, so a
flood waits on itself, and checks a name with no account, or no password, against a fixed decoy
hash, false in the same time. Under the same flood the rest of the site answers at p95 57ms,
`ipx status` at most 90ms, the gap is 0.2ms, and twice as many attempts are answered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checking every feed made all the feeds' icons in the feed list flash while it ran. A check sends
each feed's new row as it reads the feed, and renderFeeds, run once a frame while they come in,
emptied the list and built every row anew, every <img> with it: each icon went blank and was
loaded and drawn again, many times a second through a scan of 150 feeds.
The rows are still rebuilt, but the images already drawn go into the new ones in place of the
fresh copies, matched by their markup, so only an icon that has changed is made anew. A browser
test checks the rows are new and the images in them the same elements, and fails without this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The load tests (#133) found a daemon on SQLite scanning 1.3s a feed of 20 items served from the
same machine: 1,500 feeds would have taken about 50 minutes. On Tower's /tmp, a loop device, one
synchronous 4 KB write takes 52ms, and SQLite was in WAL mode with its default synchronous=FULL,
which waits for the disk on every commit; nearly every statement is a commit of its own, a
feed's entries and enclosures being written a row at a time. Subscribing the first admin to an
imported catalogue of 1,500 feeds took 39s, before the web server listened.
Every SQLite connection now sets synchronous=NORMAL, SQLite's own advice for WAL: it syncs at
checkpoints. A crash of ipx loses nothing; a power cut can lose the last transactions, and never
corrupts the file. The same scan takes 27s and the same subscribing 0.75s. Postgres, which
production runs on, is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed's page in the Directory (#128) had its cover, category and latest items, but not the
feed's own description, which Apple's show page leads with: iPX read every item's description
and threw the channel's away, and the feeds table had nowhere to keep it.
It is parsed now, RSS's <description>, or iTunes' summary when that is empty, or Atom's
subtitle, kept in feeds.description (kept when a later read has none, as title and image are),
and sent, sanitized, with the items from /api/directory/{id}, which is now an object, not a
list. The page shows it as plain text under the header, three lines of it, with More when there
is more. A description that only repeats the title is left out.
feeds.description is the first column added to a table that already exists. create_missing
looks for it with a SELECT and runs the ALTER only when it is missing, since it runs on every
open, the healthcheck's included, and an ALTER's lock is what made that time out before. The
same once-only step forgets every feed's ETag and Last-Modified: a feed is read whole only when
it has changed, so its description would otherwise wait for its next item.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hovering a category tile turned its outer two covers further out over 0.2s, and it looked
choppy. Traced in Chromium, the covers were given layers of their own only once the transition
began, so the browser drew them again on its first frames and again on the way back, 22 raster
tasks for a hover in and out, and the main thread produced 34 frames for 0.4s of motion.
Each cover now has its layer before the pointer arrives (will-change), all three move, rotate
and translate are animated as properties of their own rather than as one transform list, and
the fan takes 0.45s on an ease that settles. The same hover is 5 raster tasks, and the
compositor presents 116 frames to the main thread's 26.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each tile took a faint tint from a hash of its category's name, as an initials tile does, so
neighbours differed. Five of them, Comedy, Government, Kids & Family, Leisure and Music, drew
--good, and green means yours everywhere else on the page, the subscribed tick and a downloaded
file among them: those tiles read as holding something you subscribe to, when the tint meant
nothing. The fanned covers already give each tile its colour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Routes that take a feed or an enclosure id did not check who was asking. Anyone signed in could
read any feed's items through GET /api/feeds/{id}/entries, a paid feed's included, with the
addresses of its files, which can carry the subscriber's key: the Directory leaves such feeds
out for that reason, and this route handed them back to whoever guessed the id, a slug of the
title. In production it answered 25 items of a feed the asking account does not subscribe to.
/media/{id} served any downloaded file by its sequential id, POST /api/enclosures/{id}/download
and /api/feeds/{id}/download-latest queued any feed's downloads, and DELETE
/api/enclosures/{id}?force=true deleted any file.
Each now answers 404, "you do not subscribe to that feed", unless the person subscribes to it.
A feed inside an OPML has a subscription row of its own for everyone subscribed to the OPML, so
that holds for those feeds too. Found while adding the Directory's feed page (#128), which has
its own route that answers only for listed feeds and carries no files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Directory was one grid of every listed feed, 1,472 in production, under Podcasts and Blogs
tabs, a wall of about 30 category buttons and a sort menu, with a picked category's
subcategories appearing as a second row of buttons that looked like the first. It read as a
list to scroll, and the two rows were hard to tell apart.
It opens now on the ten most subscribed feeds, ranked, and the categories as tiles, each with
three of its shows' covers fanned in its corner on the tint its initials would get, so the
shows colour it and no theme's palette changes. A category has a page of its own: its most
subscribed, its subcategories with how many each holds, and its grid. See all is every feed,
as the grid was. Podcasts or Blogs holds across every page. Popular, the ten most subscribed,
had a place of its own in the feed list; it is the Directory's first section instead, and its
key, g p, is gone. /api/popular stays for scripts.
The search box finds a feed in the Directory by name (#126). It said "Search items…" there and
did nothing, since loadEntries returns early for a place that lists feeds.
A feed you do not subscribe to opens a page of its own (#128): its cover, category (a link to
that category's page), how many here subscribe, a subscribe button and its latest twenty items,
each with its title, linking to the post where it has one, its first lines, date and length.
Before, a click on it did nothing; only its + did. The items come from GET /api/directory/{id},
which answers only for a feed the Directory lists, so a guessed id reaches nothing private, and
carries no file or its address. The feed's own description would belong at the top, but the
database does not keep one.
Every tile and chart row takes the keyboard, as the feed list's rows do. Feed text made plain
keeps a space at a line break or a paragraph's end, which glued "2010)Recorded" together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Directory listed Football (6 feeds), Soccer (7) and "Soccer," (2) as categories of their
own beside Sports, and Software How-To (2) beside Technology. CATEGORIES spelt two of Apple's
Sports subcategories American Football and Football (Soccer), names no feed uses: Apple's list
says Football and Soccer, so a feed naming either fell through as a category of its own.
They are Apple's spelling now. The old spellings are aliases, since Jev picked from the list
while it had them and its answers are stored, and so is Software How-To, Apple's Technology
subcategory before its 2019 list. A comma or full stop at the end of a feed's category, as in
"Soccer,", is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API took a session cookie, a proxy's word or the shared admin token, so a script or an
agent working for one person had to sign in with their password and carry the cookie, or be
given the admin token. Settings now makes named tokens, ipx_ and 256 random bits, sent as
Authorization: Bearer. A token is its owner and no more. Only its SHA-256 is kept, in the
new api_tokens table, with when it was made and last used; it is shown once and revoked from
the same list. An unknown or revoked one gets a 401 rather than falling through to a cookie.
Cloudflare Access still stands in front of the tunnel, so from outside a token needs an
Access service token beside it; docs/sso.md says how.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Directory always listed A to Z, as the server sends it. A menu beside its filters now
sorts it A to Z, Z to A or by most subscribers (then by name), in the page, without asking
the server again, and the choice stays while the pane is redrawn, as the filters do. The
blurb no longer says A to Z, since it may not be.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pinned feed has a disc in the theme's accent, its pin in the ink the theme pairs with it;
a pinned item only turned its pin solid in the text colour. The item's pin button now draws
the same disc, as a background inside its 22px button so the row does not move when an
item is pinned. The colours are the ones .fpin already uses; no palette changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Directory had one row of chips holding whatever each feed's category was, a category
(Technology) or a subcategory (Tech News, Video Games) side by side: a podcast's own
<itunes:category> is stored as its subcategory, and Jev's answers (#117) are often
subcategories too, so after the first forced scan the row held 22 chips. /api/directory now
gives each feed's Apple category and subcategory, worked out from Apple's list, and the
page shows the categories, then a picked one's subcategories on a line of their own. A
category that is not Apple's, one an admin typed, stands as a category of its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most blogs and news sites name no <itunes:category>, so they sat under no chip in the
Directory unless an admin set one by hand. When TYPESAFE_KEY is set, a feed's first read, or
a forced refresh, asks TypeSafe's Jev to pick one of Apple's categories or subcategories from
the feed's title and up to 15 item titles, for a catalogue feed with neither its own category
nor an admin's. The answer is stored as the catalogue's category, so a feed is not asked
twice, and an admin can change it. Only on a first read or a forced refresh to keep the calls
down: feeds already subscribed are categorised by one `ipx fetch --force`.
Tried on eight subscribed feeds on 2026-10-04: XDA, Daring Fireball, Ars Technica and The
Verge as Tech News, The Old New Thing as Technology, John D. Cook as Mathematics,
BoardGameGeek as Games, Pluralistic as News Commentary (0.55). About 2,000 input tokens a
feed at $0.042 a million. The likeliest answer is kept however unsure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RSS 2.0 makes an item's title optional, and some blogs leave it out on purpose: Scripting News
titles almost none of its posts. Fifty rows of "(untitled)" said nothing about any of them.
entryName gives an item its title, or the opening of its text (HTML read through DOMParser, an
inert document that loads nothing; cut at a word near 120 characters), or its file's name, or
its show and date, with a flag for a name that is not a title. The list sets that one in the
regular weight, as the text it is rather than a heading; the reader leaves out the heading so
the post starts with itself; the player, the lock screen, Currently Listening, the native shell,
Share and the queued toast use the same name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit appended it to a file without a final newline, joining it to
.claude/settings.local.json and un-ignoring both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed moved to its new address was only logged by follow_move. It is now an event, feed_moved
with the feed and its old and new addresses, so it goes where every other event goes: the log,
with from and to as fields, the admin page's Scans view, `ipx fetch`, and the page, which gets
the feed's new row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed whose address answered with a permanent redirect was read through it on every check,
and the catalogue kept the old address: 28 of 147 feeds in production, most http to https, some
to a new path or domain.
Feeds are now fetched with a client of their own that follows no redirects (Ctx::feed_client),
and feed::fetch follows them itself, up to 10 hops, so it sees each one. When every hop was
permanent (301 or 308) it says where the feed ended up, and the scan moves the feed there in the
catalogue (follow_move). A temporary hop (302, 307) anywhere moves nothing. A feed an OPML lists
is left alone, as the OPML would put the old address back, and so is a move onto an address
another feed has. A password goes only to the feed's own host, never to a redirect elsewhere;
reqwest's own following dropped it the same way. Ten hops is a loop, worded as reqwest worded
it so it still reads as redirect_loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon ticked every 60 s and ran a scan pass each time: the sweep, then a check-state query
per feed (about 180) to find which were due. In the six hours before, 293 of 362 passes found
nothing due. Now, after each pass, it works out when the earliest feed is due (due_at, shared
with the scan's own check, over Db::http_states, one query) and sleeps until then: at least
30 s, so a feed that never gets a check time cannot spin it, and at most 10 minutes, so what no
command announces, ipx add or a shorter schedule, is picked up. Commands still wake it at once,
and the first pass after starting runs straight away, as the tick's did. The scan reads every
feed's state in one query too.
The prod-check skill says what to expect now: tens of scans in six hours, and pending as the
real queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every file in state 'pending' counted as waiting to download: 4420 in production, across 12
shows. Since #97 a scan only downloads among a feed's newest max_new_per_check items, so those
were back-catalogue episodes no scan would take; the real queue was 0.
A new state, 'held': listed and downloadable by hand, but outside the feed's newest items, or of
a feed nothing downloads automatically. Db::hold_back moves a feed's waiting files between
'pending' and 'held' each time the feed is due, changed or not, and again after its items are
stored, so a new episode, a raised limit or auto-download turned on or off moves them. A held
file keeps its item's place among the newest, as a downloaded one does. 'pending' now means
queued, so ipx status, /api/status and the dashboard read true without changing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The page loaded artwork from each publisher's server, or through /api/art, fetched every time,
for http-only hosts. Nothing was kept, every visit asked every publisher, and artwork went when
a publisher's server did.
- The page draws every image a feed or item names from /api/art. The first time, iPX fetches
it (only an address a feed or item names, only an image, up to 5 MB, within the feed timeout)
and keeps it in art/ beside the database, under a hash of its address with its type beside
it (src/art.rs). Later it comes from disk, which marks it as used.
- art_cache_mb, a server setting on the admin page, 500 by default, caps what is kept: the
sweep before each scan drops the least recently shown until it fits. 0 keeps nothing, and
artwork is fetched through iPX each time. Settings saved before it get the default.
- Served from iPX's own address, someone else's image must stay an image: nosniff, and a CSP
with sandbox, so an SVG opened on its own runs no script as iPX.
- The fixture server sends .jpg as image/jpeg, which /api/art requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first pass only asked the hosts the feed's current body names. IGN's feed kept five 2009
items with artwork on assets1/assets2.ignimgs.com, which its feed no longer mentions, so they
stayed on http. A scan now also asks, once per host, the hosts of the feed's stored http
artwork (Db::http_images).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#108: the HTTP client had no timeout, and scans handle feeds in order, so a hung server held
every scan. Dreamwidth answered 504 after 60-67 s for a day and each scan took 70-80 s instead of
15. A feed fetch, and a Patreon creator's show list, now gets 30 s from connecting to the last
byte (feed::FEED_TIMEOUT); the client gets a 10 s connect timeout, which bounds a download's
start but not a long download.
#109: iOS asks for /apple-touch-icon-precomposed.png first when the site is added to a home
screen; it was a 404 and the only non-feed warning in the log. It serves the same icon.
#110: the page is https and loads no http. Artwork on http came through /api/art (#90) even
when its host serves https too. A scan now tries each http artwork host on https once per feed
(feed::prefer_https) and stores the https address where the host answers with an image,
rewriting that feed's stored items from the same host (Db::secure_images). 4 of the 5 hosts in
production do; cdn.thesecretcabal.com presents another name's certificate and stays on
/api/art.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pin was a small accent-coloured icon before the feed's name. It is now a disc on the
artwork's corner, where a failing feed's mark is, in the accent and the ink the theme already
pairs with it for primary buttons (checked by tests/contrast.js), so no palette changes. A feed
both pinned and failing keeps the error mark at the bottom corner and the pin at the top.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With no subscribers a feed falls back to its own settings, where auto_download is on, so a
listed feed with audio would have downloaded files for no one. The seeded news feeds carry
only images, which media_types already skips, so nothing was downloaded. Files skipped for it
are judged again, by the new subscriber's settings, on the next scan after someone subscribes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It refused one already there ("already subscribed as ..."), so a feed added before listing
existed, such as CBC's, could not be put in the Directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Directory showed a catalogue feed only once someone subscribed, and a feed left the
catalogue with its last subscriber, so nothing could be put there for others to find.
- A feed has a listed flag, set by ipx add --list (with --category for the Directory's chip).
The web page keeps a listed feed in the catalogue when its last subscriber leaves.
- The Directory lists every catalogue feed; Popular still only what people subscribe to.
popular() reads titles, artwork and categories through Db::feed_list, not three queries a
feed. Subscribing from the Directory scans the feed at once.
- A feed nobody subscribes to is checked once a day at most.
- clean_directory, in the sweep before each scan, removes from the catalogue and the database
a feed nobody subscribes to, with no file on disk and not from an OPML, that has failed for
30 days or published nothing in a year. Run against production first: it removes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every start logged WARN "web ui is reachable off this machine; the token is all that guards
it". A container has to bind 0.0.0.0 for its port to be published, so it fired on every start
of production, and it was out of date: signing in takes an account's password or the admin
token, and through the tunnel Cloudflare Access. It was the only warning in a healthy log. Now
it names what guards it, at info when Access is configured and a warning otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a feed was checked, failed or downloaded a file, and after every item read, the page
fetched /api/feeds whole, about 60 ms for 160 rows, though one row had changed. The live event
stream now knows who is connected and, after an event that changes a feed, sends that person
its row (feed_row), built by the same code as the list (feed_rows, with Db::feed_list asked for
one feed). Marking an item read answers with the feed's row. The page puts the row in place
and redraws once a frame. A routine skip of a feed not due sends nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fetching was 65-90% of a scan, each feed waiting for the one before: 13 s of fetches in a 20 s
refresh of 32 feeds. The scan now works out which feeds are due, fetches their bodies up to six
ahead in tasks of their own, and handles each in order as before, so database writes,
downloads and OPML syncs stay one at a time. A Patreon creator still fetches in scan_one.
The page reloaded /api/feeds, and /api/settings with it, on every scan_done: the scheduler
scans every minute, so each open page reloaded the list once a minute, 169 times an hour. It
now reloads only when the scan checked a feed, and asks for settings once, on first load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding an address looked for the feed a web page links and, finding none, added the address
as it was: every check then failed, and the sidebar called it a feed that had moved. cnn.com
is one; its page links no feed. find_feed replaces feed_behind_page: the address is added if
it is a feed or an OPML list, the feed its page links if it is a web page that links one (and
that is a feed), and otherwise the add is refused with the reason, from the web page (400, the
dialog stays open) and from `ipx add`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cnn-com was added as 'cnn.com', stored as typed, and every check failed with "relative URL
without a base" before it reached the site to look for its feed. expand_input, which both the
web page and `ipx add` pass the address through, now makes one without a scheme https, and a
protocol-relative //host/path https too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unsubscribing leaves a feed's row and history, which suits one that worked. One that never did
stayed with its error for good and was never scanned again: cnn-com, added as a bare 'cnn.com'
(#101), sat there failing with no subscriber. The reaper, before each scan, now deletes a feed
that is failing, has no subscriber, is not in the catalogue and has no file on disk, with its
items, file rows, read state and block list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed that failed was tried again on its usual schedule however long it had been failing:
gizmodo's 404, pelgrane's 403, daily-quests' 503 and toddstashwick's redirect loop every hour,
each a request to a site that had said no, a warning and scan time. A failing feed now waits as
long as it has been failing, from error_since to its last check, never less than its usual
interval and never more than a day: 1h, 1h, 2h, 4h, 8h, 16h, then daily on an hourly schedule.
No new column: error_since already marks the run's start and the first success clears it. A
forced refresh skips the due check, so it still tries at once. The feed list's next check
follows the backoff.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pending() took the newest files still pending, up to the limit, so once a show's latest three
were down, each full read of its feed took the three before them, working back through its
whole history. In production 4420 files (about 310 GB) were queued this way across 12 shows,
all on the default limit of 3, which is meant as "the latest three". It now takes only from the
feed's newest `limit` items with a file. 0, unlimited, still takes the whole back catalogue:
that is how the shows kept as an archive are set, along with limits of 100 and 10000.
The settings' wording followed the old behaviour ("The rest wait for the next scan"); the field
is now "Newest episodes to download", and says what 0 does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With max_new_per_check at 0 and no per-subscription limit, the budget is usize::MAX, and
pending() bound it `as i64`: -1. SQLite reads LIMIT -1 as no limit; Postgres refuses it, so a
feed's downloads failed. The new test fails with "LIMIT must not be negative" on Postgres
without the clamp and passes with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spans added in 2799704 showed it: in a full scan of 134 feeds (trace da9a419b...,
2026-09-29 17:31, 315 s), storing items took 117 s, fetching 44 s and every other database call
about 2 s together. A scan inserted every item and file the feed listed, stored or not, one
round trip of about 10 ms each; Clarkesworld's 1200 items took 13 s. It now reads the feed's
stored guids and file URLs once (Db::stored_items) and inserts only the rest. A file URL not
among the feed's own may still be another feed's, so that one still goes to the insert, which
finds it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed read in full checked its own artwork and, without one, asked its website for an icon,
every time; a feed without validators is read in full every scan, so looking-for-group spent
2 s of every scan loading lfg.co's home page. Now the check runs when the feed names different
artwork from what is stored, or the scan was asked for, which keeps #80's point: a refresh
still picks up an icon the site changes or fixes.
Feed spans ran seconds past their fetch with nothing to say where (#96). The artwork lookup,
the loop that stores each item, and the per-feed database calls (feed_summary, record_feed,
subscribers, adopt, skipped_by_filter, rehide, pending) now have spans of their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/feeds called feed_summary, http_state, blocklist and unread_count for every feed:
about 950 round trips to Postgres for 160 feeds, 320 ms on every page load. Db::feed_list asks
for the feed rows, entry counts, download counts, the person's unread counts and block lists
once each, and the handler reads from that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From Dash0's structured logging guide, what applies here:
- Each JSON line inside a traced span ends with its trace_id and span_id, so a line in Loki leads
to its trace in Tempo; the access log is written inside its request's span so it has one too.
The JSON formatter takes no extra fields, so WithTrace appends them to the object it writes.
- A feed or download failure carries error.type (the HTTP status, or dns, redirect_loop,
timeout, ...) and http.response.status_code, from failure_kind beside explain_failure, so
failures group by kind without a regex over msg.
- Each event was logged twice: words under ipx::scan and fields under ipx::io. It is now one
line under ipx::scan with both; the wire copy is at debug, for the admin page's Daemon I/O tab,
and out of production's log. The healthcheck's status reply stays under ipx::io.
- The access log's ms is duration_ms. The dashboard and the prod-check skill follow.
- error fields are Display with the anyhow chain everywhere, not a mix of Debug and Display.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
