Commit Graph

266 Commits

Author SHA1 Message Date
552d20443e Release 0.10.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v0.10.0
2026-10-05 17:09:51 +00:00
367f431e1f Sync SQLite at checkpoints, not after every write (#135)
The load tests (#133) found a daemon on SQLite scanning 1.3s a feed of 20 items served from the
same machine: 1,500 feeds would have taken about 50 minutes. On Tower's /tmp, a loop device, one
synchronous 4 KB write takes 52ms, and SQLite was in WAL mode with its default synchronous=FULL,
which waits for the disk on every commit; nearly every statement is a commit of its own, a
feed's entries and enclosures being written a row at a time. Subscribing the first admin to an
imported catalogue of 1,500 feeds took 39s, before the web server listened.

Every SQLite connection now sets synchronous=NORMAL, SQLite's own advice for WAL: it syncs at
checkpoints. A crash of ipx loses nothing; a power cut can lose the last transactions, and never
corrupts the file. The same scan takes 27s and the same subscribing 0.75s. Postgres, which
production runs on, is unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:09:29 +00:00
ef8e2526f2 Show what a feed says it is on its page in the Directory (#130)
A feed's page in the Directory (#128) had its cover, category and latest items, but not the
feed's own description, which Apple's show page leads with: iPX read every item's description
and threw the channel's away, and the feeds table had nowhere to keep it.

It is parsed now, RSS's <description>, or iTunes' summary when that is empty, or Atom's
subtitle, kept in feeds.description (kept when a later read has none, as title and image are),
and sent, sanitized, with the items from /api/directory/{id}, which is now an object, not a
list. The page shows it as plain text under the header, three lines of it, with More when there
is more. A description that only repeats the title is left out.

feeds.description is the first column added to a table that already exists. create_missing
looks for it with a SELECT and runs the ALTER only when it is missing, since it runs on every
open, the healthcheck's included, and an ALTER's lock is what made that time out before. The
same once-only step forgets every feed's ETag and Last-Modified: a feed is read whole only when
it has changed, so its description would otherwise wait for its next item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:46:22 +00:00
5058adc93e Open a Directory category's fan of covers as one smooth motion (#132)
Hovering a category tile turned its outer two covers further out over 0.2s, and it looked
choppy. Traced in Chromium, the covers were given layers of their own only once the transition
began, so the browser drew them again on its first frames and again on the way back, 22 raster
tasks for a hover in and out, and the main thread produced 34 frames for 0.4s of motion.

Each cover now has its layer before the pointer arrives (will-change), all three move, rotate
and translate are animated as properties of their own rather than as one transform list, and
the fan takes 0.45s on an ease that settles. The same hover is 5 raster tasks, and the
compositor presents 116 frames to the main thread's 26.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:46:22 +00:00
9e39f2d756 Draw every Directory category tile on the panel, untinted (#131)
Each tile took a faint tint from a hash of its category's name, as an initials tile does, so
neighbours differed. Five of them, Comedy, Government, Kids & Family, Leisure and Music, drew
--good, and green means yours everywhere else on the page, the subscribed tick and a downloaded
file among them: those tiles read as holding something you subscribe to, when the tint meant
nothing. The fanned covers already give each tile its colour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:33:57 +00:00
b86062b97a Keep a feed's items and files to the people who subscribe to it (#129)
Routes that take a feed or an enclosure id did not check who was asking. Anyone signed in could
read any feed's items through GET /api/feeds/{id}/entries, a paid feed's included, with the
addresses of its files, which can carry the subscriber's key: the Directory leaves such feeds
out for that reason, and this route handed them back to whoever guessed the id, a slug of the
title. In production it answered 25 items of a feed the asking account does not subscribe to.
/media/{id} served any downloaded file by its sequential id, POST /api/enclosures/{id}/download
and /api/feeds/{id}/download-latest queued any feed's downloads, and DELETE
/api/enclosures/{id}?force=true deleted any file.

Each now answers 404, "you do not subscribe to that feed", unless the person subscribes to it.
A feed inside an OPML has a subscription row of its own for everyone subscribed to the OPML, so
that holds for those feeds too. Found while adding the Directory's feed page (#128), which has
its own route that answers only for listed feeds and carries no files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:38 +00:00
95acde3046 Lay the Directory out as Apple's is, with the most subscribed in it (#124, #126, #128)
The Directory was one grid of every listed feed, 1,472 in production, under Podcasts and Blogs
tabs, a wall of about 30 category buttons and a sort menu, with a picked category's
subcategories appearing as a second row of buttons that looked like the first. It read as a
list to scroll, and the two rows were hard to tell apart.

It opens now on the ten most subscribed feeds, ranked, and the categories as tiles, each with
three of its shows' covers fanned in its corner on the tint its initials would get, so the
shows colour it and no theme's palette changes. A category has a page of its own: its most
subscribed, its subcategories with how many each holds, and its grid. See all is every feed,
as the grid was. Podcasts or Blogs holds across every page. Popular, the ten most subscribed,
had a place of its own in the feed list; it is the Directory's first section instead, and its
key, g p, is gone. /api/popular stays for scripts.

The search box finds a feed in the Directory by name (#126). It said "Search items…" there and
did nothing, since loadEntries returns early for a place that lists feeds.

A feed you do not subscribe to opens a page of its own (#128): its cover, category (a link to
that category's page), how many here subscribe, a subscribe button and its latest twenty items,
each with its title, linking to the post where it has one, its first lines, date and length.
Before, a click on it did nothing; only its + did. The items come from GET /api/directory/{id},
which answers only for a feed the Directory lists, so a guessed id reaches nothing private, and
carries no file or its address. The feed's own description would belong at the top, but the
database does not keep one.

Every tile and chart row takes the keyboard, as the feed list's rows do. Feed text made plain
keeps a space at a line break or a paragraph's end, which glued "2010)Recorded" together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:38 +00:00
0b4f5443b7 File Apple's Football and Soccer under Sports in the Directory (#125)
The Directory listed Football (6 feeds), Soccer (7) and "Soccer," (2) as categories of their
own beside Sports, and Software How-To (2) beside Technology. CATEGORIES spelt two of Apple's
Sports subcategories American Football and Football (Soccer), names no feed uses: Apple's list
says Football and Soccer, so a feed naming either fell through as a category of its own.

They are Apple's spelling now. The old spellings are aliases, since Jev picked from the list
while it had them and its answers are stored, and so is Software How-To, Apple's Technology
subcategory before its 2019 list. A comma or full stop at the end of a feed's category, as in
"Soccer,", is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:12 +00:00
e699baa22a Say plainly when a feed is given a category, in the changelog
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:18:18 +00:00
00bd58ac9d API tokens a person makes for scripts and agents to act as them (#123)
The API took a session cookie, a proxy's word or the shared admin token, so a script or an
agent working for one person had to sign in with their password and carry the cookie, or be
given the admin token. Settings now makes named tokens, ipx_ and 256 random bits, sent as
Authorization: Bearer. A token is its owner and no more. Only its SHA-256 is kept, in the
new api_tokens table, with when it was made and last used; it is shown once and revoked from
the same list. An unknown or revoked one gets a 401 rather than falling through to a cookie.

Cloudflare Access still stands in front of the tunnel, so from outside a token needs an
Access service token beside it; docs/sso.md says how.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:15:17 +00:00
cc17b1ddab Sort the Directory by name either way or by subscribers (#122)
The Directory always listed A to Z, as the server sends it. A menu beside its filters now
sorts it A to Z, Z to A or by most subscribers (then by name), in the page, without asking
the server again, and the choice stays while the pane is redrawn, as the filters do. The
blurb no longer says A to Z, since it may not be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:54:32 +00:00
119336a20d Give a pinned item the pinned feed's disc (#121)
A pinned feed has a disc in the theme's accent, its pin in the ink the theme pairs with it;
a pinned item only turned its pin solid in the text colour. The item's pin button now draws
the same disc, as a background inside its 22px button so the row does not move when an
item is pinned. The colours are the ones .fpin already uses; no palette changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:50:48 +00:00
02a46bb500 Browse the Directory by category, then subcategory, as Apple does (#118)
The Directory had one row of chips holding whatever each feed's category was, a category
(Technology) or a subcategory (Tech News, Video Games) side by side: a podcast's own
<itunes:category> is stored as its subcategory, and Jev's answers (#117) are often
subcategories too, so after the first forced scan the row held 22 chips. /api/directory now
gives each feed's Apple category and subcategory, worked out from Apple's list, and the
page shows the categories, then a picked one's subcategories on a line of their own. A
category that is not Apple's, one an admin typed, stands as a category of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:33:26 +00:00
3e384f34f9 Give a feed with no category of its own one of Apple's (#117)
Most blogs and news sites name no <itunes:category>, so they sat under no chip in the
Directory unless an admin set one by hand. When TYPESAFE_KEY is set, a feed's first read, or
a forced refresh, asks TypeSafe's Jev to pick one of Apple's categories or subcategories from
the feed's title and up to 15 item titles, for a catalogue feed with neither its own category
nor an admin's. The answer is stored as the catalogue's category, so a feed is not asked
twice, and an admin can change it. Only on a first read or a forced refresh to keep the calls
down: feeds already subscribed are categorised by one `ipx fetch --force`.

Tried on eight subscribed feeds on 2026-10-04: XDA, Daring Fireball, Ars Technica and The
Verge as Tech News, The Old New Thing as Technology, John D. Cook as Mathematics,
BoardGameGeek as Games, Pluralistic as News Commentary (0.55). About 2,000 input tokens a
feed at $0.042 a million. The likeliest answer is kept however unsure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:20:00 +00:00
d238681ec1 Name an item without a title from its own text, not "(untitled)" (#116)
RSS 2.0 makes an item's title optional, and some blogs leave it out on purpose: Scripting News
titles almost none of its posts. Fifty rows of "(untitled)" said nothing about any of them.

entryName gives an item its title, or the opening of its text (HTML read through DOMParser, an
inert document that loads nothing; cut at a word near 120 characters), or its file's name, or
its show and date, with a flag for a name that is not a title. The list sets that one in the
regular weight, as the text it is rather than a heading; the reader leaves out the heading so
the post starts with itself; the player, the lock screen, Currently Listening, the native shell,
Share and the queued toast use the same name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:46:47 +00:00
0395717c14 Put the bytecode cache's ignore line on a line of its own
The previous commit appended it to a file without a final newline, joining it to
.claude/settings.local.json and un-ignoring both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:27:19 +00:00
8ce68a881a Ignore Python's bytecode cache from the prod-check script
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:27:07 +00:00
e7c59489ee Announce a followed move as a feed_moved event (#115)
A feed moved to its new address was only logged by follow_move. It is now an event, feed_moved
with the feed and its old and new addresses, so it goes where every other event goes: the log,
with from and to as fields, the admin page's Scans view, `ipx fetch`, and the page, which gets
the feed's new row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:22:28 +00:00
f7f4b466dc Follow a feed that has moved for good to its new address (#115)
A feed whose address answered with a permanent redirect was read through it on every check,
and the catalogue kept the old address: 28 of 147 feeds in production, most http to https, some
to a new path or domain.

Feeds are now fetched with a client of their own that follows no redirects (Ctx::feed_client),
and feed::fetch follows them itself, up to 10 hops, so it sees each one. When every hop was
permanent (301 or 308) it says where the feed ended up, and the scan moves the feed there in the
catalogue (follow_move). A temporary hop (302, 307) anywhere moves nothing. A feed an OPML lists
is left alone, as the OPML would put the old address back, and so is a move onto an address
another feed has. A password goes only to the feed's own host, never to a redirect elsewhere;
reqwest's own following dropped it the same way. Ten hops is a loop, worded as reqwest worded
it so it still reads as redirect_loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:05:43 +00:00
ec8fd5dd86 Sleep until the next feed is due instead of scanning every minute (#114)
The daemon ticked every 60 s and ran a scan pass each time: the sweep, then a check-state query
per feed (about 180) to find which were due. In the six hours before, 293 of 362 passes found
nothing due. Now, after each pass, it works out when the earliest feed is due (due_at, shared
with the scan's own check, over Db::http_states, one query) and sleeps until then: at least
30 s, so a feed that never gets a check time cannot spin it, and at most 10 minutes, so what no
command announces, ipx add or a shorter schedule, is picked up. Commands still wake it at once,
and the first pass after starting runs straight away, as the tick's did. The scan reads every
feed's state in one query too.

The prod-check skill says what to expect now: tens of scans in six hours, and pending as the
real queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:56:13 +00:00
a387012c69 Count as queued only what a scan will download on its own (#113)
Every file in state 'pending' counted as waiting to download: 4420 in production, across 12
shows. Since #97 a scan only downloads among a feed's newest max_new_per_check items, so those
were back-catalogue episodes no scan would take; the real queue was 0.

A new state, 'held': listed and downloadable by hand, but outside the feed's newest items, or of
a feed nothing downloads automatically. Db::hold_back moves a feed's waiting files between
'pending' and 'held' each time the feed is due, changed or not, and again after its items are
stored, so a new episode, a raised limit or auto-download turned on or off moves them. A held
file keeps its item's place among the newest, as a downloaded one does. 'pending' now means
queued, so ipx status, /api/status and the dashboard read true without changing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:47:52 +00:00
a00516687a Keep artwork on disk and serve every image from iPX (#111)
The page loaded artwork from each publisher's server, or through /api/art, fetched every time,
for http-only hosts. Nothing was kept, every visit asked every publisher, and artwork went when
a publisher's server did.

- The page draws every image a feed or item names from /api/art. The first time, iPX fetches
  it (only an address a feed or item names, only an image, up to 5 MB, within the feed timeout)
  and keeps it in art/ beside the database, under a hash of its address with its type beside
  it (src/art.rs). Later it comes from disk, which marks it as used.
- art_cache_mb, a server setting on the admin page, 500 by default, caps what is kept: the
  sweep before each scan drops the least recently shown until it fits. 0 keeps nothing, and
  artwork is fetched through iPX each time. Settings saved before it get the default.
- Served from iPX's own address, someone else's image must stay an image: nosniff, and a CSP
  with sandbox, so an SVG opened on its own runs no script as iPX.
- The fixture server sends .jpg as image/jpeg, which /api/art requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 13:50:46 +00:00
38ebc7b02b Move old items' http artwork to https too, for hosts the feed no longer names (#110)
The first pass only asked the hosts the feed's current body names. IGN's feed kept five 2009
items with artwork on assets1/assets2.ignimgs.com, which its feed no longer mentions, so they
stayed on http. A scan now also asks, once per host, the hosts of the feed's stored http
artwork (Db::http_images).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 13:35:51 +00:00
54d827f655 Time out a hung feed, serve the precomposed touch icon, and store http artwork on https (#108, #109, #110)
#108: the HTTP client had no timeout, and scans handle feeds in order, so a hung server held
every scan. Dreamwidth answered 504 after 60-67 s for a day and each scan took 70-80 s instead of
15. A feed fetch, and a Patreon creator's show list, now gets 30 s from connecting to the last
byte (feed::FEED_TIMEOUT); the client gets a 10 s connect timeout, which bounds a download's
start but not a long download.

#109: iOS asks for /apple-touch-icon-precomposed.png first when the site is added to a home
screen; it was a 404 and the only non-feed warning in the log. It serves the same icon.

#110: the page is https and loads no http. Artwork on http came through /api/art (#90) even
when its host serves https too. A scan now tries each http artwork host on https once per feed
(feed::prefer_https) and stores the https address where the host answers with an image,
rewriting that feed's stored items from the same host (Db::secure_images). 4 of the 5 hosts in
production do; cdn.thesecretcabal.com presents another name's certificate and stays on
/api/art.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 13:31:50 +00:00
e2593eaa50 Put a pinned feed's pin on the corner of its artwork
The pin was a small accent-coloured icon before the feed's name. It is now a disc on the
artwork's corner, where a failing feed's mark is, in the accent and the ink the theme already
pairs with it for primary buttons (checked by tests/contrast.js), so no palette changes. A feed
both pinned and failing keeps the error mark at the bottom corner and the pin at the top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:23:40 +00:00
d52ecfcbd7 A listed feed nobody subscribes to downloads nothing (#107)
With no subscribers a feed falls back to its own settings, where auto_download is on, so a
listed feed with audio would have downloaded files for no one. The seeded news feeds carry
only images, which media_types already skips, so nothing was downloaded. Files skipped for it
are judged again, by the new subscriber's settings, on the next scan after someone subscribes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:20:09 +00:00
142610e8b8 ipx add --list or --category updates a feed the catalogue already has (#107)
It refused one already there ("already subscribed as ..."), so a feed added before listing
existed, such as CBC's, could not be put in the Directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:17:49 +00:00
43acc62259 List feeds in the Directory before anyone subscribes, and clear out dead ones (#107)
The Directory showed a catalogue feed only once someone subscribed, and a feed left the
catalogue with its last subscriber, so nothing could be put there for others to find.

- A feed has a listed flag, set by ipx add --list (with --category for the Directory's chip).
  The web page keeps a listed feed in the catalogue when its last subscriber leaves.
- The Directory lists every catalogue feed; Popular still only what people subscribe to.
  popular() reads titles, artwork and categories through Db::feed_list, not three queries a
  feed. Subscribing from the Directory scans the feed at once.
- A feed nobody subscribes to is checked once a day at most.
- clean_directory, in the sweep before each scan, removes from the catalogue and the database
  a feed nobody subscribes to, with no file on disk and not from an OPML, that has failed for
  30 days or published nothing in a year. Run against production first: it removes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:13:13 +00:00
65fdab8b13 Between releases, the version says a release is in progress: 0.9.2-dev
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:52:00 +00:00
4d312a87ec Release 0.9.1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v0.9.1
2026-09-29 20:48:16 +00:00
ef5bdb4244 Say what guards the web UI, and warn only without Cloudflare Access (#106)
Every start logged WARN "web ui is reachable off this machine; the token is all that guards
it". A container has to bind 0.0.0.0 for its port to be published, so it fired on every start
of production, and it was out of date: signing in takes an account's password or the admin
token, and through the tunnel Cloudflare Access. It was the only warning in a healthy log. Now
it names what guards it, at info when Access is configured and a warning otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:46:43 +00:00
c7f13eea2f Send a changed feed's row to the page instead of it reloading the list (#105)
After a feed was checked, failed or downloaded a file, and after every item read, the page
fetched /api/feeds whole, about 60 ms for 160 rows, though one row had changed. The live event
stream now knows who is connected and, after an event that changes a feed, sends that person
its row (feed_row), built by the same code as the list (feed_rows, with Db::feed_list asked for
one feed). Marking an item read answers with the feed's row. The page puts the row in place
and redraws once a frame. A routine skip of a feed not due sends nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:41:43 +00:00
7a134b810f Fetch feeds six ahead in a scan; reload the list only after a scan that checked something (#103, #104)
Fetching was 65-90% of a scan, each feed waiting for the one before: 13 s of fetches in a 20 s
refresh of 32 feeds. The scan now works out which feeds are due, fetches their bodies up to six
ahead in tasks of their own, and handles each in order as before, so database writes,
downloads and OPML syncs stay one at a time. A Patreon creator still fetches in scan_one.

The page reloaded /api/feeds, and /api/settings with it, on every scan_done: the scheduler
scans every minute, so each open page reloaded the list once a minute, 169 times an hour. It
now reloads only when the scan checked a feed, and asks for settings once, on first load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:27:51 +00:00
79bc006a30 Add only what is a feed or links one; refuse the rest (#102)
Adding an address looked for the feed a web page links and, finding none, added the address
as it was: every check then failed, and the sidebar called it a feed that had moved. cnn.com
is one; its page links no feed. find_feed replaces feed_behind_page: the address is added if
it is a feed or an OPML list, the feed its page links if it is a web page that links one (and
that is a feed), and otherwise the add is refused with the reason, from the web page (400, the
dialog stays open) and from `ipx add`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:06:04 +00:00
9084b61bb6 Read an address typed without a scheme as https (#101)
cnn-com was added as 'cnn.com', stored as typed, and every check failed with "relative URL
without a base" before it reached the site to look for its feed. expand_input, which both the
web page and `ipx add` pass the address through, now makes one without a scheme https, and a
protocol-relative //host/path https too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:46:10 +00:00
0cc956b7c6 Forget a failing feed once nobody subscribes to it (#100)
Unsubscribing leaves a feed's row and history, which suits one that worked. One that never did
stayed with its error for good and was never scanned again: cnn-com, added as a bare 'cnn.com'
(#101), sat there failing with no subscriber. The reaper, before each scan, now deletes a feed
that is failing, has no subscriber, is not in the catalogue and has no file on disk, with its
items, file rows, read state and block list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:40:18 +00:00
db87bc0f42 Back off a failing feed exponentially, up to a day (#99)
A feed that failed was tried again on its usual schedule however long it had been failing:
gizmodo's 404, pelgrane's 403, daily-quests' 503 and toddstashwick's redirect loop every hour,
each a request to a site that had said no, a warning and scan time. A failing feed now waits as
long as it has been failing, from error_since to its last check, never less than its usual
interval and never more than a day: 1h, 1h, 2h, 4h, 8h, 16h, then daily on an hourly schedule.
No new column: error_since already marks the run's start and the first success clears it. A
forced refresh skips the due check, so it still tries at once. The feed list's next check
follows the backoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:33:57 +00:00
527777efdf A download limit means a show's newest episodes, not a pace (#97)
pending() took the newest files still pending, up to the limit, so once a show's latest three
were down, each full read of its feed took the three before them, working back through its
whole history. In production 4420 files (about 310 GB) were queued this way across 12 shows,
all on the default limit of 3, which is meant as "the latest three". It now takes only from the
feed's newest `limit` items with a file. 0, unlimited, still takes the whole back catalogue:
that is how the shows kept as an archive are set, along with limits of 100 and 10000.

The settings' wording followed the old behaviour ("The rest wait for the next scan"); the field
is now "Newest episodes to download", and says what 0 does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:50:11 +00:00
4a26c73b82 Clamp an unlimited download queue's LIMIT for Postgres (#98)
With max_new_per_check at 0 and no per-subscription limit, the budget is usize::MAX, and
pending() bound it `as i64`: -1. SQLite reads LIMIT -1 as no limit; Postgres refuses it, so a
feed's downloads failed. The new test fails with "LIMIT must not be negative" on Postgres
without the clamp and passes with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:37:39 +00:00
98ed9b6498 ignore local claude settings 2026-09-29 18:26:33 +00:00
57ab419718 Insert only a feed's new items and files on a scan (#96)
The spans added in 2799704 showed it: in a full scan of 134 feeds (trace da9a419b...,
2026-09-29 17:31, 315 s), storing items took 117 s, fetching 44 s and every other database call
about 2 s together. A scan inserted every item and file the feed listed, stored or not, one
round trip of about 10 ms each; Clarkesworld's 1200 items took 13 s. It now reads the feed's
stored guids and file URLs once (Db::stored_items) and inserts only the rest. A file URL not
among the feed's own may still be another feed's, so that one still goes to the insert, which
finds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:40:52 +00:00
2799704d30 Look at a feed's artwork only when it may have changed, and trace a scan's database work (#95, #96)
A feed read in full checked its own artwork and, without one, asked its website for an icon,
every time; a feed without validators is read in full every scan, so looking-for-group spent
2 s of every scan loading lfg.co's home page. Now the check runs when the feed names different
artwork from what is stored, or the scan was asked for, which keeps #80's point: a refresh
still picks up an icon the site changes or fixes.

Feed spans ran seconds past their fetch with nothing to say where (#96). The artwork lookup,
the loop that stores each item, and the per-feed database calls (feed_summary, record_feed,
subscribers, adopt, skipped_by_filter, rehide, pending) now have spans of their own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:27:31 +00:00
e8fd3fe9ed Load the feed list in five queries, not six per feed (#94)
GET /api/feeds called feed_summary, http_state, blocklist and unread_count for every feed:
about 950 round trips to Postgres for 160 feeds, 320 ms on every page load. Db::feed_list asks
for the feed rows, entry counts, download counts, the person's unread counts and block lists
once each, and the handler reads from that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:24:09 +00:00
448e557272 Trace ids, failure kinds and one line per event in the JSON log (#91)
From Dash0's structured logging guide, what applies here:

- Each JSON line inside a traced span ends with its trace_id and span_id, so a line in Loki leads
  to its trace in Tempo; the access log is written inside its request's span so it has one too.
  The JSON formatter takes no extra fields, so WithTrace appends them to the object it writes.
- A feed or download failure carries error.type (the HTTP status, or dns, redirect_loop,
  timeout, ...) and http.response.status_code, from failure_kind beside explain_failure, so
  failures group by kind without a regex over msg.
- Each event was logged twice: words under ipx::scan and fields under ipx::io. It is now one
  line under ipx::scan with both; the wire copy is at debug, for the admin page's Daemon I/O tab,
  and out of production's log. The healthcheck's status reply stays under ipx::io.
- The access log's ms is duration_ms. The dashboard and the prod-check skill follow.
- error fields are Display with the anyhow chain everywhere, not a mix of Debug and Display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:25:53 +00:00
36b16c7f5d Fetch artwork offered only over http for the https page (#90)
Through ipodderx.sdf1.net the page is https, the browser upgrades an http:// image to https,
and a host with no https, such as The Secret Cabal's CDN, answers nothing, so no artwork. On an
https page, the page now asks /api/art for those, and ipx fetches them. It only fetches an
address some feed or entry names as its artwork, and only an image, up to 5 MB, so the route
cannot be pointed at anything else on the network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:19:06 +00:00
928cbaf8f4 Show each feed's id labelled in ipx list (#82)
The id led the title's line unlabelled, so antirez.com's, "feed" with no title beside it, read
as a heading; 'ipx fetch antirez' was tried instead and failed. The title now heads the entry and
the id has its own row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:16:25 +00:00
0945f3a9f4 Remove ipx copy-db (#85)
It was the one-off copy from SQLite to Postgres (#18), run once on 2026-09-18. Production has run
on Postgres since; rolling back needs only the old state.db, which is kept, not this command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:15:51 +00:00
e9f2832e1e Show a failing feed on its artwork, in words, and grey (#93)
The mark was a 12px "!" in the sidebar's margin, told apart by --bad alone; a dark theme's --bad
is a pale pink, and at that size it vanished. It is now a solid disc on the artwork's corner, the
subtitle says what is wrong in place of the counts, and a feed failing for a day or more has its
artwork greyed out. Lightness and words carry it, so no theme's palette changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 15:17:46 +00:00
d4e00be085 A feed's own artwork has to be there before it is used (#89)
A feed's itunes:image or <image><url> was stored without being asked
for, so a dead one stood in the way of the site's icon. Ken and Robin
Talk About Stuff names http://kenandrobin.wpengine.com/.../kartas_podcast.png,
a 404, while its site's apple-touch-icon works. The feed's artwork now has
to answer as an image, as the site icon already did, when the feed is
read in full; otherwise the site's icon is looked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 14:11:16 +00:00
66e40e3c71 A skill for checking production from Loki and Tempo
.claude/skills/ipx-prod-check: what production's JSON log and traces
carry, the queries that find trouble (warnings grouped, failing feeds and
downloads, 5xx and slow routes, whether the worker keeps up, slow and
failed traces), how to tell a publisher's dead feed from an ipx bug, and
filing what is found as issues per CLAUDE.md. query.py beside it runs the
LogQL and TraceQL through a throwaway container on the monitoring
network, since Loki and Tempo publish no query port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 14:07:24 +00:00