A feed whose address answered with a permanent redirect was read through it on every check,
and the catalogue kept the old address: 28 of 147 feeds in production, most http to https, some
to a new path or domain.
Feeds are now fetched with a client of their own that follows no redirects (Ctx::feed_client),
and feed::fetch follows them itself, up to 10 hops, so it sees each one. When every hop was
permanent (301 or 308) it says where the feed ended up, and the scan moves the feed there in the
catalogue (follow_move). A temporary hop (302, 307) anywhere moves nothing. A feed an OPML lists
is left alone, as the OPML would put the old address back, and so is a move onto an address
another feed has. A password goes only to the feed's own host, never to a redirect elsewhere;
reqwest's own following dropped it the same way. Ten hops is a loop, worded as reqwest worded
it so it still reads as redirect_loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon ticked every 60 s and ran a scan pass each time: the sweep, then a check-state query
per feed (about 180) to find which were due. In the six hours before, 293 of 362 passes found
nothing due. Now, after each pass, it works out when the earliest feed is due (due_at, shared
with the scan's own check, over Db::http_states, one query) and sleeps until then: at least
30 s, so a feed that never gets a check time cannot spin it, and at most 10 minutes, so what no
command announces, ipx add or a shorter schedule, is picked up. Commands still wake it at once,
and the first pass after starting runs straight away, as the tick's did. The scan reads every
feed's state in one query too.
The prod-check skill says what to expect now: tens of scans in six hours, and pending as the
real queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every file in state 'pending' counted as waiting to download: 4420 in production, across 12
shows. Since #97 a scan only downloads among a feed's newest max_new_per_check items, so those
were back-catalogue episodes no scan would take; the real queue was 0.
A new state, 'held': listed and downloadable by hand, but outside the feed's newest items, or of
a feed nothing downloads automatically. Db::hold_back moves a feed's waiting files between
'pending' and 'held' each time the feed is due, changed or not, and again after its items are
stored, so a new episode, a raised limit or auto-download turned on or off moves them. A held
file keeps its item's place among the newest, as a downloaded one does. 'pending' now means
queued, so ipx status, /api/status and the dashboard read true without changing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The page loaded artwork from each publisher's server, or through /api/art, fetched every time,
for http-only hosts. Nothing was kept, every visit asked every publisher, and artwork went when
a publisher's server did.
- The page draws every image a feed or item names from /api/art. The first time, iPX fetches
it (only an address a feed or item names, only an image, up to 5 MB, within the feed timeout)
and keeps it in art/ beside the database, under a hash of its address with its type beside
it (src/art.rs). Later it comes from disk, which marks it as used.
- art_cache_mb, a server setting on the admin page, 500 by default, caps what is kept: the
sweep before each scan drops the least recently shown until it fits. 0 keeps nothing, and
artwork is fetched through iPX each time. Settings saved before it get the default.
- Served from iPX's own address, someone else's image must stay an image: nosniff, and a CSP
with sandbox, so an SVG opened on its own runs no script as iPX.
- The fixture server sends .jpg as image/jpeg, which /api/art requires.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first pass only asked the hosts the feed's current body names. IGN's feed kept five 2009
items with artwork on assets1/assets2.ignimgs.com, which its feed no longer mentions, so they
stayed on http. A scan now also asks, once per host, the hosts of the feed's stored http
artwork (Db::http_images).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#108: the HTTP client had no timeout, and scans handle feeds in order, so a hung server held
every scan. Dreamwidth answered 504 after 60-67 s for a day and each scan took 70-80 s instead of
15. A feed fetch, and a Patreon creator's show list, now gets 30 s from connecting to the last
byte (feed::FEED_TIMEOUT); the client gets a 10 s connect timeout, which bounds a download's
start but not a long download.
#109: iOS asks for /apple-touch-icon-precomposed.png first when the site is added to a home
screen; it was a 404 and the only non-feed warning in the log. It serves the same icon.
#110: the page is https and loads no http. Artwork on http came through /api/art (#90) even
when its host serves https too. A scan now tries each http artwork host on https once per feed
(feed::prefer_https) and stores the https address where the host answers with an image,
rewriting that feed's stored items from the same host (Db::secure_images). 4 of the 5 hosts in
production do; cdn.thesecretcabal.com presents another name's certificate and stays on
/api/art.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pin was a small accent-coloured icon before the feed's name. It is now a disc on the
artwork's corner, where a failing feed's mark is, in the accent and the ink the theme already
pairs with it for primary buttons (checked by tests/contrast.js), so no palette changes. A feed
both pinned and failing keeps the error mark at the bottom corner and the pin at the top.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With no subscribers a feed falls back to its own settings, where auto_download is on, so a
listed feed with audio would have downloaded files for no one. The seeded news feeds carry
only images, which media_types already skips, so nothing was downloaded. Files skipped for it
are judged again, by the new subscriber's settings, on the next scan after someone subscribes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It refused one already there ("already subscribed as ..."), so a feed added before listing
existed, such as CBC's, could not be put in the Directory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Directory showed a catalogue feed only once someone subscribed, and a feed left the
catalogue with its last subscriber, so nothing could be put there for others to find.
- A feed has a listed flag, set by ipx add --list (with --category for the Directory's chip).
The web page keeps a listed feed in the catalogue when its last subscriber leaves.
- The Directory lists every catalogue feed; Popular still only what people subscribe to.
popular() reads titles, artwork and categories through Db::feed_list, not three queries a
feed. Subscribing from the Directory scans the feed at once.
- A feed nobody subscribes to is checked once a day at most.
- clean_directory, in the sweep before each scan, removes from the catalogue and the database
a feed nobody subscribes to, with no file on disk and not from an OPML, that has failed for
30 days or published nothing in a year. Run against production first: it removes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every start logged WARN "web ui is reachable off this machine; the token is all that guards
it". A container has to bind 0.0.0.0 for its port to be published, so it fired on every start
of production, and it was out of date: signing in takes an account's password or the admin
token, and through the tunnel Cloudflare Access. It was the only warning in a healthy log. Now
it names what guards it, at info when Access is configured and a warning otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a feed was checked, failed or downloaded a file, and after every item read, the page
fetched /api/feeds whole, about 60 ms for 160 rows, though one row had changed. The live event
stream now knows who is connected and, after an event that changes a feed, sends that person
its row (feed_row), built by the same code as the list (feed_rows, with Db::feed_list asked for
one feed). Marking an item read answers with the feed's row. The page puts the row in place
and redraws once a frame. A routine skip of a feed not due sends nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fetching was 65-90% of a scan, each feed waiting for the one before: 13 s of fetches in a 20 s
refresh of 32 feeds. The scan now works out which feeds are due, fetches their bodies up to six
ahead in tasks of their own, and handles each in order as before, so database writes,
downloads and OPML syncs stay one at a time. A Patreon creator still fetches in scan_one.
The page reloaded /api/feeds, and /api/settings with it, on every scan_done: the scheduler
scans every minute, so each open page reloaded the list once a minute, 169 times an hour. It
now reloads only when the scan checked a feed, and asks for settings once, on first load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding an address looked for the feed a web page links and, finding none, added the address
as it was: every check then failed, and the sidebar called it a feed that had moved. cnn.com
is one; its page links no feed. find_feed replaces feed_behind_page: the address is added if
it is a feed or an OPML list, the feed its page links if it is a web page that links one (and
that is a feed), and otherwise the add is refused with the reason, from the web page (400, the
dialog stays open) and from `ipx add`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cnn-com was added as 'cnn.com', stored as typed, and every check failed with "relative URL
without a base" before it reached the site to look for its feed. expand_input, which both the
web page and `ipx add` pass the address through, now makes one without a scheme https, and a
protocol-relative //host/path https too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unsubscribing leaves a feed's row and history, which suits one that worked. One that never did
stayed with its error for good and was never scanned again: cnn-com, added as a bare 'cnn.com'
(#101), sat there failing with no subscriber. The reaper, before each scan, now deletes a feed
that is failing, has no subscriber, is not in the catalogue and has no file on disk, with its
items, file rows, read state and block list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed that failed was tried again on its usual schedule however long it had been failing:
gizmodo's 404, pelgrane's 403, daily-quests' 503 and toddstashwick's redirect loop every hour,
each a request to a site that had said no, a warning and scan time. A failing feed now waits as
long as it has been failing, from error_since to its last check, never less than its usual
interval and never more than a day: 1h, 1h, 2h, 4h, 8h, 16h, then daily on an hourly schedule.
No new column: error_since already marks the run's start and the first success clears it. A
forced refresh skips the due check, so it still tries at once. The feed list's next check
follows the backoff.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pending() took the newest files still pending, up to the limit, so once a show's latest three
were down, each full read of its feed took the three before them, working back through its
whole history. In production 4420 files (about 310 GB) were queued this way across 12 shows,
all on the default limit of 3, which is meant as "the latest three". It now takes only from the
feed's newest `limit` items with a file. 0, unlimited, still takes the whole back catalogue:
that is how the shows kept as an archive are set, along with limits of 100 and 10000.
The settings' wording followed the old behaviour ("The rest wait for the next scan"); the field
is now "Newest episodes to download", and says what 0 does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With max_new_per_check at 0 and no per-subscription limit, the budget is usize::MAX, and
pending() bound it `as i64`: -1. SQLite reads LIMIT -1 as no limit; Postgres refuses it, so a
feed's downloads failed. The new test fails with "LIMIT must not be negative" on Postgres
without the clamp and passes with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spans added in 2799704 showed it: in a full scan of 134 feeds (trace da9a419b...,
2026-09-29 17:31, 315 s), storing items took 117 s, fetching 44 s and every other database call
about 2 s together. A scan inserted every item and file the feed listed, stored or not, one
round trip of about 10 ms each; Clarkesworld's 1200 items took 13 s. It now reads the feed's
stored guids and file URLs once (Db::stored_items) and inserts only the rest. A file URL not
among the feed's own may still be another feed's, so that one still goes to the insert, which
finds it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed read in full checked its own artwork and, without one, asked its website for an icon,
every time; a feed without validators is read in full every scan, so looking-for-group spent
2 s of every scan loading lfg.co's home page. Now the check runs when the feed names different
artwork from what is stored, or the scan was asked for, which keeps #80's point: a refresh
still picks up an icon the site changes or fixes.
Feed spans ran seconds past their fetch with nothing to say where (#96). The artwork lookup,
the loop that stores each item, and the per-feed database calls (feed_summary, record_feed,
subscribers, adopt, skipped_by_filter, rehide, pending) now have spans of their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/feeds called feed_summary, http_state, blocklist and unread_count for every feed:
about 950 round trips to Postgres for 160 feeds, 320 ms on every page load. Db::feed_list asks
for the feed rows, entry counts, download counts, the person's unread counts and block lists
once each, and the handler reads from that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From Dash0's structured logging guide, what applies here:
- Each JSON line inside a traced span ends with its trace_id and span_id, so a line in Loki leads
to its trace in Tempo; the access log is written inside its request's span so it has one too.
The JSON formatter takes no extra fields, so WithTrace appends them to the object it writes.
- A feed or download failure carries error.type (the HTTP status, or dns, redirect_loop,
timeout, ...) and http.response.status_code, from failure_kind beside explain_failure, so
failures group by kind without a regex over msg.
- Each event was logged twice: words under ipx::scan and fields under ipx::io. It is now one
line under ipx::scan with both; the wire copy is at debug, for the admin page's Daemon I/O tab,
and out of production's log. The healthcheck's status reply stays under ipx::io.
- The access log's ms is duration_ms. The dashboard and the prod-check skill follow.
- error fields are Display with the anyhow chain everywhere, not a mix of Debug and Display.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through ipodderx.sdf1.net the page is https, the browser upgrades an http:// image to https,
and a host with no https, such as The Secret Cabal's CDN, answers nothing, so no artwork. On an
https page, the page now asks /api/art for those, and ipx fetches them. It only fetches an
address some feed or entry names as its artwork, and only an image, up to 5 MB, so the route
cannot be pointed at anything else on the network.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The id led the title's line unlabelled, so antirez.com's, "feed" with no title beside it, read
as a heading; 'ipx fetch antirez' was tried instead and failed. The title now heads the entry and
the id has its own row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was the one-off copy from SQLite to Postgres (#18), run once on 2026-09-18. Production has run
on Postgres since; rolling back needs only the old state.db, which is kept, not this command.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mark was a 12px "!" in the sidebar's margin, told apart by --bad alone; a dark theme's --bad
is a pale pink, and at that size it vanished. It is now a solid disc on the artwork's corner, the
subtitle says what is wrong in place of the counts, and a feed failing for a day or more has its
artwork greyed out. Lightness and words carry it, so no theme's palette changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A feed's itunes:image or <image><url> was stored without being asked
for, so a dead one stood in the way of the site's icon. Ken and Robin
Talk About Stuff names http://kenandrobin.wpengine.com/.../kartas_podcast.png,
a 404, while its site's apple-touch-icon works. The feed's artwork now has
to answer as an image, as the site icon already did, when the feed is
read in full; otherwise the site's icon is looked for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.claude/skills/ipx-prod-check: what production's JSON log and traces
carry, the queries that find trouble (warnings grouped, failing feeds and
downloads, 5xx and slow routes, whether the worker keeps up, slow and
failed traces), how to tell a publisher's dead feed from an ipx bug, and
filing what is found as issues per CLAUDE.md. query.py beside it runs the
LogQL and TraceQL through a throwaway container on the monitoring
network, since Loki and Tempo publish no query port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The log was text, so the Grafana dashboard picked lines apart with
regular expressions, and a change of wording would have blanked its
panels. With IPX_LOG_FORMAT=json each line is one JSON object: the
access log carries method, path, route, status and ms as fields (the
route passed from the routing layer in the response's extensions), and
each wire event its ev, feed, new, downloaded, failed, bytes, msg and
the rest (log_wire), beside the old message. The two startup lines that
were println! are logged, so no line breaks the JSON. Text stays the
default, for a terminal. The dashboard reads the fields with Loki's json
parser, and groups requests by route rather than path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Built on what the monitoring project already collects: the container's
log in Loki (through Alloy) and the traces in Tempo. It parses the
access log and the event log's wire JSON, so there are no metrics to
add to ipx: what is waiting and downloaded (from the healthcheck's
status), new items, downloads and bytes, failing feeds and downloads,
requests by status and response time, the slowest and busiest paths,
recent and slow traces, and the log. Provisioned from a file, so it is
regenerated here, not edited in Grafana.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The browser suite's daemon took its environment from the shell running
it, so a shell with OTEL_EXPORTER_OTLP_ENDPOINT set would have sent its
fixture scans to production's Tempo, and one with IPX_DATABASE_URL set
would have run the suite against production's database. Both are now
blanked for it. Production's traces carry
deployment.environment.name=production, which the dashboard filters on,
so a daemon run by hand stays out as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A request's trace was named by its path, so every item's GUID in
POST /api/entries/{feed_id}/{guid}/flags made a trace name of its own and
nothing grouped in Tempo. A route layer now renames it once routing has
matched. It renames the OpenTelemetry span directly: tracing-opentelemetry
drops a recorded otel.name once the span has been entered, and access_log
enters it before routing runs.
tracing-subscriber's fmt layer writes ANSI colour by default, so docker
logs and Loki (through Alloy) carried escape codes on every line, which
each query had to strip. Colour is now for a terminal only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx had no spans, only log lines, so there was no way to see where a
slow scan, download or request spent its time. With
OTEL_EXPORTER_OTLP_ENDPOINT set, the daemon now exports traces over
OTLP/HTTP (Tempo on Tower): a scan, each feed in it, the feed fetch and
site icon lookup, downloads, torrents, reaps, and web requests. Log lines
inside a span ride along as its events.
Only the daemon exports: the healthcheck runs ipx status every 30s and
would bury everything else. The web event stream and the log view's
polling get no span, for the same reason. The exporter shares ipx's
reqwest 0.13, so no second HTTP stack comes in.
The stderr log now prefixes lines inside a span with it, as
tracing-subscriber's fmt layer does (scan{only=None force=false}: ...).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before 0.6.0 the parser took WordPress's numbered player URLs (?_=2) for
separate files and downloaded some episodes twice. Since then it drops
the repeats while reading (same_file_key), and merge_repeated_enclosures
cleaned up what was already stored. Production has run it; on every
start since it has only cost a query that finds nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The icon standing in for a feed's missing artwork was looked up once and
kept, so a site that changed or fixed its icon, or a feed that dropped
its own artwork, kept whatever was found first. A dead icon stored
before #79 would have stayed dead. It is now looked up whenever the
feed is read in full: when it has changed, or on a refresh someone asks
for, which reads in full since #77.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
site_icon took the icon a site's page names in its <link> tags without
asking for it, so a dead one was stored and /favicon.ico never tried.
antirez.com names /images/favicon.png, which is a 404, while its
/favicon.ico is there; the feed showed no artwork, and since the lookup
happens once, never would. The named icon now has to answer with an
image, as /favicon.ico already did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The feed events already put a spinner on the sidebar row, which a phone
hides. The same state now sets scan-this (the open feed, or a feed in the
open folder) and scan-any (any of your feeds) on <body>, and the refresh
icons turn under them. On <body> because the feed page is redrawn as items
come in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check every feed, a feed's refresh, pull to refresh and ipx fetch --force
all send force, which only skipped the not-due wait: the request still
carried the stored ETag and Last-Modified, so an unchanged feed answered
304 and was not read. anil-dash got no site icon from a refresh for this
reason. A forced scan now drops the validators; the scheduled scan keeps
them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check-now button on a feed's page, a folder's page and All
Subscriptions was class primary, drawn filled in the accent colour among
plain buttons. Primary stays for a dialog's confirm button; the rules
that only the feed pages used go with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4ed2d59 drew a pressed toggle in each theme's accent colour; the themes'
colours were not to change. Back as they were, pinned rows included. The
read button carries its state in its shape instead, as the pin does with
outline and solid: a tick when read, the envelope when not, where before
it showed the action (the envelope on a read item).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep of all 24 theme palettes, measuring each icon's drawn colour,
found pinned in three colours: accent in the feed list, the text colour
on an item's row, and uncoloured on the toolbar, beside the title and on
the feed page. The read button showed the action (an envelope on a read
item) beside a pin showing the state. Now each toggle shows what is, with
aria-pressed, and a pressed one is the accent colour; Classic needs its
own rule, as its buttons set their colour at higher specificity. The
contrast test checks the accent on the button grounds, where it now draws.
Directory's Subscribe button carried the Subscribed tick; it is a plus.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The icon lookup ran only when a scan got the feed's body, and most feeds
answer 304 to their stored validators, so anildash.com and 68 others
stayed blank until their next post. A feed whose image was never looked
for is now refetched once without validators, as an empty one already
was. A miss is stored as "" (drawn as no art), so neither the refetch nor
the site lookup repeats on every scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both add paths, the CLI's and the web's, look behind the URL first: a web
page that names its feed with <link rel="alternate"> is swapped for that
feed, before the duplicate check so it finds a feed someone already has.
Before, the page itself was added and every scan failed on it.
alternate_feed_link found tags in a to_lowercase() copy and sliced the
original at those offsets; Unicode lowercasing changes some characters'
length, so a page with one before its <link> tags lost the href or
panicked off a char boundary. ASCII lowercasing keeps offsets aligned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a feed names no image and none is stored, the scan fetches the
channel's site link (RSS <link>, Atom rel=alternate) and uses the
apple-touch-icon or icon it names, falling back to /favicon.ico when that
answers with an image.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The theme was kept on the account, so every browser signed in as the
same person got the same one: no Glass on the phone with Dracula on the
desktop. It is now the ipx_theme cookie (<theme>.<mode>), written by
theme.ts, and read by the server to draw the page in it from the first
frame as before. /api/me no longer reports or takes a theme, and
set_theme is gone.
A browser with no cookie yet is sent the theme the account kept, and
takes it as its cookie on that first load, so nobody loses their choice
in the move. users.theme and theme_mode are only read now, for that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Letting go of a pull removed its note at once and showed nothing else;
the sidebar's scanning spinner is hidden on a phone. With no sign the
check had started, people pulled again, and again. A "Checking for new
items" pill with a spinner now sits under the top bar until the request
is sent and two seconds have passed, and a pull meanwhile does nothing.
It lives outside #list, which a feed's render rebuilds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
