29 Commits

Author SHA1 Message Date
552d20443e Release 0.10.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:09:51 +00:00
367f431e1f Sync SQLite at checkpoints, not after every write (#135)
The load tests (#133) found a daemon on SQLite scanning 1.3s a feed of 20 items served from the
same machine: 1,500 feeds would have taken about 50 minutes. On Tower's /tmp, a loop device, one
synchronous 4 KB write takes 52ms, and SQLite was in WAL mode with its default synchronous=FULL,
which waits for the disk on every commit; nearly every statement is a commit of its own, a
feed's entries and enclosures being written a row at a time. Subscribing the first admin to an
imported catalogue of 1,500 feeds took 39s, before the web server listened.

Every SQLite connection now sets synchronous=NORMAL, SQLite's own advice for WAL: it syncs at
checkpoints. A crash of ipx loses nothing; a power cut can lose the last transactions, and never
corrupts the file. The same scan takes 27s and the same subscribing 0.75s. Postgres, which
production runs on, is unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:09:29 +00:00
ef8e2526f2 Show what a feed says it is on its page in the Directory (#130)
A feed's page in the Directory (#128) had its cover, category and latest items, but not the
feed's own description, which Apple's show page leads with: iPX read every item's description
and threw the channel's away, and the feeds table had nowhere to keep it.

It is parsed now, RSS's <description>, or iTunes' summary when that is empty, or Atom's
subtitle, kept in feeds.description (kept when a later read has none, as title and image are),
and sent, sanitized, with the items from /api/directory/{id}, which is now an object, not a
list. The page shows it as plain text under the header, three lines of it, with More when there
is more. A description that only repeats the title is left out.

feeds.description is the first column added to a table that already exists. create_missing
looks for it with a SELECT and runs the ALTER only when it is missing, since it runs on every
open, the healthcheck's included, and an ALTER's lock is what made that time out before. The
same once-only step forgets every feed's ETag and Last-Modified: a feed is read whole only when
it has changed, so its description would otherwise wait for its next item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:46:22 +00:00
5058adc93e Open a Directory category's fan of covers as one smooth motion (#132)
Hovering a category tile turned its outer two covers further out over 0.2s, and it looked
choppy. Traced in Chromium, the covers were given layers of their own only once the transition
began, so the browser drew them again on its first frames and again on the way back, 22 raster
tasks for a hover in and out, and the main thread produced 34 frames for 0.4s of motion.

Each cover now has its layer before the pointer arrives (will-change), all three move, rotate
and translate are animated as properties of their own rather than as one transform list, and
the fan takes 0.45s on an ease that settles. The same hover is 5 raster tasks, and the
compositor presents 116 frames to the main thread's 26.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:46:22 +00:00
9e39f2d756 Draw every Directory category tile on the panel, untinted (#131)
Each tile took a faint tint from a hash of its category's name, as an initials tile does, so
neighbours differed. Five of them, Comedy, Government, Kids & Family, Leisure and Music, drew
--good, and green means yours everywhere else on the page, the subscribed tick and a downloaded
file among them: those tiles read as holding something you subscribe to, when the tint meant
nothing. The fanned covers already give each tile its colour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:33:57 +00:00
b86062b97a Keep a feed's items and files to the people who subscribe to it (#129)
Routes that take a feed or an enclosure id did not check who was asking. Anyone signed in could
read any feed's items through GET /api/feeds/{id}/entries, a paid feed's included, with the
addresses of its files, which can carry the subscriber's key: the Directory leaves such feeds
out for that reason, and this route handed them back to whoever guessed the id, a slug of the
title. In production it answered 25 items of a feed the asking account does not subscribe to.
/media/{id} served any downloaded file by its sequential id, POST /api/enclosures/{id}/download
and /api/feeds/{id}/download-latest queued any feed's downloads, and DELETE
/api/enclosures/{id}?force=true deleted any file.

Each now answers 404, "you do not subscribe to that feed", unless the person subscribes to it.
A feed inside an OPML has a subscription row of its own for everyone subscribed to the OPML, so
that holds for those feeds too. Found while adding the Directory's feed page (#128), which has
its own route that answers only for listed feeds and carries no files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:38 +00:00
95acde3046 Lay the Directory out as Apple's is, with the most subscribed in it (#124, #126, #128)
The Directory was one grid of every listed feed, 1,472 in production, under Podcasts and Blogs
tabs, a wall of about 30 category buttons and a sort menu, with a picked category's
subcategories appearing as a second row of buttons that looked like the first. It read as a
list to scroll, and the two rows were hard to tell apart.

It opens now on the ten most subscribed feeds, ranked, and the categories as tiles, each with
three of its shows' covers fanned in its corner on the tint its initials would get, so the
shows colour it and no theme's palette changes. A category has a page of its own: its most
subscribed, its subcategories with how many each holds, and its grid. See all is every feed,
as the grid was. Podcasts or Blogs holds across every page. Popular, the ten most subscribed,
had a place of its own in the feed list; it is the Directory's first section instead, and its
key, g p, is gone. /api/popular stays for scripts.

The search box finds a feed in the Directory by name (#126). It said "Search items…" there and
did nothing, since loadEntries returns early for a place that lists feeds.

A feed you do not subscribe to opens a page of its own (#128): its cover, category (a link to
that category's page), how many here subscribe, a subscribe button and its latest twenty items,
each with its title, linking to the post where it has one, its first lines, date and length.
Before, a click on it did nothing; only its + did. The items come from GET /api/directory/{id},
which answers only for a feed the Directory lists, so a guessed id reaches nothing private, and
carries no file or its address. The feed's own description would belong at the top, but the
database does not keep one.

Every tile and chart row takes the keyboard, as the feed list's rows do. Feed text made plain
keeps a space at a line break or a paragraph's end, which glued "2010)Recorded" together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:38 +00:00
0b4f5443b7 File Apple's Football and Soccer under Sports in the Directory (#125)
The Directory listed Football (6 feeds), Soccer (7) and "Soccer," (2) as categories of their
own beside Sports, and Software How-To (2) beside Technology. CATEGORIES spelt two of Apple's
Sports subcategories American Football and Football (Soccer), names no feed uses: Apple's list
says Football and Soccer, so a feed naming either fell through as a category of its own.

They are Apple's spelling now. The old spellings are aliases, since Jev picked from the list
while it had them and its answers are stored, and so is Software How-To, Apple's Technology
subcategory before its 2019 list. A comma or full stop at the end of a feed's category, as in
"Soccer,", is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:19:12 +00:00
e699baa22a Say plainly when a feed is given a category, in the changelog
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:18:18 +00:00
00bd58ac9d API tokens a person makes for scripts and agents to act as them (#123)
The API took a session cookie, a proxy's word or the shared admin token, so a script or an
agent working for one person had to sign in with their password and carry the cookie, or be
given the admin token. Settings now makes named tokens, ipx_ and 256 random bits, sent as
Authorization: Bearer. A token is its owner and no more. Only its SHA-256 is kept, in the
new api_tokens table, with when it was made and last used; it is shown once and revoked from
the same list. An unknown or revoked one gets a 401 rather than falling through to a cookie.

Cloudflare Access still stands in front of the tunnel, so from outside a token needs an
Access service token beside it; docs/sso.md says how.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 00:15:17 +00:00
cc17b1ddab Sort the Directory by name either way or by subscribers (#122)
The Directory always listed A to Z, as the server sends it. A menu beside its filters now
sorts it A to Z, Z to A or by most subscribers (then by name), in the page, without asking
the server again, and the choice stays while the pane is redrawn, as the filters do. The
blurb no longer says A to Z, since it may not be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:54:32 +00:00
119336a20d Give a pinned item the pinned feed's disc (#121)
A pinned feed has a disc in the theme's accent, its pin in the ink the theme pairs with it;
a pinned item only turned its pin solid in the text colour. The item's pin button now draws
the same disc, as a background inside its 22px button so the row does not move when an
item is pinned. The colours are the ones .fpin already uses; no palette changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:50:48 +00:00
02a46bb500 Browse the Directory by category, then subcategory, as Apple does (#118)
The Directory had one row of chips holding whatever each feed's category was, a category
(Technology) or a subcategory (Tech News, Video Games) side by side: a podcast's own
<itunes:category> is stored as its subcategory, and Jev's answers (#117) are often
subcategories too, so after the first forced scan the row held 22 chips. /api/directory now
gives each feed's Apple category and subcategory, worked out from Apple's list, and the
page shows the categories, then a picked one's subcategories on a line of their own. A
category that is not Apple's, one an admin typed, stands as a category of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:33:26 +00:00
3e384f34f9 Give a feed with no category of its own one of Apple's (#117)
Most blogs and news sites name no <itunes:category>, so they sat under no chip in the
Directory unless an admin set one by hand. When TYPESAFE_KEY is set, a feed's first read, or
a forced refresh, asks TypeSafe's Jev to pick one of Apple's categories or subcategories from
the feed's title and up to 15 item titles, for a catalogue feed with neither its own category
nor an admin's. The answer is stored as the catalogue's category, so a feed is not asked
twice, and an admin can change it. Only on a first read or a forced refresh to keep the calls
down: feeds already subscribed are categorised by one `ipx fetch --force`.

Tried on eight subscribed feeds on 2026-10-04: XDA, Daring Fireball, Ars Technica and The
Verge as Tech News, The Old New Thing as Technology, John D. Cook as Mathematics,
BoardGameGeek as Games, Pluralistic as News Commentary (0.55). About 2,000 input tokens a
feed at $0.042 a million. The likeliest answer is kept however unsure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:20:00 +00:00
d238681ec1 Name an item without a title from its own text, not "(untitled)" (#116)
RSS 2.0 makes an item's title optional, and some blogs leave it out on purpose: Scripting News
titles almost none of its posts. Fifty rows of "(untitled)" said nothing about any of them.

entryName gives an item its title, or the opening of its text (HTML read through DOMParser, an
inert document that loads nothing; cut at a word near 120 characters), or its file's name, or
its show and date, with a flag for a name that is not a title. The list sets that one in the
regular weight, as the text it is rather than a heading; the reader leaves out the heading so
the post starts with itself; the player, the lock screen, Currently Listening, the native shell,
Share and the queued toast use the same name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:46:47 +00:00
0395717c14 Put the bytecode cache's ignore line on a line of its own
The previous commit appended it to a file without a final newline, joining it to
.claude/settings.local.json and un-ignoring both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:27:19 +00:00
8ce68a881a Ignore Python's bytecode cache from the prod-check script
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:27:07 +00:00
e7c59489ee Announce a followed move as a feed_moved event (#115)
A feed moved to its new address was only logged by follow_move. It is now an event, feed_moved
with the feed and its old and new addresses, so it goes where every other event goes: the log,
with from and to as fields, the admin page's Scans view, `ipx fetch`, and the page, which gets
the feed's new row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:22:28 +00:00
f7f4b466dc Follow a feed that has moved for good to its new address (#115)
A feed whose address answered with a permanent redirect was read through it on every check,
and the catalogue kept the old address: 28 of 147 feeds in production, most http to https, some
to a new path or domain.

Feeds are now fetched with a client of their own that follows no redirects (Ctx::feed_client),
and feed::fetch follows them itself, up to 10 hops, so it sees each one. When every hop was
permanent (301 or 308) it says where the feed ended up, and the scan moves the feed there in the
catalogue (follow_move). A temporary hop (302, 307) anywhere moves nothing. A feed an OPML lists
is left alone, as the OPML would put the old address back, and so is a move onto an address
another feed has. A password goes only to the feed's own host, never to a redirect elsewhere;
reqwest's own following dropped it the same way. Ten hops is a loop, worded as reqwest worded
it so it still reads as redirect_loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:05:43 +00:00
ec8fd5dd86 Sleep until the next feed is due instead of scanning every minute (#114)
The daemon ticked every 60 s and ran a scan pass each time: the sweep, then a check-state query
per feed (about 180) to find which were due. In the six hours before, 293 of 362 passes found
nothing due. Now, after each pass, it works out when the earliest feed is due (due_at, shared
with the scan's own check, over Db::http_states, one query) and sleeps until then: at least
30 s, so a feed that never gets a check time cannot spin it, and at most 10 minutes, so what no
command announces, ipx add or a shorter schedule, is picked up. Commands still wake it at once,
and the first pass after starting runs straight away, as the tick's did. The scan reads every
feed's state in one query too.

The prod-check skill says what to expect now: tens of scans in six hours, and pending as the
real queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:56:13 +00:00
a387012c69 Count as queued only what a scan will download on its own (#113)
Every file in state 'pending' counted as waiting to download: 4420 in production, across 12
shows. Since #97 a scan only downloads among a feed's newest max_new_per_check items, so those
were back-catalogue episodes no scan would take; the real queue was 0.

A new state, 'held': listed and downloadable by hand, but outside the feed's newest items, or of
a feed nothing downloads automatically. Db::hold_back moves a feed's waiting files between
'pending' and 'held' each time the feed is due, changed or not, and again after its items are
stored, so a new episode, a raised limit or auto-download turned on or off moves them. A held
file keeps its item's place among the newest, as a downloaded one does. 'pending' now means
queued, so ipx status, /api/status and the dashboard read true without changing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:47:52 +00:00
a00516687a Keep artwork on disk and serve every image from iPX (#111)
The page loaded artwork from each publisher's server, or through /api/art, fetched every time,
for http-only hosts. Nothing was kept, every visit asked every publisher, and artwork went when
a publisher's server did.

- The page draws every image a feed or item names from /api/art. The first time, iPX fetches
  it (only an address a feed or item names, only an image, up to 5 MB, within the feed timeout)
  and keeps it in art/ beside the database, under a hash of its address with its type beside
  it (src/art.rs). Later it comes from disk, which marks it as used.
- art_cache_mb, a server setting on the admin page, 500 by default, caps what is kept: the
  sweep before each scan drops the least recently shown until it fits. 0 keeps nothing, and
  artwork is fetched through iPX each time. Settings saved before it get the default.
- Served from iPX's own address, someone else's image must stay an image: nosniff, and a CSP
  with sandbox, so an SVG opened on its own runs no script as iPX.
- The fixture server sends .jpg as image/jpeg, which /api/art requires.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 13:50:46 +00:00
38ebc7b02b Move old items' http artwork to https too, for hosts the feed no longer names (#110)
The first pass only asked the hosts the feed's current body names. IGN's feed kept five 2009
items with artwork on assets1/assets2.ignimgs.com, which its feed no longer mentions, so they
stayed on http. A scan now also asks, once per host, the hosts of the feed's stored http
artwork (Db::http_images).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 13:35:51 +00:00
54d827f655 Time out a hung feed, serve the precomposed touch icon, and store http artwork on https (#108, #109, #110)
#108: the HTTP client had no timeout, and scans handle feeds in order, so a hung server held
every scan. Dreamwidth answered 504 after 60-67 s for a day and each scan took 70-80 s instead of
15. A feed fetch, and a Patreon creator's show list, now gets 30 s from connecting to the last
byte (feed::FEED_TIMEOUT); the client gets a 10 s connect timeout, which bounds a download's
start but not a long download.

#109: iOS asks for /apple-touch-icon-precomposed.png first when the site is added to a home
screen; it was a 404 and the only non-feed warning in the log. It serves the same icon.

#110: the page is https and loads no http. Artwork on http came through /api/art (#90) even
when its host serves https too. A scan now tries each http artwork host on https once per feed
(feed::prefer_https) and stores the https address where the host answers with an image,
rewriting that feed's stored items from the same host (Db::secure_images). 4 of the 5 hosts in
production do; cdn.thesecretcabal.com presents another name's certificate and stays on
/api/art.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 13:31:50 +00:00
e2593eaa50 Put a pinned feed's pin on the corner of its artwork
The pin was a small accent-coloured icon before the feed's name. It is now a disc on the
artwork's corner, where a failing feed's mark is, in the accent and the ink the theme already
pairs with it for primary buttons (checked by tests/contrast.js), so no palette changes. A feed
both pinned and failing keeps the error mark at the bottom corner and the pin at the top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:23:40 +00:00
d52ecfcbd7 A listed feed nobody subscribes to downloads nothing (#107)
With no subscribers a feed falls back to its own settings, where auto_download is on, so a
listed feed with audio would have downloaded files for no one. The seeded news feeds carry
only images, which media_types already skips, so nothing was downloaded. Files skipped for it
are judged again, by the new subscriber's settings, on the next scan after someone subscribes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:20:09 +00:00
142610e8b8 ipx add --list or --category updates a feed the catalogue already has (#107)
It refused one already there ("already subscribed as ..."), so a feed added before listing
existed, such as CBC's, could not be put in the Directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:17:49 +00:00
43acc62259 List feeds in the Directory before anyone subscribes, and clear out dead ones (#107)
The Directory showed a catalogue feed only once someone subscribed, and a feed left the
catalogue with its last subscriber, so nothing could be put there for others to find.

- A feed has a listed flag, set by ipx add --list (with --category for the Directory's chip).
  The web page keeps a listed feed in the catalogue when its last subscriber leaves.
- The Directory lists every catalogue feed; Popular still only what people subscribe to.
  popular() reads titles, artwork and categories through Db::feed_list, not three queries a
  feed. Subscribing from the Directory scans the feed at once.
- A feed nobody subscribes to is checked once a day at most.
- clean_directory, in the sweep before each scan, removes from the catalogue and the database
  a feed nobody subscribes to, with no file on disk and not from an OPML, that has failed for
  30 days or published nothing in a year. Run against production first: it removes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:13:13 +00:00
65fdab8b13 Between releases, the version says a release is in progress: 0.9.2-dev
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:52:00 +00:00
32 changed files with 1902 additions and 273 deletions

View File

@@ -32,12 +32,13 @@ lines had no trace id, the access log's time was `ms`, and each event was logged
| target | fields | what |
|---|---|---|
| `ipx::http` | `method`, `path`, `route`, `status`, `duration_ms` | one per web request; `route` is the pattern, empty for an unrouted path |
| `ipx::scan` | `ev` and the event's own: `feed`, `new`, `downloaded`, `failed`, `bytes`, `msg`, `url`, `feeds`, `reason`; on a failure `error.type` and, from an HTTP error, `http.response.status_code` | the daemon's events, one line each, in words; warnings are feed and download failures |
| `ipx::scan` | `ev` and the event's own: `feed`, `new`, `downloaded`, `failed`, `bytes`, `msg`, `url`, `feeds`, `reason`, `from`, `to`; on a failure `error.type` and, from an HTTP error, `http.response.status_code` | the daemon's events, one line each, in words; warnings are feed and download failures |
| `ipx::io` | `ev`, `feeds`, `pending`, `downloaded` on the `status` reply | commands arriving (`-> {...}`) and the healthcheck's answer |
| `ipx` | message, sometimes fields | start-up, shutdown, account and config messages |
Events (`ev`): `feed_start`, `feed_done` (new, downloaded, failed, torrents), `feed_skip` (not due,
routine), `feed_error` (msg), `download_done` (bytes), `download_error` (msg, url),
routine), `feed_error` (msg), `feed_moved` (from, to: a permanent redirect followed, the address
updated), `download_done` (bytes), `download_error` (msg, url),
`torrent_deferred`, `reaped`, `scan_done` (feeds checked), `reap_done`, `status` (feeds, pending,
downloaded: the healthcheck's, every 30s), `error` (msg).
@@ -85,6 +86,12 @@ a count says something happened, the lines and traces say why.
crash or an OOM kill: check `docker inspect iPX -f '{{.State.OOMKilled}} {{.RestartCount}}'`
and the lines just before it. A pending count that only grows means downloads are not
keeping up or not running.
Since 2026-10-02 the daemon sleeps until the next feed is due (at most 10 minutes) instead of
scanning every minute (#114), so expect tens of scans in 6 hours, not 360, nearly all with
`feeds` above 0; none at all for over 10 minutes means the worker is stuck. And `pending` is
the real queue (#113): files a scan will download on its own. Back-catalogue files are
`held`, listed but not counted, so it is usually 0 or a handful.
5. **Slow and failed traces.**
```
$Q traces '{resource.deployment.environment.name="production" && duration > 5s}'

3
.gitignore vendored
View File

@@ -4,4 +4,5 @@
/playwright-report
/web/dist
.claude/settings.local.json
.claude/settings.local.json
__pycache__/

View File

@@ -7,6 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.10.0] - 2026-10-05
### Added
- API tokens: in Settings, make a named token that lets a script or an agent use iPX as you, sent as `Authorization: Bearer`, and revoke it there. Each shows when it was last used.
- In the Directory, a feed you do not subscribe to opens a page of its own: its cover, its category, how many people here subscribe, what the feed says it is, and its latest episodes or posts, with a button to subscribe. Before, only its subscribe button did anything.
- Feeds that don't set a category, as most blogs don't, now get one automatically, so they show up under a category in the Directory. iPX picks the closest of Apple's podcast categories from the feed's title and latest headlines. This happens once per feed: the first time a new feed is read, or for a feed you already had, the next time you force a refresh. It needs `TYPESAFE_KEY` set. An admin can change the category in the feed's settings.
- Artwork is kept on disk once shown and loads from iPX, not from each publisher's server: fast after the first time, and still there when the publisher's server is not. The admin page sets how much is kept (500 MB by default).
- `ipx add --list --category News <url>` puts a feed in the Directory for anyone to subscribe to,
and it stays there when its last subscriber leaves. Run for a feed already in the catalogue,
it lists it or sets its category.
- Feeds nobody subscribes to that are dead for a month or quiet for a year are removed, so the
Directory lists feeds worth taking.
### Changed
- The Directory is laid out as Apple's is. It opens on the ten most subscribed feeds, ranked, and the categories as tiles showing some of their shows' covers. A category opens a page of its own with its most subscribed feeds, its subcategories (Tech News under News, Video Games under Leisure) and its feeds; See all lists every one. This replaces the grid of every feed under rows of category buttons.
- The most subscribed feeds are the Directory's first section instead of a place of their own: Popular is gone from the feed list, and so is its key, g p.
- A pinned item in a list wears the same disc as a pinned feed, in the theme's accent, instead of a plain solid pin.
- The Directory can be sorted by name, A to Z or Z to A, or by most subscribers. It still opens A to Z.
- An item published without a title shows its opening words, in plain text rather than bold, instead of "(untitled)"; one with no text either shows its file's name, or its show and date. Opened, it starts with its text.
- A feed that has moved for good (a permanent redirect) is followed to its new address, which iPX then reads from, and says so in the log as `feed_moved`. A temporary redirect changes nothing.
- The daemon sleeps until the next feed is due, at most ten minutes, instead of looking every minute; refreshing or adding a feed still wakes it at once.
- A pinned feed's pin sits on the corner of its artwork, as a failing feed's mark does, instead of before its name.
- The Directory lists feeds nobody subscribes to yet; its most subscribed still lists what people subscribe to.
A feed nobody subscribes to is checked once a day, and at once when someone subscribes. A
listed feed downloads nothing until someone subscribes.
- The Directory loads faster: it asked the database three questions per feed.
### Fixed
- A database on SQLite, the default, no longer waits for the disk after nearly every write. On a slow disk a scan of 1,500 feeds took most of an hour and now takes half a minute. Postgres is unaffected.
- The search box finds a feed in the Directory by name. It did nothing there.
- Feeds that file themselves under Apple's Football or Soccer show under Sports, and those under the old Software How-To under Technology, instead of as categories of their own.
- The download queue, in `ipx status`, `/api/status` and the dashboard, counts only what iPX will download on its own: older episodes beyond a show's limit, and files of feeds that do not download automatically, are listed but no longer counted as waiting.
- Artwork published on http is stored on https when its host serves it there, so the page loads it directly; the rest still comes through iPX.
- An iPhone adding the site to its home screen finds the icon at the first address it tries.
- A feed whose server hangs no longer holds up every scan: a feed gets 30 seconds, and connecting anywhere 10.
### Security
- A feed's items and files reach only the people who subscribe to it. Anyone signed in could read any feed's items, a paid feed's included, with the addresses of its files, which can carry the subscriber's key, and could play, download or delete its files.
## [0.9.1] - 2026-09-29
### Added
@@ -692,7 +735,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
[unreleased]: https://git.sdf1.net/rays/ipx/compare/v0.9.0...main
[unreleased]: https://git.sdf1.net/rays/ipx/compare/v0.10.0...main
[0.10.0]: https://git.sdf1.net/rays/ipx/compare/v0.9.1...v0.10.0
[0.9.1]: https://git.sdf1.net/rays/ipx/compare/v0.9.0...v0.9.1
[0.9.0]: https://git.sdf1.net/rays/ipx/compare/v0.8.4...v0.9.0
[0.8.4]: https://git.sdf1.net/rays/ipx/compare/v0.8.3...v0.8.4

View File

@@ -182,7 +182,8 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
command, the healthcheck's `ipx status` included, so it must never write when nothing is
missing: SeaORM's experimental schema sync dropped and remade an index on every open, the
write lock that took made `ipx status` time out behind a busy daemon, and it was removed for
it. A new column on an existing table needs its own `ALTER`; nothing adds one for you.
it. A new column on an existing table needs its own `ALTER`; nothing adds one for you. Look for
the column first and alter only when it is missing, as `feeds.description` does.
* **SQL written by hand in `db.rs` has to run on SQLite and Postgres both** (issue #18): `$1`
parameters, bound only if used; `ON CONFLICT`, not `INSERT OR IGNORE`; yes/no columns tested
as themselves (`NOT coalesce(s.read, false)`) and written as `true`/`false`, never compared to

3
Cargo.lock generated
View File

@@ -1820,7 +1820,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]]
name = "ipx"
version = "0.9.1"
version = "0.10.0"
dependencies = [
"ammonia",
"anyhow",
@@ -1843,6 +1843,7 @@ dependencies = [
"sea-orm",
"serde",
"serde_json",
"sha2 0.10.9",
"tokio",
"toml",
"tower",

View File

@@ -1,6 +1,6 @@
[package]
name = "ipx"
version = "0.9.1"
version = "0.10.0"
edition = "2024"
[dependencies]
@@ -25,6 +25,7 @@ rss = "2.1.1"
sea-orm = { version = "2.0.3", default-features = false, features = ["sqlx-sqlite", "sqlx-postgres", "runtime-tokio-rustls", "macros", "with-json", "sqlite-use-returning-for-3_35"] }
serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151"
sha2 = "0.10.9"
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
toml = "1.1.5"
tower = { version = "0.5.3", features = ["util"] }

View File

@@ -9,7 +9,7 @@ behind iPodderX (2004-2008, Ray Slakinski & August Trometer).
## What it does
- **The web UI.** It has a toolbar, and a feed list that opens with Directory, Popular and All
- **The web UI.** It has a toolbar, and a feed list that opens with the Directory, Currently Listening and All
Subscriptions. Items sit in a sortable table with a Files pane, and there is a player bar. It
comes in Dark, Light and Classic themes, and works on a phone.
- **Several people, one copy.** Each person has their own subscriptions and their own read, pinned

View File

@@ -51,8 +51,8 @@ build needs node and `npm ci` run once.
## Data model
```
feeds id, url, title, image, etag, last_modified, last_checked, ttl_mins,
last_error, orphaned, group_id, managed
feeds id, url, title, image, category, description, etag, last_modified,
last_checked, ttl_mins, last_error, orphaned, group_id, managed
entries feed_id, guid, title, link, published, description, first_seen,
image, duration, episode, season PK (feed_id, guid)
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
@@ -71,7 +71,8 @@ before accounts; two bugs came from queries still reading them, and they were dr
Schema changes: the tables are the entities in `src/entity.rs`, and `Db::open` creates whatever
table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It
never alters a table that exists, so a new column on one needs its own `ALTER` in
`create_missing`, or `sea-orm-migration` once there are several. `Db::memory()` builds its
`create_missing`, looked for first and run only when it is missing, as `feeds.description`'s is,
or `sea-orm-migration` once there are several. `Db::memory()` builds its
database the same way, so the tests run on the schema production gets. A database from before
0.7 takes its last columns from the old `migrate()`, so it upgrades through a 0.7 release first.
@@ -110,7 +111,8 @@ so a long download had to be able to notice the signal itself.
## HTTP API
Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
else a `401`.
else a `401`. A feed's items and files (its entries, `download-latest`, `/api/enclosures/{id}` and
`/media/{id}`) answer only someone who subscribes to the feed; anyone else gets a `404`.
| Route | |
|---|---|
@@ -126,6 +128,7 @@ else a `401`.
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch` | |
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
| `GET /api/directory/{id}` | a listed feed's description and latest twenty items, for its page before you subscribe: title, link, date, length and text, never a file or its address, and only for a feed the directory lists |
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours, the feed's iTunes category, whether it carries audio or video; never a URL, never a private feed); subscribe by id |
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |

View File

@@ -4,7 +4,7 @@ Two places. **config.toml** holds what ipx needs before it reaches its database,
who gets in: where things are (`download_dir`, `socket`, `organize`), `[torrent]` and `[web]`.
**The database** holds the catalogue of feeds (`[feeds.<id>]` below) and the server settings the
admin page edits (`schedule`, `max_total_gb`, `max_age_days`, `max_new_per_check`,
`media_types`). Change those in the web UI, or with `ipx add`, `ipx rm` and `ipx import`; they
`media_types`, `art_cache_mb`). Change those in the web UI, or with `ipx add`, `ipx rm` and `ipx import`; they
take effect without a restart.
The first time ipx meets a database that holds no catalogue, it takes the feeds and those
@@ -39,9 +39,10 @@ max_total_gb = 50 # 0 = unlimited
max_age_days = 30 # 0 = keep forever
max_new_per_check = 3 # per feed, per scan. 0 = unlimited
media_types = ["audio", "video"]
art_cache_mb = 500 # artwork kept on disk. 0 = none
```
`schedule`, `max_total_gb`, `max_age_days`, `max_new_per_check` and `media_types` move into the
`schedule`, `max_total_gb`, `max_age_days`, `max_new_per_check`, `media_types` and `art_cache_mb` move into the
database as described above; `download_dir`, `socket` and `organize` stay in config.toml.
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
@@ -54,6 +55,10 @@ database as described above; `download_dir`, `socket` and `organize` stay in con
read it. `0` disables it entirely.
* **`max_age_days`** — items older than this with no file on disk are pruned from the database.
Kept ones stay. `0` disables it.
* **`art_cache_mb`** — how much show and episode artwork iPX keeps on disk, in `art/` beside the
database, so the page loads it from iPX rather than from every publisher's server. Over this,
what has gone longest unshown goes first. `0` keeps none: artwork still comes through iPX, fetched
each time. 500 by default.
* **`max_new_per_check`** — how many of a feed's newest episodes are downloaded; older ones stay
listed to download by hand. It stops a new subscription pulling a whole back catalogue.
`0` means every episode, for an archive; set it on the feeds you want archived, since on the
@@ -124,6 +129,7 @@ folder = "Accidental Tech Podcast" # default: the feed title
schedule = "every 6h" # overrides [general] for this feed
media_types = ["audio"] # overrides [general] for this feed
category = "Technology" # the Directory's, if the feed names none
listed = true # in the Directory with no subscribers
username = "ray" # HTTP basic auth
password_env = "IPX_ATP_PASS" # preferred over a literal `password`
```
@@ -133,6 +139,12 @@ With more than one account, **`keywords`, `auto_download`, `allow_explicit` and
config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself
and are the same for everyone. See [users.md](users.md).
**The Directory** lists every feed in the catalogue, whether or not anyone subscribes to it yet.
To put one there for others to find, `ipx add --list --category News <url>`: it subscribes no one,
and a listed feed stays when its last subscriber leaves, where another is dropped. A feed nobody
subscribes to is checked once a day. One that is dead (failing for 30 days) or quiet (nothing
new in a year), with nobody subscribed and no file on disk, is removed from the catalogue.
Feeds derived from a subscribed OPML are **not** in the catalogue: the OPML is the source of truth
and they are re-derived on every scan. Editing one in the UI promotes it to a catalogue entry.

View File

@@ -233,5 +233,19 @@ Set `auto_create_users = false` once everyone who should have an account has one
proxy vouching for an unknown name is logged and refused. Make people ahead of time instead, with
the exact name the header will carry.
## API tokens for scripts and agents
Anyone can make API tokens for their own account in Settings. A token is sent as
`Authorization: Bearer ipx_...` and acts as the person who made it, admin only if they are. Only
its SHA-256 is kept; it is shown once, when made, and revoked there too.
Through the tunnel, Cloudflare Access turns a request with no Access sign-in towards Authentik
before ipx sees it, so a token alone does not get in that way. On the LAN, `http://192.168.1.130:8099`
takes it directly. From outside, make an Access service token, add a Service Auth policy for it to
the `ipodderx` application, and send `CF-Access-Client-Id` and `CF-Access-Client-Secret` beside the
`Authorization` header: Access lets the request through, vouches for no name, and ipx takes the
API token as who is asking. Do not bypass Access for `/api/*` instead; the token would then be the
only thing between the internet and the API.
See also [users.md](users.md) for what several people share, [configuration.md](configuration.md)
for every `[web]` key, and [cli.md](cli.md) for the `ipx user` commands.

View File

@@ -70,11 +70,15 @@ list with their own read state. Unsubscribing removes it from their list alone;
subscriber leaves does the feed stop being scanned, and even then its files and history stay, so
re-subscribing does not pull the back catalogue again.
**Popular** and **Directory** sit at the top of the feed list, above your own feeds. Popular, also
shown in the Add feed dialog, lists the ten feeds with the most subscribers on this server, you
included. Directory shows every one of them A to Z as a grid of cover art. Above it, a filter
picks Podcasts (anything with audio or video) or Blogs (the rest), and chips pick the category
each show gives itself in iTunes; the two combine. Your own feeds are marked Subscribed.
**Directory** sits at the top of the feed list, above your own feeds. It opens on the ten feeds
with the most subscribers on this server, you included, ranked, then the categories, each a tile
showing a few of its shows' covers. A category has a page of its own: its most subscribed feeds,
then all of them as a grid of cover art, with its subcategories along the top (Tech News under
News, Video Games under Leisure) and a choice of order. See all lists every feed, and the search
box finds one by name. A feed you do not subscribe to opens a page of its own, with what it says it
is and its latest episodes or posts, so you can see what it is first. Every page can be narrowed to Podcasts (anything with audio or video) or
Blogs (the rest). A show's category is the one it gives itself in iTunes, or for one that names
none, one an admin or iPX picked for it. Your own feeds are marked Subscribed.
It shows a title, artwork and a count, never a URL or who reads it. An OPML subscription is listed
as the feeds inside it, one by one, and never the OPML itself, so you can take just the shows you
want. Anything that looks private is left out: a login configured for the feed, credentials in its URL,

92
src/art.rs Normal file
View File

@@ -0,0 +1,92 @@
//! Artwork kept on disk, so the page loads it from iPX: fast after the first time, nothing asked
//! of each publisher's server for every visit, and still there when a publisher is not.
use std::path::{Path, PathBuf};
pub fn dir() -> PathBuf {
crate::config::data_dir().join("art")
}
/// Where an image's bytes are kept; its content type is beside it, in `<name>.type`.
// ponytail: std's hasher, 64 bits, keyed by the URL. Its algorithm may change with Rust, which
// only makes the cache miss once and refill; a collision among a few thousand images is about
// one in 10^12. Move to a SHA if either ever matters.
pub fn path(url: &str) -> PathBuf {
use std::hash::{Hash, Hasher};
let mut h = std::collections::hash_map::DefaultHasher::new();
url.hash(&mut h);
dir().join(format!("{:016x}", h.finish()))
}
/// A kept image and its type, marked as just used, which is what keeps it from being trimmed.
pub fn read(file: &Path) -> Option<(String, Vec<u8>)> {
let kind = std::fs::read_to_string(file.with_extension("type")).ok()?;
let body = std::fs::read(file).ok()?;
if let Ok(f) = std::fs::File::options().write(true).open(file) {
let _ = f.set_modified(std::time::SystemTime::now());
}
Some((kind, body))
}
/// Keeps an image, written aside and renamed into place, so a page asking for it meanwhile
/// never reads half of one.
pub fn write(file: &Path, kind: &str, body: &[u8]) -> std::io::Result<()> {
std::fs::create_dir_all(file.parent().unwrap_or(Path::new(".")))?;
let tmp = file.with_extension("part");
std::fs::write(&tmp, body)?;
std::fs::write(file.with_extension("type"), kind)?;
std::fs::rename(&tmp, file)
}
/// Removes the least recently used images until what is kept fits in `limit` bytes; 0 empties it.
/// Returns how many went.
pub fn trim(dir: &Path, limit: u64) -> usize {
let Ok(entries) = std::fs::read_dir(dir) else { return 0 };
let mut kept: Vec<(std::time::SystemTime, u64, PathBuf)> = entries
.flatten()
.map(|e| e.path())
.filter(|p| p.extension().is_none())
.filter_map(|p| {
let m = p.metadata().ok()?;
Some((m.modified().ok()?, m.len(), p))
})
.collect();
let mut total: u64 = kept.iter().map(|(_, n, _)| n).sum();
kept.sort();
let mut gone = 0;
for (_, n, p) in kept {
if total <= limit {
break;
}
let _ = std::fs::remove_file(p.with_extension("type"));
if std::fs::remove_file(&p).is_ok() {
total -= n;
gone += 1;
}
}
gone
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_least_recently_used_go_first_until_it_fits() {
let d = std::env::temp_dir().join(format!("ipx-art-test-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&d);
for (name, age) in [("old", 300), ("mid", 200), ("new", 100)] {
let f = d.join(name);
write(&f, "image/png", &[0; 1000]).unwrap();
let t = std::time::SystemTime::now() - std::time::Duration::from_secs(age);
std::fs::File::options().write(true).open(&f).unwrap().set_modified(t).unwrap();
}
// Shown just now: no longer the oldest.
assert_eq!(read(&d.join("old")).unwrap().0, "image/png");
assert_eq!(trim(&d, 2000), 1);
assert!(!d.join("mid").exists() && !d.join("mid.type").exists());
assert!(d.join("old").exists() && d.join("new").exists());
assert_eq!(trim(&d, 0), 2);
let _ = std::fs::remove_dir_all(&d);
}
}

View File

@@ -40,6 +40,19 @@ pub fn new_session_token() -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
/// A new API token: `ipx_` and a session's 256 random bits, the prefix so one found in a log
/// or a file says what it opens (#123).
pub fn new_api_token() -> String {
format!("ipx_{}", new_session_token())
}
/// What is kept of an API token. A fast hash is enough: the token is 256 random bits, not a
/// password, so there is nothing to guess from a stolen hash.
pub fn api_token_hash(token: &str) -> String {
use sha2::Digest;
sha2::Sha256::digest(token.as_bytes()).iter().map(|b| format!("{b:02x}")).collect()
}
fn getrandom(buf: &mut [u8]) -> std::io::Result<()> {
use std::io::Read;
std::fs::File::open("/dev/urandom")?.read_exact(buf)
@@ -74,6 +87,21 @@ mod tests {
assert!(hash_password("short").is_err());
}
#[test]
fn api_tokens_are_prefixed_and_hash_to_hex() {
let t = new_api_token();
assert!(t.starts_with("ipx_") && t.len() == 68);
let h = api_token_hash(&t);
assert_eq!(h.len(), 64);
assert_eq!(h, api_token_hash(&t), "the same token, the same hash");
assert_ne!(h, api_token_hash(&new_api_token()));
assert_eq!(
api_token_hash("abc"),
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
"SHA-256"
);
}
#[test]
fn session_tokens_are_long_and_distinct() {
let a = new_session_token();

View File

@@ -39,6 +39,9 @@ pub struct General {
/// image in an <enclosure>, so taking everything filled the disk with artwork and
/// counted it as episodes. Empty means take anything.
pub media_types: Vec<String>,
/// How much artwork iPX keeps on disk, in MB, so the page loads it from iPX rather than from
/// every publisher; the least recently shown goes first. 0 keeps none.
pub art_cache_mb: u64,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize, Serialize)]
@@ -163,6 +166,10 @@ pub struct Feed {
/// Preferred over `password`: name of an env var holding the password.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub password_env: Option<String>,
/// Put in the Directory by an admin (`ipx add --list`): listed with no subscribers, and kept
/// in the catalogue when the last one leaves, where anyone else's feed is dropped.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub listed: bool,
}
fn yes() -> bool {
@@ -180,6 +187,7 @@ impl Default for General {
max_age_days: 0,
max_new_per_check: 3,
media_types: vec!["audio".into(), "video".into()],
art_cache_mb: 500,
}
}
}
@@ -290,10 +298,18 @@ pub struct Stored {
pub max_age_days: u64,
pub max_new_per_check: usize,
pub media_types: Vec<String>,
/// Settings saved before it existed have none: they get the default.
#[serde(default = "art_cache_mb")]
pub art_cache_mb: u64,
}
fn art_cache_mb() -> u64 {
General::default().art_cache_mb
}
/// `[general]` keys that live in the database once it holds the configuration.
const STORED_KEYS: [&str; 5] = ["schedule", "max_total_gb", "max_age_days", "max_new_per_check", "media_types"];
const STORED_KEYS: [&str; 6] =
["schedule", "max_total_gb", "max_age_days", "max_new_per_check", "media_types", "art_cache_mb"];
impl Stored {
pub fn of(cfg: &Config) -> Self {
@@ -304,6 +320,7 @@ impl Stored {
max_age_days: g.max_age_days,
max_new_per_check: g.max_new_per_check,
media_types: g.media_types.clone(),
art_cache_mb: g.art_cache_mb,
}
}
@@ -314,6 +331,7 @@ impl Stored {
g.max_age_days = self.max_age_days;
g.max_new_per_check = self.max_new_per_check;
g.media_types = self.media_types;
g.art_cache_mb = self.art_cache_mb;
}
}
@@ -571,6 +589,12 @@ mod tests {
assert!(wanted_media(Some("image/jpeg"), &["image/jpeg".to_string()]));
}
#[test]
fn settings_saved_before_the_art_cache_keep_the_default() {
let old = r#"{"schedule":"every 1h","max_total_gb":0.0,"max_age_days":0,"max_new_per_check":3,"media_types":["audio"]}"#;
assert_eq!(serde_json::from_str::<Stored>(old).unwrap().art_cache_mb, 500);
}
#[test]
fn slugs_are_readable_and_unique() {
assert_eq!(slug("Accidental Tech Podcast"), "accidental-tech-podcast");
@@ -584,7 +608,7 @@ mod tests {
taken.insert("the-daily".to_string(), Feed {
url: "u".into(), folder: None, group: None, media_types: None, schedule: None, keywords: vec![], allow_explicit: false,
auto_download: true, max_new_per_check: None, username: None,
password: None, password_env: None, category: None,
password: None, password_env: None, category: None, listed: false,
});
assert_eq!(unique_slug("The Daily", &taken), "the-daily-2");
}
@@ -605,6 +629,7 @@ mod tests {
password: Some("literal".into()),
password_env: None,
category: None,
listed: false,
};
assert_eq!(f.password().as_deref(), Some("literal"));

325
src/db.rs
View File

@@ -2,7 +2,7 @@
//! per-feed .ipxd plists, history.dat and qmcache.dat.
use anyhow::{Context, Result};
use crate::entity::{blocklists, catalogue, enclosures, entries, feeds, hidden, sessions, settings, subscriptions, users};
use crate::entity::{api_tokens, blocklists, catalogue, enclosures, entries, feeds, hidden, sessions, settings, subscriptions, users};
use sea_orm::sea_query::{Expr, Func};
use sea_orm::{
ActiveModelTrait, ColumnTrait, ConnectionTrait, EntityTrait, PaginatorTrait, QueryFilter, QueryOrder, Set,
@@ -59,6 +59,7 @@ async fn create_missing(orm: &sea_orm::DatabaseConnection) -> Result<()> {
schema.create_table_from_entity(settings::Entity),
schema.create_table_from_entity(blocklists::Entity),
schema.create_table_from_entity(hidden::Entity),
schema.create_table_from_entity(api_tokens::Entity),
] {
orm.execute(table.if_not_exists()).await.context("creating the schema")?;
}
@@ -68,6 +69,21 @@ async fn create_missing(orm: &sea_orm::DatabaseConnection) -> Result<()> {
] {
orm.execute_unprepared(sql).await.with_context(|| sql.to_owned())?;
}
// A column added to a table that already exists, which CREATE TABLE IF NOT EXISTS leaves
// as it was. Looked for first, and added only when it is missing: this runs on every open,
// the healthcheck's `ipx status` included, and an ALTER takes a lock that waits behind a
// busy daemon, which is what made the healthcheck time out once before.
if orm.execute_unprepared("SELECT description FROM feeds LIMIT 0").await.is_err() {
// A feed is read whole only when it has changed since the last read, so each one's
// description would wait for its next item. Forgetting how each was last seen makes
// its next read, when it is due anyway, a whole one.
for sql in [
"ALTER TABLE feeds ADD COLUMN description TEXT",
"UPDATE feeds SET etag = NULL, last_modified = NULL",
] {
orm.execute_unprepared(sql).await.with_context(|| sql.to_owned())?;
}
}
Ok(())
}
@@ -98,11 +114,17 @@ fn is_postgres(location: &str) -> bool {
location.starts_with("postgres://") || location.starts_with("postgresql://")
}
/// sqlx's defaults for SQLite are what ipx wants: foreign keys on, and a five-second wait for a
/// lock, which is what rusqlite was set to.
/// sqlx's defaults for SQLite are what ipx wants, foreign keys on and a five-second wait for a
/// lock, which is what rusqlite was set to, but for one.
async fn connect(location: &str) -> Result<sea_orm::DatabaseConnection> {
let mut opts = sea_orm::ConnectOptions::new(url_for(location));
opts.sqlx_logging(false);
// NORMAL, SQLite's own advice for WAL: synced at checkpoints, not after every commit. FULL,
// the default, waited for the disk after nearly every statement, and a feed's items are
// written a row at a time: 1.3s a feed on Tower's /tmp, where one synchronous write takes
// 52ms, and 39s to subscribe an admin to 1,500 feeds (#135). A crash of ipx loses nothing;
// only a power cut can lose the last transactions, and it never corrupts the file.
opts.map_sqlx_sqlite_opts(|o| o.synchronous(sea_orm::sqlx::sqlite::SqliteSynchronous::Normal));
sea_orm::Database::connect(opts)
.await
.with_context(|| format!("opening {}", redact(location)))
@@ -313,6 +335,11 @@ impl Db {
Ok(())
}
/// What a feed says it is, for its page in the Directory.
pub async fn feed_description(&self, feed_id: &str) -> Result<Option<String>> {
Ok(feeds::Entity::find_by_id(feed_id.to_owned()).one(&self.orm).await?.and_then(|f| f.description))
}
#[tracing::instrument(skip_all)]
pub async fn feed_summary(&self, feed_id: &str) -> Result<FeedSummary> {
let mut sum = feeds::Entity::find_by_id(feed_id.to_owned())
@@ -438,19 +465,27 @@ pub struct HttpState {
pub error_since: Option<i64>,
}
impl From<feeds::Model> for HttpState {
fn from(f: feeds::Model) -> Self {
HttpState {
etag: f.etag,
last_modified: f.last_modified,
last_checked: f.last_checked,
ttl_mins: f.ttl_mins.map(|t| t.max(0) as u64),
error_since: f.error_since,
}
}
}
impl Db {
pub async fn http_state(&self, feed_id: &str) -> Result<HttpState> {
Ok(feeds::Entity::find_by_id(feed_id.to_owned())
.one(&self.orm)
.await?
.map(|f| HttpState {
etag: f.etag,
last_modified: f.last_modified,
last_checked: f.last_checked,
ttl_mins: f.ttl_mins.map(|t| t.max(0) as u64),
error_since: f.error_since,
})
.unwrap_or_default())
Ok(feeds::Entity::find_by_id(feed_id.to_owned()).one(&self.orm).await?.map(HttpState::from).unwrap_or_default())
}
/// Every feed's, in one query: a scan, and the daemon working out when the next is due, asked
/// feed by feed, 180 queries a minute (#114).
pub async fn http_states(&self) -> Result<std::collections::HashMap<String, HttpState>> {
Ok(feeds::Entity::find().all(&self.orm).await?.into_iter().map(|f| (f.id.clone(), HttpState::from(f))).collect())
}
/// Upsert after a successful poll. Clears any previous error.
@@ -466,13 +501,14 @@ impl Db {
ttl_mins: Option<u64>,
image: Option<&str>,
category: Option<&str>,
description: Option<&str>,
) -> Result<()> {
// category is taken as it comes, unlike title and image: a show that leaves a category
// should leave the Directory's chip too.
let o = |v: Option<&str>| sea_orm::Value::from(v.map(str::to_owned));
self.exec(
"INSERT INTO feeds (id, url, title, etag, last_modified, last_checked, ttl_mins, last_error, image, category)
VALUES ($1, $2, $3, $4, $5, $6, $7, NULL, $8, $9)
"INSERT INTO feeds (id, url, title, etag, last_modified, last_checked, ttl_mins, last_error, image, category, description)
VALUES ($1, $2, $3, $4, $5, $6, $7, NULL, $8, $9, $10)
ON CONFLICT (id) DO UPDATE SET
url = excluded.url,
title = coalesce(excluded.title, feeds.title),
@@ -482,6 +518,7 @@ impl Db {
ttl_mins = excluded.ttl_mins,
image = coalesce(excluded.image, feeds.image),
category = excluded.category,
description = coalesce(excluded.description, feeds.description),
last_error = NULL,
error_since = NULL",
vec![
@@ -494,6 +531,7 @@ impl Db {
ttl_mins.map(|t| t as i64).into(),
o(image),
o(category),
o(description),
],
)
.await?;
@@ -652,6 +690,35 @@ impl Db {
/// through every show's history: 4420 files queued across 12 shows on the default of 3
/// (#97). Unlimited, 0 in the settings, is the whole back catalogue, for an archive. An item
/// whose files were all skipped by a filter does not hold one of the places.
/// Puts a feed's waiting files in the queue or out of it: 'pending' for those among its
/// `limit` newest items, which the scan downloads on its own, and 'held' for the rest, listed
/// to download by hand. With all of them 'pending', the queue counted 4420 back-catalogue
/// files no scan would ever take (#97, #113). 0 holds them all, for a feed nobody downloads
/// automatically. A held file still holds its item's place among the newest.
#[tracing::instrument(skip_all)]
pub async fn hold_back(&self, feed_id: &str, limit: usize) -> Result<()> {
let newest = "SELECT n.guid FROM entries n
WHERE n.feed_id = $1
AND EXISTS (SELECT 1 FROM enclosures y
WHERE y.feed_id = n.feed_id AND y.guid = n.guid
AND y.state <> 'skipped')
ORDER BY coalesce(n.published, n.first_seen) DESC, n.guid DESC
LIMIT $2";
let limit: sea_orm::Value = (limit.min(i64::MAX as usize) as i64).into();
self.exec(
&format!("UPDATE enclosures SET state = 'held' WHERE feed_id = $1 AND state = 'pending' AND guid NOT IN ({newest})"),
vec![feed_id.into(), limit.clone()],
)
.await?;
// And back, when a limit is raised or a feed downloads again.
self.exec(
&format!("UPDATE enclosures SET state = 'pending' WHERE feed_id = $1 AND state = 'held' AND guid IN ({newest})"),
vec![feed_id.into(), limit],
)
.await?;
Ok(())
}
#[tracing::instrument(skip_all)]
pub async fn pending(&self, feed_id: &str, limit: usize) -> Result<Vec<Pending>> {
self.rows(
@@ -778,13 +845,76 @@ impl Db {
.map(|r| Ok(r.try_get("", "id")?))
.collect::<Result<_>>()?;
for id in &gone {
for table in ["enclosures", "entry_state", "hidden", "blocklists", "entries"] {
self.exec(&format!("DELETE FROM {table} WHERE feed_id = $1"), vec![id.clone().into()]).await?;
}
self.forget_rows(id).await?;
}
Ok(gone)
}
/// Everything stored about a feed but its row: items, file rows, read state, block list.
async fn forget_rows(&self, id: &str) -> Result<()> {
for table in ["enclosures", "entry_state", "hidden", "blocklists", "entries"] {
self.exec(&format!("DELETE FROM {table} WHERE feed_id = $1"), vec![id.into()]).await?;
}
Ok(())
}
/// A feed's stored artwork still on http, items' and its own: hosts a feed no longer names
/// keep their old items' artwork on http otherwise (IGN's 2009 items, #110).
pub async fn http_images(&self, feed_id: &str) -> Result<Vec<String>> {
self.rows(
"SELECT DISTINCT image FROM entries WHERE feed_id = $1 AND image LIKE 'http://%'
UNION SELECT image FROM feeds WHERE id = $1 AND image LIKE 'http://%'",
vec![feed_id.into()],
)
.await?
.iter()
.map(|r| Ok(r.try_get("", "image")?))
.collect()
}
/// Rewrites a feed's stored artwork on `host` from http to https, once the host is known to
/// serve it there (`feed::prefer_https`), for the items a scan does not write again.
pub async fn secure_images(&self, feed_id: &str, host: &str) -> Result<()> {
let like = format!("http://{host}/%");
for table in ["entries", "feeds"] {
let col = if table == "feeds" { "id" } else { "feed_id" };
self.exec(
&format!("UPDATE {table} SET image = 'https://' || substr(image, 8) WHERE {col} = $1 AND image LIKE $2"),
vec![feed_id.into(), like.clone().into()],
)
.await?;
}
Ok(())
}
/// Forgets a feed and everything stored about it.
pub async fn forget_feed(&self, id: &str) -> Result<()> {
self.forget_rows(id).await?;
self.exec("DELETE FROM feeds WHERE id = $1", vec![id.into()]).await?;
Ok(())
}
/// Feeds nobody subscribes to that are dead, failing since before `dead_before`, or quiet,
/// their newest item from before `quiet_before`: what keeps the Directory to feeds worth
/// taking. One with a file on disk stays, and so does one an OPML subscription lists, which
/// comes back with the OPML anyway. A feed with no items yet is not quiet, only new.
pub async fn stale_unsubscribed(&self, dead_before: i64, quiet_before: i64) -> Result<Vec<String>> {
self.rows(
"SELECT f.id FROM feeds f
WHERE NOT EXISTS (SELECT 1 FROM subscriptions s WHERE s.feed_id = f.id)
AND NOT EXISTS (SELECT 1 FROM enclosures x WHERE x.feed_id = f.id AND x.path IS NOT NULL)
AND f.group_id IS NULL AND NOT coalesce(f.managed, false)
AND ((f.error_since IS NOT NULL AND f.error_since < $1)
OR (SELECT max(coalesce(e.published, e.first_seen)) FROM entries e
WHERE e.feed_id = f.id) < $2)",
vec![dead_before.into(), quiet_before.into()],
)
.await?
.iter()
.map(|r| Ok(r.try_get("", "id")?))
.collect()
}
pub async fn prune_entries(&self, older_than: i64) -> Result<usize> {
let n = self
.exec(
@@ -1585,6 +1715,7 @@ impl Db {
/// The foreign key would take them anyway; this does not rely on it being switched on.
pub async fn delete_user(&self, id: i64) -> Result<()> {
sessions::Entity::delete_many().filter(sessions::Column::UserId.eq(id)).exec(&self.orm).await?;
api_tokens::Entity::delete_many().filter(api_tokens::Column::UserId.eq(id)).exec(&self.orm).await?;
users::Entity::delete_by_id(id).exec(&self.orm).await?;
Ok(())
}
@@ -1637,6 +1768,56 @@ impl Db {
Ok(found.map(User::from))
}
/// Keeps a new API token's hash under its owner; the token itself is shown once and not kept.
pub async fn create_api_token(&self, user_id: i64, name: &str, hash: &str) -> Result<i64> {
let made = api_tokens::ActiveModel {
user_id: Set(user_id),
name: Set(name.to_owned()),
hash: Set(hash.to_owned()),
created: Set(now()),
last_used: Set(None),
..Default::default()
}
.insert(&self.orm)
.await?;
Ok(made.id)
}
/// One person's tokens, newest first: names and dates, as the tokens themselves are not kept.
pub async fn api_tokens(&self, user_id: i64) -> Result<Vec<api_tokens::Model>> {
Ok(api_tokens::Entity::find()
.filter(api_tokens::Column::UserId.eq(user_id))
.order_by_desc(api_tokens::Column::Id)
.all(&self.orm)
.await?)
}
/// Revokes one of this person's tokens; someone else's id does nothing. Whether it was theirs.
pub async fn delete_api_token(&self, user_id: i64, id: i64) -> Result<bool> {
let gone = api_tokens::Entity::delete_many()
.filter(api_tokens::Column::Id.eq(id))
.filter(api_tokens::Column::UserId.eq(user_id))
.exec(&self.orm)
.await?;
Ok(gone.rows_affected > 0)
}
/// The person an API token acts as, noting when it was used, as a session notes `seen`.
pub async fn api_token_user(&self, hash: &str) -> Result<Option<User>> {
let found = api_tokens::Entity::find()
.filter(api_tokens::Column::Hash.eq(hash))
.find_also_related(users::Entity)
.one(&self.orm)
.await?;
let Some((token, Some(user))) = found else { return Ok(None) };
api_tokens::Entity::update_many()
.col_expr(api_tokens::Column::LastUsed, Expr::val(now()).into())
.filter(api_tokens::Column::Id.eq(token.id))
.exec(&self.orm)
.await?;
Ok(Some(User::from(user)))
}
pub async fn delete_session(&self, token: &str) -> Result<()> {
sessions::Entity::delete_by_id(token.to_owned()).exec(&self.orm).await?;
Ok(())
@@ -1924,6 +2105,39 @@ pub fn now() -> i64 {
mod tests {
use super::*;
#[tokio::test]
async fn sqlite_syncs_at_checkpoints_not_every_commit() {
let db = Db::memory().await.unwrap();
if db.orm.get_database_backend() != sea_orm::DbBackend::Sqlite {
return;
}
let row = db.orm.query_one_raw(Statement::from_string(sea_orm::DbBackend::Sqlite, "PRAGMA synchronous")).await.unwrap();
assert_eq!(row.unwrap().try_get_by_index::<i32>(0).unwrap(), 1, "NORMAL, on every connection in the pool (#135)");
}
#[tokio::test]
async fn a_database_from_before_descriptions_gains_the_column_and_reads_each_feed_whole() {
let db = Db::memory().await.unwrap();
let record = |etag: &'static str, about: Option<&'static str>| {
let db = &db;
async move { db.record_feed("f", "https://x/f", Some("F"), Some(etag), Some("then"), None, None, None, about).await.unwrap() }
};
record("\"a\"", Some("About F")).await;
assert_eq!(db.feed_description("f").await.unwrap().as_deref(), Some("About F"));
record("\"a\"", None).await;
assert_eq!(db.feed_description("f").await.unwrap().as_deref(), Some("About F"), "kept when a read finds none");
// A database from before #130.
db.exec_for_test("ALTER TABLE feeds DROP COLUMN description").await.unwrap();
create_missing(&db.orm).await.unwrap();
assert_eq!(db.feed_description("f").await.unwrap(), None);
let st = db.http_state("f").await.unwrap();
assert!(st.etag.is_none() && st.last_modified.is_none(), "every feed is read whole next time, for its description");
// With the column there, opening the database again writes nothing.
record("\"b\"", Some("About F")).await;
create_missing(&db.orm).await.unwrap();
assert_eq!(db.http_state("f").await.unwrap().etag.as_deref(), Some("\"b\""));
}
#[tokio::test]
async fn every_sort_column_runs_and_orders_both_ways() {
let db = Db::memory().await.unwrap();
@@ -1998,6 +2212,52 @@ mod tests {
assert_eq!((list["f"].unread, list["g"].unread), (1, 1)); // a read, c hidden; sam's read is sam's
}
#[tokio::test]
async fn a_hosts_artwork_moves_to_https_for_one_feed() {
let db = Db::memory().await.unwrap();
db.exec_for_test(
"INSERT INTO feeds (id, url, image) VALUES ('f','u','http://a.example/logo.png'),('g','v','http://a.example/g.png');
INSERT INTO entries (feed_id, guid, first_seen, image) VALUES
('f','1',0,'http://a.example/1.jpg'),('f','2',0,'http://b.example/2.jpg'),('g','3',0,'http://a.example/3.jpg');",
).await
.unwrap();
let mut old = db.http_images("f").await.unwrap();
old.sort();
assert_eq!(old, ["http://a.example/1.jpg", "http://a.example/logo.png", "http://b.example/2.jpg"]);
db.secure_images("f", "a.example").await.unwrap();
assert_eq!(db.feed_summary("f").await.unwrap().image.as_deref(), Some("https://a.example/logo.png"));
assert!(db.names_image("https://a.example/1.jpg").await.unwrap());
assert!(db.names_image("http://b.example/2.jpg").await.unwrap()); // another host
assert!(db.names_image("http://a.example/3.jpg").await.unwrap()); // another feed
assert_eq!(db.feed_summary("g").await.unwrap().image.as_deref(), Some("http://a.example/g.png"));
}
#[tokio::test]
async fn stale_feeds_nobody_subscribes_to_are_found_and_the_rest_left() {
let db = Db::memory().await.unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true);
INSERT INTO feeds (id, url, error_since, managed, group_id) VALUES
('dead','a',100,false,null),('quiet','b',null,false,null),('news','c',null,false,null),
('new','d',null,false,null),('wanted','e',100,false,null),('kept','f',100,false,null),
('child','g',100,true,'opml'),('lately','h',900,false,null);
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'wanted');
INSERT INTO entries (feed_id, guid, first_seen, published) VALUES
('quiet','q',0,100),('news','n',0,950),('dead','d',0,950),('kept','k',0,100);
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES (1,'kept','k','u','/tmp/k','done');",
).await
.unwrap();
// Dead before 500, or nothing newer than 500: dead fails since 100, quiet's newest is 100.
// news published lately, new has no items yet, wanted has a subscriber, kept a file,
// child comes from an OPML, lately began failing after the cutoff.
let mut stale = db.stale_unsubscribed(500, 500).await.unwrap();
stale.sort();
assert_eq!(stale, ["dead", "quiet"]);
db.forget_feed("dead").await.unwrap();
assert_eq!(db.feed_summary("dead").await.unwrap().entries, 0);
assert!(!db.feed_urls().await.unwrap().iter().any(|(id, _)| id == "dead"));
}
#[tokio::test]
async fn a_failing_feed_nobody_subscribes_to_is_forgotten() {
let db = Db::memory().await.unwrap();
@@ -2021,6 +2281,33 @@ mod tests {
assert!(db.prune_abandoned_failures().await.unwrap().is_empty());
}
#[tokio::test]
async fn only_the_newest_files_stay_queued_and_the_rest_are_held() {
let db = Db::memory().await.unwrap();
db.exec_for_test(
"INSERT INTO entries (feed_id, guid, first_seen, published) VALUES
('f','e1',0,100),('f','e2',0,200),('f','e3',0,300),('f','e4',0,400);
INSERT INTO enclosures (id, feed_id, guid, url, state, path) VALUES
(1,'f','e1','u1','pending',null),(2,'f','e2','u2','pending',null),
(3,'f','e3','u3','done','/tmp/3'),(4,'f','e4','u4','pending',null);",
).await
.unwrap();
let states = || async {
let mut v: Vec<(i64, String)> = db.rows("SELECT id, state FROM enclosures ORDER BY id", vec![]).await.unwrap()
.iter().map(|r| (r.try_get("", "id").unwrap(), r.try_get("", "state").unwrap())).collect();
v.sort();
v.into_iter().map(|(_, s)| s).collect::<Vec<_>>()
};
db.hold_back("f", 2).await.unwrap(); // newest two: e4, e3
assert_eq!(states().await, ["held", "held", "done", "pending"]);
db.hold_back("f", 3).await.unwrap(); // a limit raised brings e2 back
assert_eq!(states().await, ["held", "pending", "done", "pending"]);
db.hold_back("f", 0).await.unwrap(); // a feed nobody downloads: all held
assert_eq!(states().await, ["held", "held", "done", "held"]);
db.hold_back("f", usize::MAX).await.unwrap(); // unlimited, an archive: all queued
assert_eq!(states().await, ["pending", "pending", "done", "pending"]);
}
#[tokio::test]
async fn a_limit_takes_the_newest_episodes_not_the_back_catalogue() {
let db = Db::memory().await.unwrap();

View File

@@ -406,7 +406,7 @@ mod tests {
let mut f = crate::config::Feed {
url: "u".into(), folder: Some("Subscriptions/Some | Show".into()), group: None, media_types: None,
schedule: None, keywords: vec![], allow_explicit: false, auto_download: true,
max_new_per_check: None, username: None, password: None, password_env: None, category: None,
max_new_per_check: None, username: None, password: None, password_env: None, category: None, listed: false,
};
assert_eq!(folder_for(&cfg, "id", &f, None), "Subscriptions/Some - Show");

View File

@@ -19,6 +19,10 @@ pub mod feeds {
/// The channel's first <itunes:category>, for the Directory.
#[sea_orm(column_type = "Text", nullable)]
pub category: Option<String>,
/// What the feed says it is, as it says it, for its page in the Directory. Added after
/// the table was, so `create_missing` adds it to a database without it.
#[sea_orm(column_type = "Text", nullable)]
pub description: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub etag: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
@@ -102,6 +106,10 @@ pub mod enclosures {
pub length: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub path: Option<String>,
/// 'pending' is queued: a scan downloads it on its own. 'held' is listed but outside the
/// feed's newest max_new_per_check, or its feed downloads nothing automatically; it can
/// be downloaded by hand (`Db::hold_back`). 'skipped' is filtered out ('last_error' says
/// why), then 'downloading', 'done', 'error', 'reaped'.
#[sea_orm(column_type = "Text")]
pub state: String,
#[sea_orm(default_value = 0)]
@@ -283,6 +291,29 @@ pub mod sessions {
owned_by_user!();
}
/// A token a script or agent sends as `Authorization: Bearer`, acting as the person who made
/// it (#123). Kept as its SHA-256, so the table is no use to anyone who reads it.
pub mod api_tokens {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "api_tokens")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i64,
pub user_id: i64,
/// What its owner called it, to tell one from another when revoking.
#[sea_orm(column_type = "Text")]
pub name: String,
#[sea_orm(unique, column_type = "Text")]
pub hash: String,
pub created: i64,
pub last_used: Option<i64>,
}
owned_by_user!();
}
/// The catalogue: every feed configured, with its shared settings as `config::Feed` in JSON, so a
/// new setting on a feed needs no new column. It was config.toml's `[feeds]` (issue #18).
pub mod catalogue {

View File

@@ -13,8 +13,11 @@ pub struct ParsedFeed {
pub image: Option<String>,
/// The site the feed belongs to, where a favicon can stand in for missing artwork.
pub site: Option<String>,
/// The channel's first `<itunes:category>`, for the Directory's chips.
/// The channel's first `<itunes:category>`, for the Directory's categories.
pub category: Option<String>,
/// What the feed says it is, for its page in the Directory (#130): RSS's `<description>`,
/// or iTunes' summary where that is empty, or Atom's subtitle. HTML as often as not.
pub description: Option<String>,
pub entries: Vec<Entry>,
}
@@ -53,43 +56,72 @@ pub enum Fetched {
},
}
/// Conditional GET. reqwest handles gzip and redirects; the original's hand-rolled
/// CONNECT/socket.ssl proxy path is gone -- `system-proxy` reads http_proxy/https_proxy.
/// The longest a feed may take, connecting to the last byte: a scan handles feeds in order, and
/// with no limit one hung server held every scan for as long as it did. Dreamwidth answered 504
/// after 60-67 s for a day, and each scan took 70-80 s instead of 15 (#108). A feed is small;
/// downloads, which are not, have no such limit.
pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
/// Conditional GET, following redirects itself: `client` must follow none (`feed_client`), so
/// each hop is seen. The second value is where the feed now is when every hop said so for good
/// (301 or 308): a publisher that moved its feed, which the catalogue should follow rather
/// than be redirected on every read. A temporary redirect (302, 307) moves nothing.
/// `system-proxy` reads http_proxy/https_proxy.
#[tracing::instrument(skip_all, fields(url = %cfg.url))]
pub async fn fetch(
client: &reqwest::Client,
cfg: &FeedCfg,
etag: Option<&str>,
last_modified: Option<&str>,
) -> Result<Fetched> {
let mut req = client.get(&cfg.url);
if let Some(tag) = etag {
req = req.header(IF_NONE_MATCH, tag);
}
if let Some(lm) = last_modified {
req = req.header(IF_MODIFIED_SINCE, lm);
}
if let Some(user) = &cfg.username {
req = req.basic_auth(user, cfg.password());
}
) -> Result<(Fetched, Option<String>)> {
let start = reqwest::Url::parse(&cfg.url).context("the feed's address")?;
let mut url = start.clone();
let mut permanent = true;
for _ in 0..10 {
let mut req = client.get(url.clone()).timeout(FEED_TIMEOUT);
if let Some(tag) = etag {
req = req.header(IF_NONE_MATCH, tag);
}
if let Some(lm) = last_modified {
req = req.header(IF_MODIFIED_SINCE, lm);
}
// The feed's own host only: a redirect elsewhere must not be handed the password.
if let Some(user) = &cfg.username
&& url.host_str() == start.host_str()
{
req = req.basic_auth(user, cfg.password());
}
let resp = req.send().await.context("connecting")?;
if resp.status() == StatusCode::NOT_MODIFIED {
return Ok(Fetched::NotModified);
let resp = req.send().await.context("connecting")?;
let status = resp.status();
if status.is_redirection() && status != StatusCode::NOT_MODIFIED {
let to = resp
.headers()
.get(reqwest::header::LOCATION)
.and_then(|v| v.to_str().ok())
.ok_or_else(|| anyhow!("HTTP {status} without a Location to go to"))?;
url = url.join(to).with_context(|| format!("redirected to {to:?}, which is not an address"))?;
permanent &= matches!(status, StatusCode::MOVED_PERMANENTLY | StatusCode::PERMANENT_REDIRECT);
continue;
}
let moved = (permanent && url != start).then(|| url.to_string());
if status == StatusCode::NOT_MODIFIED {
return Ok((Fetched::NotModified, moved));
}
if !status.is_success() {
// The original surfaced 401/407 specially; the code is enough for a UI to switch on.
return Err(anyhow!("HTTP {status}"));
}
let header = |h: reqwest::header::HeaderName| {
resp.headers().get(&h).and_then(|v| v.to_str().ok()).map(str::to_owned)
};
let etag = header(ETAG);
let last_modified = header(LAST_MODIFIED);
let bytes = resp.bytes().await.context("reading body")?.to_vec();
return Ok((Fetched::Body { bytes, etag, last_modified }, moved));
}
let status = resp.status();
if !status.is_success() {
// The original surfaced 401/407 specially; the code is enough for a UI to switch on.
return Err(anyhow!("HTTP {status}"));
}
let header = |h: reqwest::header::HeaderName| {
resp.headers().get(&h).and_then(|v| v.to_str().ok()).map(str::to_owned)
};
let etag = header(ETAG);
let last_modified = header(LAST_MODIFIED);
let bytes = resp.bytes().await.context("reading body")?.to_vec();
Ok(Fetched::Body { bytes, etag, last_modified })
// Worded as reqwest worded it, which failure_kind reads as a redirect loop.
Err(anyhow!("error following redirect for url ({url}): too many redirects"))
}
/// A stored `last_error`, translated into plain words for whoever subscribes: whose problem
@@ -253,7 +285,7 @@ pub async fn patreon_shows(
) -> Result<(Option<String>, Vec<(String, String)>)> {
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
let mut resp = client.get(url).send().await.context("connecting")?;
let mut resp = client.get(url).timeout(FEED_TIMEOUT).send().await.context("connecting")?;
if !resp.status().is_success() {
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
}
@@ -451,6 +483,30 @@ pub async fn site_icon(client: &reqwest::Client, site: &str) -> Option<String> {
is_image(client, ico.as_str()).await.then(|| ico.into())
}
/// Artwork's address on https when its host serves it there, else as it was. The page is https
/// and must not load http; the host is asked once per `known` (one feed's read), and an http
/// address it does not serve on https stays, for /api/art to fetch (#90). Four of the five
/// hosts the catalogue had on http served the same image on https; The Secret Cabal's CDN
/// presents another name's certificate (#110).
pub async fn prefer_https(
client: &reqwest::Client,
url: &str,
known: &mut std::collections::HashMap<String, bool>,
) -> String {
let Some(rest) = url.strip_prefix("http://") else { return url.to_owned() };
let host = rest.split('/').next().unwrap_or("").to_owned();
let secure = format!("https://{rest}");
let ok = match known.get(&host) {
Some(ok) => *ok,
None => {
let ok = is_image(client, &secure).await;
known.insert(host, ok);
ok
}
};
if ok { secure } else { url.to_owned() }
}
/// Whether `url` answers with an image. A site with no favicon often answers 200 with its home
/// page, which is not an icon.
pub async fn is_image(client: &reqwest::Client, url: &str) -> bool {
@@ -681,6 +737,7 @@ fn from_rss(ch: rss::Channel, bytes: &[u8]) -> ParsedFeed {
// Only the iTunes one: Apple's list is fixed, while a plain <category> is freeform and
// would fill the Directory with one-off tags. The subcategory where there is one: Apple
// files every tabletop and gaming show under Leisure, which says little; Games says it.
description: non_empty(Some(ch.description())).or_else(|| non_empty(ch.itunes_ext().and_then(|i| i.summary()))),
category: ch
.itunes_ext()
.and_then(|i| i.categories().first())
@@ -750,6 +807,7 @@ fn from_atom(feed: atom_syndication::Feed) -> ParsedFeed {
.map(|l| l.href().trim().to_owned())
.filter(|h| !h.is_empty()),
category: None,
description: non_empty(feed.subtitle().map(|t| t.as_str())),
entries,
}
}
@@ -955,8 +1013,151 @@ fn parse_date(s: &str) -> Option<i64> {
.map(|d| d.timestamp())
}
/// Apple's podcast categories, each with its subcategories
/// (https://podcasters.apple.com/support/1691-apple-podcasts-categories): the list a podcast's own
/// `<itunes:category>` comes from, and a fixed one, so the Directory's categories stay few.
const CATEGORIES: &[(&str, &[&str])] = &[
("Arts", &["Books", "Design", "Fashion & Beauty", "Food", "Performing Arts", "Visual Arts"]),
("Business", &["Careers", "Entrepreneurship", "Investing", "Management", "Marketing", "Non-Profit"]),
("Comedy", &["Comedy Interviews", "Improv", "Stand-Up"]),
("Education", &["Courses", "How To", "Language Learning", "Self-Improvement"]),
("Fiction", &["Comedy Fiction", "Drama", "Science Fiction"]),
("Government", &[]),
("History", &[]),
("Health & Fitness", &["Alternative Health", "Fitness", "Medicine", "Mental Health", "Nutrition", "Sexuality"]),
("Kids & Family", &["Education for Kids", "Parenting", "Pets & Animals", "Stories for Kids"]),
("Leisure", &["Animation & Manga", "Automotive", "Aviation", "Crafts", "Games", "Hobbies", "Home & Garden", "Video Games"]),
("Music", &["Music Commentary", "Music History", "Music Interviews"]),
("News", &["Business News", "Daily News", "Entertainment News", "News Commentary", "Politics", "Sports News", "Tech News"]),
("Religion & Spirituality", &["Buddhism", "Christianity", "Hinduism", "Islam", "Judaism", "Religion", "Spirituality"]),
("Science", &["Astronomy", "Chemistry", "Earth Sciences", "Life Sciences", "Mathematics", "Natural Sciences", "Nature", "Physics", "Social Sciences"]),
("Society & Culture", &["Documentary", "Personal Journals", "Philosophy", "Places & Travel", "Relationships"]),
("Sports", &["Baseball", "Basketball", "Cricket", "Fantasy Sports", "Football", "Golf", "Hockey", "Rugby", "Running", "Soccer", "Swimming", "Tennis", "Volleyball", "Wilderness", "Wrestling"]),
("Technology", &[]),
("True Crime", &[]),
("TV & Film", &["After Shows", "Film History", "Film Interviews", "Film Reviews", "TV Reviews"]),
];
/// Names that are not in Apple's list but mean one that is (#125). Jev picked from this list when
/// it spelt Football and Soccer its own way, and stored those answers; Software How-To is Apple's
/// Technology subcategory from before its 2019 list, which older feeds still name.
const ALIASES: &[(&str, &str)] =
&[("American Football", "Football"), ("Football (Soccer)", "Soccer"), ("Software How-To", "Technology")];
/// Where a category sits in Apple's list, as (category, subcategory), so the Directory browses a
/// category and then its subcategories as Apple's does (#118). A podcast's own is stored as its
/// subcategory, and one an admin typed that is not Apple's stands as a category of its own.
pub fn category_path(name: &str) -> (String, Option<String>) {
// A feed's own text, which can end in a stray comma: "Soccer," was a category of its own.
let name = name.trim().trim_end_matches([',', ';', '.']).trim_end();
let name = ALIASES.iter().find(|(old, _)| old.eq_ignore_ascii_case(name)).map_or(name, |(_, now)| now);
for (category, subs) in CATEGORIES {
if category.eq_ignore_ascii_case(name) {
return (category.to_string(), None);
}
if let Some(sub) = subs.iter().find(|s| s.eq_ignore_ascii_case(name)) {
return (category.to_string(), Some(sub.to_string()));
}
}
(name.to_string(), None)
}
/// A Directory category for a feed that names none of its own, which most blogs do not (#117):
/// TypeSafe's Jev picks one of Apple's categories or subcategories from the feed's title and
/// latest items, the subcategory being what a podcast's own category is stored as. The likeliest
/// is taken however unsure, since an admin can change it and asking again would cost a call a scan.
pub async fn classify(client: &reqwest::Client, key: &str, feed: &ParsedFeed) -> Result<String> {
let items: Vec<&str> = feed.entries.iter().filter_map(|e| e.title.as_deref()).take(15).collect();
// A subcategory says which category it is in; without that, "Games" or "Drama" is ambiguous.
let mut criteria = serde_json::Map::new();
for (category, subs) in CATEGORIES {
criteria.insert(category.to_string(), serde_json::Value::Null);
for sub in *subs {
criteria.insert(sub.to_string(), format!("Within {category}").into());
}
}
let body = serde_json::json!({
"model": "jev-latest",
"state": { "title": feed.title, "recent_items": items },
"questions": { "category": {
"type": "choice",
"instructions": "Which category best describes what this feed publishes?",
"criteria": criteria,
}},
});
let answer: serde_json::Value = client
.post("https://api.typesafe.ai/v1/systemone")
.bearer_auth(key)
.json(&body)
.timeout(FEED_TIMEOUT)
.send()
.await?
.error_for_status()?
.json()
.await?;
picked(&answer)
}
fn picked(answer: &serde_json::Value) -> Result<String> {
let choice = answer["answers"]["category"]["choice"].as_str().unwrap_or_default();
CATEGORIES
.iter()
.flat_map(|(c, subs)| std::iter::once(c).chain(subs.iter()))
.find(|c| **c == choice)
.map(|c| c.to_string())
.ok_or_else(|| anyhow!("no category in the answer: {answer}"))
}
#[cfg(test)]
mod tests {
/// A server answering by path: /old moves for good to /new, /tmp for now, /chain for good
/// to /tmp, /new is the feed.
async fn redirecting_server() -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
loop {
let Ok((mut sock, _)) = listener.accept().await else { return };
tokio::spawn(async move {
let mut buf = [0u8; 2048];
let n = sock.read(&mut buf).await.unwrap_or(0);
let req = String::from_utf8_lossy(&buf[..n]);
let path = req.split_whitespace().nth(1).unwrap_or("/").to_owned();
let body = "<?xml version=\"1.0\"?><rss version=\"2.0\"><channel><title>T</title></channel></rss>";
let resp = match path.as_str() {
"/old" => "HTTP/1.1 301 Moved Permanently\r\nLocation: /new\r\nContent-Length: 0\r\n\r\n".to_owned(),
"/tmp" => "HTTP/1.1 302 Found\r\nLocation: /new\r\nContent-Length: 0\r\n\r\n".to_owned(),
"/chain" => "HTTP/1.1 308 Permanent Redirect\r\nLocation: /tmp\r\nContent-Length: 0\r\n\r\n".to_owned(),
"/loop" => "HTTP/1.1 301 Moved Permanently\r\nLocation: /loop\r\nContent-Length: 0\r\n\r\n".to_owned(),
_ => format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n{body}", body.len()),
};
let _ = sock.write_all(resp.as_bytes()).await;
});
}
});
format!("http://{addr}")
}
#[tokio::test]
async fn a_feed_that_moved_for_good_says_where_and_one_moved_for_now_does_not() {
let base = redirecting_server().await;
let client = reqwest::Client::builder().redirect(reqwest::redirect::Policy::none()).build().unwrap();
let get = |path: &str| {
let cfg: crate::config::Feed = serde_json::from_value(serde_json::json!({ "url": format!("{base}{path}") })).unwrap();
let client = client.clone();
async move { super::fetch(&client, &cfg, None, None).await }
};
let (got, moved) = get("/old").await.unwrap();
assert!(matches!(got, super::Fetched::Body { .. }));
assert_eq!(moved, Some(format!("{base}/new")));
assert_eq!(get("/tmp").await.unwrap().1, None); // 302: for now
assert_eq!(get("/chain").await.unwrap().1, None); // 308 then 302: not for good
assert_eq!(get("/new").await.unwrap().1, None); // never moved
let looped = get("/loop").await.err().unwrap().to_string();
assert_eq!(super::failure_kind(&looped).0, "redirect_loop");
}
use super::*;
#[test]
@@ -967,6 +1168,7 @@ mod tests {
assert_eq!(feed.title.as_deref(), Some("Test Cast"));
assert_eq!(feed.ttl_mins, Some(45));
assert_eq!(feed.category.as_deref(), Some("Podcasting"), "the first, by its subcategory");
assert_eq!(feed.description.as_deref(), Some("A synthetic feed used by the parser tests."));
assert_eq!(feed.entries.len(), 3);
let ep = &feed.entries[0];
@@ -991,6 +1193,17 @@ mod tests {
);
}
#[test]
fn a_feed_says_what_it_is_in_itunes_summary_or_atom_subtitle() {
// An empty <description> gives way to iTunes' summary (#130).
let rss = br#"<?xml version="1.0"?><rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd">
<channel><title>R</title><description> </description><itunes:summary>Said here.</itunes:summary></channel></rss>"#;
assert_eq!(parse(rss).unwrap().description.as_deref(), Some("Said here."));
let atom = br#"<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom"><title>A</title>
<subtitle>Or here.</subtitle><id>urn:a</id><updated>2026-01-01T00:00:00Z</updated></feed>"#;
assert_eq!(parse(atom).unwrap().description.as_deref(), Some("Or here."));
}
#[test]
fn a_file_wordpress_lists_twice_is_one_enclosure() {
let xml = br#"<?xml version="1.0"?><rss version="2.0"><channel><title>R</title>
@@ -1390,5 +1603,29 @@ mod tests {
assert_eq!(pick_guid(None, None, Some("e"), Some("t")).as_deref(), Some("e"));
assert_eq!(pick_guid(None, None, None, None), None);
}
#[test]
fn picked_takes_only_an_apple_category() {
let answer = |c: &str| serde_json::json!({"answers": {"category": {"type": "choice", "choice": c}}});
assert_eq!(super::picked(&answer("Technology")).unwrap(), "Technology");
assert_eq!(super::picked(&answer("Tech News")).unwrap(), "Tech News", "a subcategory");
assert!(super::picked(&answer("Blogs")).is_err(), "not one of Apple's");
assert!(super::picked(&serde_json::json!({"error": "rate limited"})).is_err());
}
#[test]
fn category_path_finds_a_subcategory_s_category() {
use super::category_path;
assert_eq!(category_path("Tech News"), ("News".into(), Some("Tech News".into())));
assert_eq!(category_path("video games"), ("Leisure".into(), Some("Video Games".into())), "Apple's spelling");
assert_eq!(category_path("Technology"), ("Technology".into(), None));
assert_eq!(category_path("Homebrew"), ("Homebrew".into(), None), "not Apple's: its own category");
// Apple's spelling, an old answer of Jev's, a stray comma and Apple's old list (#125).
assert_eq!(category_path("Soccer"), ("Sports".into(), Some("Soccer".into())));
assert_eq!(category_path("Football (Soccer)"), ("Sports".into(), Some("Soccer".into())));
assert_eq!(category_path("Soccer,"), ("Sports".into(), Some("Soccer".into())));
assert_eq!(category_path("Football"), ("Sports".into(), Some("Football".into())));
assert_eq!(category_path("Software How-To"), ("Technology".into(), None));
}
}

View File

@@ -15,6 +15,9 @@ pub enum Event {
FeedSkip { feed: String, reason: String },
FeedDone { feed: String, new: usize, downloaded: usize, failed: usize, torrents: usize },
FeedError { feed: String, msg: String },
/// The feed answered from a new address after redirects that all said it moved for good,
/// and the catalogue now has that address (#115).
FeedMoved { feed: String, from: String, to: String },
Progress {
feed: String,
/// Which enclosure this is about. Without it a UI cannot tell one download's
@@ -50,6 +53,7 @@ impl Event {
"{feed}: {new} new entries, {downloaded} downloaded, {failed} failed, {torrents} torrents deferred"
),
Event::FeedError { feed, msg } => format!("{feed}: error: {msg}"),
Event::FeedMoved { feed, from, to } => format!("{feed}: moved for good from {from} to {to}; following it"),
Event::Progress { file, done, total, .. } => match total {
Some(t) if *t > 0 => format!(
" {file}: {:.1}% ({:.1}/{:.1} MB)",
@@ -69,7 +73,7 @@ impl Event {
*bytes as f64 / 1_048_576.0
),
Event::Status { feeds, pending, downloaded } => {
format!("{feeds} feeds, {pending} pending, {downloaded} downloaded")
format!("{feeds} feeds, {pending} queued to download, {downloaded} downloaded")
}
Event::Error { msg } => format!("error: {msg}"),
// Noise in a terminal; a UI still gets them on the socket.
@@ -172,7 +176,7 @@ fn log_event(e: &Event, wire: bool) {
($level:ident, $target:literal, $text:expr) => {
tracing::$level!(
target: $target,
ev = s("ev"), feed = s("feed"), msg = s("msg"), url = s("url"), reason = s("reason"),
ev = s("ev"), feed = s("feed"), msg = s("msg"), url = s("url"), reason = s("reason"), from = s("from"), to = s("to"),
new = n("new"), downloaded = n("downloaded"), failed = n("failed"),
torrents = n("torrents"), bytes = n("bytes"), feeds = n("feeds"),
pending = n("pending"), enclosure = n("enclosure"), files = n("files"),
@@ -328,6 +332,14 @@ pub async fn proxy(path: &Path, cmd: &Command) -> Result<()> {
mod tests {
use super::*;
#[test]
fn a_feed_that_moved_says_so_on_the_wire_and_in_words() {
let e = Event::FeedMoved { feed: "x".into(), from: "http://a/f".into(), to: "https://a/f".into() };
let wire = serde_json::to_value(&e).unwrap();
assert_eq!((wire["ev"].as_str(), wire["from"].as_str(), wire["to"].as_str()), (Some("feed_moved"), Some("http://a/f"), Some("https://a/f")));
assert_eq!(e.human().unwrap(), "x: moved for good from http://a/f to https://a/f; following it");
}
#[test]
fn commands_parse_from_the_wire_form() {
let got: Command = serde_json::from_str(r#"{"cmd":"fetch"}"#).unwrap();

View File

@@ -1,4 +1,5 @@
mod access;
mod art;
mod auth;
mod config;
mod db;
@@ -63,6 +64,12 @@ enum Command {
/// Only take enclosures matching these keywords
#[arg(long, value_delimiter = ',')]
keywords: Vec<String>,
/// The Directory's category, for a feed that names none of its own (News, Technology)
#[arg(long)]
category: Option<String>,
/// Put it in the Directory for anyone to subscribe to, and keep it there when nobody does
#[arg(long)]
list: bool,
},
/// Unsubscribe. Downloads and history are left alone.
Rm { feed: String },
@@ -118,6 +125,9 @@ pub struct Ctx {
pub cfg: std::sync::RwLock<std::sync::Arc<config::Config>>,
pub db: db::Db,
pub client: reqwest::Client,
/// For feeds only: follows no redirects, so `feed::fetch` sees each hop and can tell a feed
/// that moved for good from one sent elsewhere for now.
pub feed_client: reqwest::Client,
pub out: Emitter,
/// Started on first use: a BitTorrent session binds ports and starts a DHT, which is
/// rude to do for a config that has never seen a torrent.
@@ -248,6 +258,13 @@ async fn main() -> Result<()> {
db,
client: reqwest::Client::builder()
.user_agent(concat!("ipx/", env!("CARGO_PKG_VERSION")))
// Connecting only, so it bounds a download's start, not a long download (#108).
.connect_timeout(std::time::Duration::from_secs(10))
.build()?,
feed_client: reqwest::Client::builder()
.user_agent(concat!("ipx/", env!("CARGO_PKG_VERSION")))
.connect_timeout(std::time::Duration::from_secs(10))
.redirect(reqwest::redirect::Policy::none())
.build()?,
out: if is_daemon { Emitter::socket(events.clone(), false) } else { Emitter::terminal() },
torrents: tokio::sync::OnceCell::new(),
@@ -259,8 +276,8 @@ async fn main() -> Result<()> {
let result = match cli.command {
Command::List => list(&ctx).await,
Command::Daemon { web } => daemon(ctx, config_path, web, events).await,
Command::Add { url, folder, keywords } => {
add(&ctx, &url, folder, keywords).await
Command::Add { url, folder, keywords, category, list } => {
add(&ctx, &url, folder, keywords, category, list).await
}
Command::Rm { feed } => rm(&ctx, &feed).await,
Command::User { cmd } => user_cmd(&ctx, cmd).await,
@@ -505,8 +522,6 @@ async fn daemon(
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
// One command at a time: the queue is what keeps two scans from overlapping.
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
tracing::info!(
feeds = subscriptions(&ctx).await.map(|s| s.len()).unwrap_or(0),
"daemon started"
@@ -543,10 +558,13 @@ async fn daemon(
}
let mut stop = rx_stop.clone();
// The first pass at once, as the minute's tick did: what came due while it was down.
let mut first = true;
loop {
if *stop.borrow() {
break;
}
let wait = if std::mem::take(&mut first) { std::time::Duration::ZERO } else { until_next_scan(&ctx).await };
tokio::select! {
biased;
_ = stop.changed() => break,
@@ -563,7 +581,7 @@ async fn daemon(
break;
}
}
_ = ticker.tick() => {
_ = tokio::time::sleep(wait) => {
// Per-feed schedule and TTL decide what actually gets polled.
let job = run(&ctx, Cmd::Fetch { feed: None, force: false, feeds: vec![] });
if !until_stopped(&ctx, &rx_stop, job).await {
@@ -673,16 +691,35 @@ async fn add(
url: &str,
folder: Option<String>,
keywords: Vec<String>,
category: Option<String>,
list: bool,
) -> Result<()> {
let mut cfg = (*ctx.cfg()).clone();
let url = &feed::find_feed(&ctx.client, &feed::expand_input(url)).await?;
// Includes feeds derived from an OPML, or the same show could be added twice.
if let Some(existing) = subscriptions(ctx).await?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
// Listing a feed the catalogue already has, or giving it a category, is the point of
// asking again: it changes those, and nothing else.
if let Some(f) = cfg.feeds.get_mut(&existing.id)
&& (list || category.is_some())
{
f.listed |= list;
if category.is_some() {
f.category = category;
}
ctx.store_cfg(cfg).await?;
println!("{} is already in the catalogue; updated its Directory listing", existing.id);
return Ok(());
}
anyhow::bail!("already subscribed as {:?}", existing.id);
}
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
if let Some(f) = cfg.feeds.get_mut(&id) {
f.category = category;
f.listed = list;
}
ctx.store_cfg(cfg).await?;
println!("added {id}");
println!("added {id}{}", if list { ", listed in the Directory" } else { "" });
Ok(())
}
@@ -709,9 +746,10 @@ pub async fn add_one(
password: None,
password_env: None,
category: None,
listed: false,
};
let title = match feed::fetch(&ctx.client, &probe, None, None).await {
let title = match feed::fetch(&ctx.feed_client, &probe, None, None).await.map(|(got, _)| got) {
// An OPML subscription is named from its own <head><title>, not by trying to
// parse it as a feed and falling back to the hostname.
Ok(feed::Fetched::Body { bytes, .. }) if feed::is_opml(&bytes) => {
@@ -835,6 +873,7 @@ pub async fn subscribe_opml(
password: None,
password_env: None,
category: None,
listed: false,
},
);
grew = true;
@@ -947,12 +986,42 @@ async fn list(ctx: &Ctx) -> Result<()> {
Ok(())
}
/// Removes feeds nobody subscribes to that are dead (failing for 30 days) or quiet (nothing new
/// in a year), from the catalogue and the database, so the Directory lists feeds worth taking.
/// A feed listed in it with no subscribers stays for as long as it works and publishes.
async fn clean_directory(ctx: &Ctx) -> Result<()> {
const DEAD: i64 = 30 * 86_400;
const QUIET: i64 = 365 * 86_400;
let now = db::now();
let stale = ctx.db.stale_unsubscribed(now - DEAD, now - QUIET).await?;
if stale.is_empty() {
return Ok(());
}
let mut cfg = (*ctx.cfg()).clone();
if stale.iter().fold(false, |any, id| cfg.feeds.remove(id).is_some() || any) {
ctx.store_cfg(cfg).await?;
}
for id in &stale {
retire_group(ctx, id).await?;
ctx.db.forget_feed(id).await?;
tracing::info!(feed = id, "removed a feed nobody subscribes to that is dead or has published nothing in a year");
}
Ok(())
}
/// `standalone` false means this is the sweep that runs before a scan: it reports what it
/// deleted, but must not emit the terminal ReapDone, or a client waiting on its `fetch`
/// would stop reading before the scan had even started.
#[tracing::instrument(name = "reap", skip_all, fields(dry_run = dry_run))]
async fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
let r = retention::run(&ctx.cfg(), &ctx.db, dry_run).await?;
if !dry_run {
clean_directory(ctx).await?;
let gone = art::trim(&art::dir(), ctx.cfg().general.art_cache_mb * 1_048_576);
if gone > 0 {
tracing::debug!(gone, "trimmed the artwork kept on disk");
}
}
for c in r.aged_out.iter().chain(r.over_quota.iter()) {
ctx.out.emit(Event::Reaped {
path: c.path.clone(),
@@ -979,15 +1048,17 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
anyhow::bail!("no feed with id {id:?}");
}
let subscribed = ctx.db.subscriber_counts().await?;
let mut scanned = 0;
let mut fresh: Vec<String> = vec![];
let in_scope = |s: &Sub| {
scope.is_empty() || scope.contains(&s.id) || s.cfg.group.as_ref().is_some_and(|g| scope.contains(g))
};
let mut states = ctx.db.http_states().await?;
let mut due = vec![];
for sub in subs.iter().filter(|s| only.is_none_or(|o| o == s.id) && in_scope(s)) {
let id = &sub.id;
let mut state = ctx.db.http_state(id).await?;
let mut state = states.remove(id).unwrap_or_default();
// A scan someone asked for reads the feed in full. With the validators it only skipped the
// wait: a feed that had not changed answered 304 and nothing was read (#77).
if force {
@@ -995,8 +1066,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
state.last_modified = None;
}
if !force && let Some(last) = state.last_checked {
let at = last + due_after(&cfg, &sub.cfg, state.ttl_mins, state.error_since, last) as i64;
if !force && let Some(at) = due_at(&cfg, &sub.cfg, &state, subscribed.contains_key(id)) {
if at > db::now() {
ctx.out.emit(Event::FeedSkip {
feed: id.clone(),
@@ -1015,7 +1085,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
const AHEAD: usize = 6;
let mut bodies = futures_util::StreamExt::buffered(
futures_util::stream::iter(due.iter().map(|(sub, state)| {
let (client, feed_cfg) = (ctx.client.clone(), sub.cfg.clone());
let (client, feed_cfg) = (ctx.feed_client.clone(), sub.cfg.clone());
let (etag, modified) = (state.etag.clone(), state.last_modified.clone());
let early = !feed::is_patreon_creator(&feed_cfg.url);
tokio::spawn(tracing::Instrument::instrument(
@@ -1156,6 +1226,7 @@ pub async fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
password: parent.and_then(|p| p.password.clone()),
password_env: parent.and_then(|p| p.password_env.clone()),
category: None,
listed: false,
},
managed: true,
});
@@ -1206,6 +1277,47 @@ async fn retire_stranded(ctx: &Ctx) -> Result<usize> {
/// Seconds to wait before re-checking a feed.
///
/// When a feed is next due, or None for one never checked, which is due now. A feed nobody
/// subscribes to, one listed in the Directory, is read once a day: enough to keep its entry
/// current, without fetching it hourly for no one.
pub fn due_at(cfg: &config::Config, feed: &config::Feed, state: &db::HttpState, subscribed: bool) -> Option<i64> {
let last = state.last_checked?;
let floor = if subscribed { 0 } else { 86_400 };
Some(last + due_after(cfg, feed, state.ttl_mins, state.error_since, last).max(floor) as i64)
}
/// How long the daemon may sleep before a feed is due: until the earliest one, at least 30 s and
/// at most 10 minutes. It ticked every minute and ran a scan pass each time, 80% of them finding
/// nothing due (#114). The floor keeps a feed that never gets a check time from spinning it; the
/// ceiling picks up within ten minutes what no command announces, such as `ipx add` or a
/// shorter schedule. A command, a refresh or a feed added on the page, wakes it at once anyway.
async fn until_next_scan(ctx: &Ctx) -> std::time::Duration {
let next = async {
let cfg = ctx.cfg();
let subscribed = ctx.db.subscriber_counts().await?;
let states = ctx.db.http_states().await?;
anyhow::Ok(
subscriptions(ctx)
.await?
.iter()
// A feed with no row yet has never been checked: due now.
.map(|s| {
states.get(&s.id).and_then(|st| due_at(&cfg, &s.cfg, st, subscribed.contains_key(&s.id))).unwrap_or(0)
})
.min(),
)
};
let wait = match next.await {
Ok(Some(at)) => (at - db::now()).max(0) as u64,
Ok(None) => u64::MAX,
Err(e) => {
tracing::warn!(error = %format!("{e:#}"), "could not work out when the next feed is due");
0
}
};
std::time::Duration::from_secs(wait.clamp(30, 600))
}
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
/// global schedule applies, but the feed's own <ttl> raises it when the publisher asks to
/// be polled less often. A feed that is failing backs off (`backoff`), from `error_since`, when
@@ -1257,6 +1369,24 @@ enum Outcome {
Opml { added: Vec<String>, removed: usize, kept: usize, total: usize },
}
/// A feed that answered from a new address after permanent redirects moves there in the
/// catalogue, so it is read from there and no longer redirected every time. One an OPML lists
/// is left alone, since the OPML would only put the old address back; so is a move onto an
/// address another feed already has.
async fn follow_move(ctx: &Ctx, id: &str, to: &str) -> Result<()> {
// Said in the event below, not logged here: the event is the log line.
if subscriptions(ctx).await?.iter().any(|s| s.id != id && feed::same_feed(&s.cfg.url, to)) {
tracing::info!(feed = id, to, "the feed moved to an address another feed already has; leaving it");
return Ok(());
}
let mut cfg = (*ctx.cfg()).clone();
let Some(f) = cfg.feeds.get_mut(id) else { return Ok(()) };
let from = std::mem::replace(&mut f.url, to.to_owned());
ctx.store_cfg(cfg).await?;
ctx.out.emit(Event::FeedMoved { feed: id.to_owned(), from, to: to.to_owned() });
Ok(())
}
#[tracing::instrument(name = "feed", skip_all, fields(feed = id))]
async fn scan_one(
ctx: &Arc<Ctx>,
@@ -1264,9 +1394,13 @@ async fn scan_one(
feed_cfg: &config::Feed,
state: &db::HttpState,
force: bool,
// The body the scan already fetched for it, if it did (#104).
prefetched: Option<Result<feed::Fetched>>,
// The body the scan already fetched for it, if it did (#104), and where it moved, if it did.
prefetched: Option<Result<(feed::Fetched, Option<String>)>>,
) -> Result<Outcome> {
// The queue follows the feed's settings each time it is due, changed or not, so files a
// limit or auto-download no longer reaches stop counting as waiting (#113).
let policy = policy_for(ctx, id, feed_cfg).await?;
ctx.db.hold_back(id, if policy.auto_download { policy.budget } else { 0 }).await?;
// A Patreon creator with more than one show is a list of feeds, like an OPML.
if feed::is_patreon_creator(&feed_cfg.url) {
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
@@ -1292,10 +1426,13 @@ async fn scan_one(
}
}
let mut fetched = match prefetched {
let (mut fetched, moved) = match prefetched {
Some(got) => got?,
None => feed::fetch(&ctx.client, feed_cfg, state.etag.as_deref(), state.last_modified.as_deref()).await?,
None => feed::fetch(&ctx.feed_client, feed_cfg, state.etag.as_deref(), state.last_modified.as_deref()).await?,
};
if let Some(to) = moved {
follow_move(ctx, id, &to).await?;
}
// A 304 while nothing is stored means the validator has outlived the data -- a restore
// from backup, a manual edit, a cleanup that removed entries. Believe the database over
@@ -1306,7 +1443,7 @@ async fn scan_one(
if matches!(fetched, feed::Fetched::NotModified) && (stored.entries == 0 || stored.image.is_none()) {
tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator");
ctx.db.clear_validators(id).await?;
fetched = feed::fetch(&ctx.client, feed_cfg, None, None).await?;
fetched = feed::fetch(&ctx.feed_client, feed_cfg, None, None).await?.0;
}
let (bytes, etag, last_modified) = match fetched {
@@ -1330,6 +1467,17 @@ async fn scan_one(
}
let mut parsed = feed::parse(&bytes)?;
// Artwork on http moves to https where its host serves it (#110), before it is compared
// with what is stored, which is then the https address too.
let mut secure = std::collections::HashMap::new();
if let Some(art) = &parsed.image {
parsed.image = Some(feed::prefer_https(&ctx.client, art, &mut secure).await);
}
for entry in parsed.entries.iter_mut() {
if let Some(art) = &entry.image {
entry.image = Some(feed::prefer_https(&ctx.client, art, &mut secure).await);
}
}
// Artwork is looked at when it may have changed: the feed names different artwork from what
// is stored, or someone asked for a refresh, so an icon the site changes or fixes still
// follows it (#80). Looked at on every full read, a feed without validators asked its site
@@ -1354,6 +1502,20 @@ async fn scan_one(
});
}
}, art_span).await;
// A site's icon comes back on whatever the site is on.
if let Some(art) = &parsed.image {
parsed.image = Some(feed::prefer_https(&ctx.client, art, &mut secure).await);
}
// Items stored before, which a scan does not write again, move with their host, and a host
// only they still name is asked too.
for art in ctx.db.http_images(id).await? {
feed::prefer_https(&ctx.client, &art, &mut secure).await;
}
for (host, ok) in &secure {
if *ok {
ctx.db.secure_images(id, host).await?;
}
}
ctx.db.record_feed(
id,
&feed_cfg.url,
@@ -1363,9 +1525,31 @@ async fn scan_one(
parsed.ttl_mins,
parsed.image.as_deref(),
parsed.category.as_deref(),
parsed.description.as_deref(),
).await?;
// Asked on a feed's first read, nothing of it stored yet, or when someone forces a refresh:
// each call costs, and a feed already here without a category can wait for a forced one.
// The answer becomes the catalogue's category, so a feed is not asked twice.
// A feed an OPML lists is not in the catalogue and has nowhere to keep one.
if parsed.category.is_none()
&& (force || stored.entries == 0)
&& !parsed.entries.is_empty()
&& ctx.cfg().feeds.get(id).is_some_and(|f| f.category.is_none())
&& let Ok(key) = std::env::var("TYPESAFE_KEY")
{
match feed::classify(&ctx.client, &key, &parsed).await {
Ok(category) => {
let mut cfg = (*ctx.cfg()).clone();
if let Some(f) = cfg.feeds.get_mut(id).filter(|f| f.category.is_none()) {
tracing::info!(feed = id, category = %category, "categorised");
f.category = Some(category);
ctx.store_cfg(cfg).await?;
}
}
Err(e) => tracing::warn!(feed = id, "could not categorise: {e:#}"),
}
}
let policy = policy_for(ctx, id, feed_cfg).await?;
if let Some(parent) = &feed_cfg.group {
let listed: Vec<(&str, &str)> = parsed
.entries
@@ -1412,6 +1596,9 @@ async fn scan_one(
if scan.new_entries > 0 {
ctx.db.rehide(id).await?; // what is new may hold someone's blocked words
}
// A new item takes a place among the newest and the oldest of them leaves the queue; a file a
// filter lets through again may be outside them.
ctx.db.hold_back(id, if policy.auto_download { policy.budget } else { 0 }).await?;
let budget = policy.budget;
if policy.auto_download && budget > 0 {
@@ -1666,7 +1853,10 @@ fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Pol
if subs.is_empty() {
return Policy {
auto_download: feed_cfg.auto_download,
// A feed listed in the Directory with nobody subscribed is there to be found, not
// downloaded: its files would be for no one. Once someone subscribes, the files
// skipped for it are judged again on the next scan, by their settings (#107).
auto_download: feed_cfg.auto_download && !feed_cfg.listed,
allow_explicit: feed_cfg.allow_explicit,
wants: vec![Want { keywords: feed_cfg.keywords.clone(), blocked: vec![] }],
budget: cap(feed_cfg.max_new_per_check),
@@ -1899,6 +2089,26 @@ fn duration(secs: u64) -> String {
mod tests {
use super::*;
#[test]
fn a_feed_is_due_after_its_wait_and_daily_with_nobody_subscribed() {
let cfg = config::Config::default();
let f = feed();
let at = |last: Option<i64>, subscribed| {
due_at(&cfg, &f, &db::HttpState { last_checked: last, ..Default::default() }, subscribed)
};
assert_eq!(at(None, true), None); // never checked: due now
assert_eq!(at(Some(1000), true), Some(1000 + 3600)); // the hourly default
assert_eq!(at(Some(1000), false), Some(1000 + 86_400)); // listed, nobody subscribed
}
#[test]
fn a_listed_feed_nobody_subscribes_to_downloads_nothing() {
let mut f = feed();
assert!(merge_policy(&[], &f, 3).auto_download);
f.listed = true;
assert!(!merge_policy(&[], &f, 3).auto_download);
}
#[test]
fn a_failing_feed_backs_off_doubling_up_to_a_day() {
let hour = 3600;
@@ -1969,6 +2179,7 @@ mod tests {
password: None,
password_env: None,
category: None,
listed: false,
}
}
@@ -2028,6 +2239,7 @@ mod tests {
cfg: std::sync::RwLock::new(Arc::new(cfg)),
db: db::Db::memory().await.unwrap(),
client: reqwest::Client::new(),
feed_client: reqwest::Client::builder().redirect(reqwest::redirect::Policy::none()).build().unwrap(),
out: Emitter::terminal(),
torrents: tokio::sync::OnceCell::new(),
torrent_slots: Arc::new(tokio::sync::Semaphore::new(2)),

View File

@@ -38,6 +38,8 @@ pub fn router(state: WebState) -> Router {
Router::new()
.route("/", get(index))
.route("/api/me", get(me).patch(patch_me))
.route("/api/tokens", get(list_tokens).post(add_token))
.route("/api/tokens/{id}", axum::routing::delete(remove_token))
.route("/api/logout", post(logout))
.route("/api/feeds", get(feeds).post(add_feed))
.route("/api/feeds/{id}", patch(patch_feed).delete(remove_feed))
@@ -56,6 +58,7 @@ pub fn router(state: WebState) -> Router {
.route("/api/status", get(status))
.route("/api/popular", get(get_popular))
.route("/api/directory", get(get_directory))
.route("/api/directory/{id}", get(get_listed))
.route("/api/popular/{id}", post(subscribe_popular))
.route("/api/users", get(list_users).post(add_user))
.route("/api/users/{id}", patch(patch_user).delete(remove_user))
@@ -76,6 +79,8 @@ pub fn router(state: WebState) -> Router {
.route("/favicon.png", get(favicon))
.route("/favicon-dark.png", get(favicon_dark))
.route("/apple-touch-icon.png", get(touch_icon))
// iOS asks for this one first when the site is added to a home screen (#109).
.route("/apple-touch-icon-precomposed.png", get(touch_icon))
.route("/app.js", get(app_js))
.route("/app.css", get(app_css))
.route("/login.js", get(login_js))
@@ -98,8 +103,8 @@ pub async fn serve(state: WebState, bind: &str) -> Result<()> {
.context("serving the web ui")
}
/// Who is asking, in order of how specific the claim is: a proxy that vouches for a name,
/// a session cookie, then the shared token (which is the admin).
/// Who is asking, in order of how specific the claim is: a proxy that vouches for a name, an
/// API token, a session cookie, then the shared token (which is the admin).
///
/// Any of them has to survive being put in a cookie: an `<audio src>` request is issued by
/// the browser, and there is no way to attach a header to it.
@@ -142,6 +147,26 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
}
let by_proxy = user.is_some();
// An API token, made by its owner in Settings, for a script or an agent acting as them
// (#123). One that is not known is turned away rather than tried as anything else: a
// script sent with a revoked token should hear so, not fall through to a cookie.
if user.is_none()
&& let Some(bearer) = req
.headers()
.get(header::AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "))
{
match state.ctx.db.api_token_user(&crate::auth::api_token_hash(bearer.trim())).await {
Ok(Some(u)) => user = Some(u),
Ok(None) => return (StatusCode::UNAUTHORIZED, "unknown or revoked API token").into_response(),
Err(e) => {
tracing::error!(error = %e, "looking up an API token");
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
}
}
// 2. A session cookie from signing in here.
if user.is_none() {
if let Some(sid) = cookie(&req, SESSION_COOKIE) {
@@ -390,6 +415,54 @@ fn last_admin(users: &[crate::db::User], id: i64) -> bool {
admins == [id]
}
/// Your API tokens: names and dates, never the tokens, which are not kept.
async fn list_tokens(
State(state): State<WebState>,
user: crate::db::User,
) -> Result<Json<serde_json::Value>, ApiError> {
let tokens: Vec<_> = state
.ctx
.db
.api_tokens(user.id).await?
.iter()
.map(|t| serde_json::json!({ "id": t.id, "name": t.name, "created": t.created, "last_used": t.last_used }))
.collect();
Ok(Json(serde_json::json!(tokens)))
}
#[derive(Deserialize)]
struct NewToken {
name: String,
}
/// Makes an API token that acts as you, and answers with it: the only time it is shown.
async fn add_token(
State(state): State<WebState>,
user: crate::db::User,
Json(body): Json<NewToken>,
) -> Result<Json<serde_json::Value>, ApiError> {
let name = body.name.trim();
if name.is_empty() {
return Err(ApiError::bad_request("give the token a name, to know it by when revoking it"));
}
let token = crate::auth::new_api_token();
let id = state.ctx.db.create_api_token(user.id, name, &crate::auth::api_token_hash(&token)).await?;
tracing::info!(user = %user.name, token = name, "API token made");
Ok(Json(serde_json::json!({ "id": id, "name": name, "token": token })))
}
async fn remove_token(
State(state): State<WebState>,
user: crate::db::User,
Path(id): Path<i64>,
) -> Result<StatusCode, ApiError> {
if !state.ctx.db.delete_api_token(user.id, id).await? {
return Err(ApiError::bad_request(format!("you have no API token with id {id}")));
}
tracing::info!(user = %user.name, id, "API token revoked");
Ok(StatusCode::NO_CONTENT)
}
async fn list_users(
State(state): State<WebState>,
user: crate::db::User,
@@ -859,8 +932,10 @@ struct PopularRow {
subscribers: i64,
/// Yours already. Everyone counts, you included, so your own feeds are listed too.
subscribed: bool,
/// The feed's own iTunes category, if it names one; most blogs do not.
/// Apple's category, from the feed's own iTunes category or else the catalogue's, and the
/// subcategory within it where there is one: the Directory browses them in that order.
category: Option<String>,
subcategory: Option<String>,
/// Any audio or video enclosure. Unlike category, every feed has an answer, so the
/// Directory's Podcasts and Blogs between them hold everything.
podcast: bool,
@@ -870,8 +945,13 @@ struct PopularRow {
/// first. Popular is the top of it, the directory is all of it, and it is all that
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
/// feeds inside it and never itself: both lists are for finding a show.
async fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
/// The catalogue as others may see it. `everything` is the Directory: every feed, with or
/// without subscribers, so the ones an admin listed show before anyone takes them. Without it,
/// Popular: only what people subscribe to.
async fn popular(state: &WebState, user_id: i64, everything: bool) -> Result<Vec<PopularRow>> {
let db = &state.ctx.db;
// One pass for every feed's title, artwork and category, not three queries a feed.
let mut listed = db.feed_list(user_id, None).await?;
let mine: std::collections::HashSet<String> =
db.subscriptions_for(user_id).await?.into_iter().map(|s| s.feed_id).collect();
let counts = db.subscriber_counts().await?;
@@ -886,23 +966,25 @@ async fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
let n = counts.get(&s.id).copied().unwrap_or(0);
// A feed inside an OPML that looks private is as private as the OPML.
let folder = s.cfg.group.as_deref().and_then(|g| by_id.get(g));
if n == 0
if (n == 0 && !everything)
|| is_folder.contains(s.id.as_str())
|| looks_private(&s.cfg)
|| folder.is_some_and(|f| looks_private(f))
{
continue;
}
let sum = db.feed_summary(&s.id).await?;
let sum = listed.remove(&s.id).map(|l| l.summary).unwrap_or_default();
let subscribed = mine.contains(&s.id);
// The feed's own wins; an admin's is for the feeds, mostly blogs, that name none.
let path = sum.category.or_else(|| s.cfg.category.clone()).map(|c| crate::feed::category_path(&c));
out.push(PopularRow {
id: s.id.clone(),
title: sum.title,
image: sum.image,
subscribers: n,
subscribed,
// The feed's own wins; an admin's is for the feeds, mostly blogs, that name none.
category: sum.category.or_else(|| s.cfg.category.clone()),
category: path.as_ref().map(|p| p.0.clone()),
subcategory: path.and_then(|p| p.1),
podcast: media.contains(&s.id),
});
}
@@ -914,7 +996,7 @@ async fn get_popular(
State(state): State<WebState>,
user: crate::db::User,
) -> Result<Json<Vec<PopularRow>>, ApiError> {
let mut rows = popular(&state, user.id).await?;
let mut rows = popular(&state, user.id, false).await?;
rows.truncate(10);
Ok(Json(rows))
}
@@ -924,11 +1006,58 @@ async fn get_directory(
State(state): State<WebState>,
user: crate::db::User,
) -> Result<Json<Vec<PopularRow>>, ApiError> {
let mut rows = popular(&state, user.id).await?;
let mut rows = popular(&state, user.id, true).await?;
rows.sort_by_key(sort_name);
Ok(Json(rows))
}
/// One of a listed feed's latest items, for its page in the Directory before anyone subscribes
/// (#128): what it is, never its files, whose addresses are where a key would be.
#[derive(Serialize)]
struct ListedItem {
guid: String,
title: Option<String>,
link: Option<String>,
published: Option<i64>,
duration: Option<i64>,
description: Option<String>,
}
#[derive(Serialize)]
struct Listed {
/// What the feed says it is (#130), sanitized as its items are.
description: Option<String>,
items: Vec<ListedItem>,
}
/// A listed feed's description and latest twenty items. Only a feed the Directory lists, so a
/// guessed id reaches nothing private, as `subscribe_popular` checks.
async fn get_listed(
State(state): State<WebState>,
user: crate::db::User,
Path(id): Path<String>,
) -> Result<Json<Listed>, ApiError> {
if !popular(&state, user.id, true).await?.iter().any(|p| p.id == id) {
return Err(ApiError::not_found(format!("{id:?} is not in the directory")));
}
let order = crate::db::order_sql("published", "desc");
let rows = state.ctx.db.entries_in(user.id, Some(&id), crate::db::Filter::parse("all"), None, 0, 20, &order).await?;
let mut sanitizer = feed_sanitizer();
let description = state.ctx.db.feed_description(&id).await?.map(|d| clean_description(&mut sanitizer, &d, None));
let items = rows
.into_iter()
.map(|e| ListedItem {
description: e.description.as_deref().map(|d| clean_description(&mut sanitizer, d, e.link.as_deref())),
guid: e.guid,
title: e.title,
link: e.link,
published: e.published,
duration: e.duration,
})
.collect();
Ok(Json(Listed { description, items }))
}
fn sort_name(p: &PopularRow) -> String {
p.title.clone().unwrap_or_else(|| p.id.clone()).to_lowercase()
}
@@ -940,10 +1069,12 @@ async fn subscribe_popular(
user: crate::db::User,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> {
if !popular(&state, user.id).await?.iter().any(|p| p.id == id) {
if !popular(&state, user.id, true).await?.iter().any(|p| p.id == id) {
return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
}
state.ctx.db.subscribe(user.id, &id).await?;
// A listed feed nobody took was checked once a day at most: read it now, not in an hour.
scan_soon(&state, Some(id.clone())).await;
Ok(Json(serde_json::json!({ "id": id })))
}
@@ -1073,6 +1204,7 @@ mod tests {
password: None,
password_env: None,
category: None,
listed: false,
};
assert!(!looks_private(&f("https://feeds.twit.tv/twit.xml")));
assert!(!looks_private(&f("https://example.com/rss?format=mp3")));
@@ -1119,7 +1251,7 @@ mod tests {
crate::config::Feed {
url: url.into(), folder: None, group: None, media_types: None, schedule: None, keywords: vec![], allow_explicit: false,
auto_download: true, max_new_per_check: None, username: None,
password: None, password_env: None, category: None,
password: None, password_env: None, category: None, listed: false,
}
}
@@ -1201,9 +1333,22 @@ async fn entries(
user: crate::db::User,
Query(page): Query<Page>,
) -> Result<Json<EntryPage>, ApiError> {
subscribed(&state, &user, &id).await?;
entry_page(&state, user.id, Some(&id), &page).await
}
/// A feed's items and files are for the people who subscribe to it (#129). Anyone signed in
/// could read any feed's by its id, a paid one's included, with its enclosure addresses, which
/// can carry the subscriber's key, and fetch, queue or delete its files by theirs. A feed in
/// an OPML has a subscription of its own for everyone subscribed to the OPML, so this holds
/// for those too. Not found, as for any other feed that is not yours.
async fn subscribed(state: &WebState, user: &crate::db::User, feed_id: &str) -> Result<(), ApiError> {
match state.ctx.db.subscription(user.id, feed_id).await? {
Some(_) => Ok(()),
None => Err(ApiError::not_found("you do not subscribe to that feed")),
}
}
/// Every subscribed feed's items together, newest first: All Subscriptions.
async fn all_entries(
State(state): State<WebState>,
@@ -1505,8 +1650,12 @@ async fn remove_feed(
}
// Nobody is left: the feed stops being scanned. Its files and history stay, so if
// someone subscribes again they do not pull the back catalogue a second time.
// someone subscribes again they do not pull the back catalogue a second time. A feed an
// admin listed stays in the Directory, for the next person.
let mut cfg = (*state.ctx.cfg()).clone();
if cfg.feeds.get(&id).is_some_and(|f| f.listed) {
return Ok(StatusCode::NO_CONTENT);
}
if cfg.feeds.remove(&id).is_none() {
// A derived feed: forget it here, though the OPML will list it again on the next
// read unless you unsubscribe from the OPML itself.
@@ -1548,12 +1697,14 @@ async fn set_flags(
async fn download_now(
State(state): State<WebState>,
Path(id): Path<i64>,
user: crate::db::User,
) -> Result<StatusCode, ApiError> {
let enc = state
.ctx
.db
.enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
subscribed(&state, &user, &enc.feed_id).await?;
if enc.path.is_some() {
return Ok(StatusCode::NO_CONTENT); // Already here.
}
@@ -1583,6 +1734,7 @@ async fn delete_file(
.db
.enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
subscribed(&state, &user, &enc.feed_id).await?;
// There is one copy of the file: deleting it deletes everyone's. Say so before doing
// it, once, and let them decide.
@@ -1687,7 +1839,10 @@ async fn events(
/// routine skip of a feed not due, dozens a minute that change nothing.
fn changed_feed(ev: &Event) -> Option<&str> {
match ev {
Event::FeedDone { feed, .. } | Event::FeedError { feed, .. } | Event::DownloadDone { feed, .. } => Some(feed),
Event::FeedDone { feed, .. }
| Event::FeedError { feed, .. }
| Event::FeedMoved { feed, .. }
| Event::DownloadDone { feed, .. } => Some(feed),
Event::FeedSkip { feed, reason } if !reason.starts_with("not due") => Some(feed),
_ => None,
}
@@ -1697,11 +1852,15 @@ fn changed_feed(ev: &Event) -> Option<&str> {
async fn media(
State(state): State<WebState>,
Path(id): Path<i64>,
user: crate::db::User,
req: Request,
) -> Response {
let Ok(Some(enc)) = state.ctx.db.enclosure(id).await else {
return (StatusCode::NOT_FOUND, "no such enclosure").into_response();
};
if let Err(e) = subscribed(&state, &user, &enc.feed_id).await {
return e.into_response();
}
let Some(path) = enc.path else {
return (StatusCode::NOT_FOUND, "not downloaded").into_response();
};
@@ -1716,16 +1875,25 @@ struct ArtQuery {
u: String,
}
/// Artwork a feed names on plain http, fetched here for the https page. The browser upgrades an
/// http image on an https page to https, and a host with no https, such as The Secret Cabal's
/// CDN, then answers nothing (issue #90).
/// Artwork, from iPX: the page asks here for every image a feed or item names, and the first
/// time it is fetched from the publisher and kept on disk (`art`, up to `art_cache_mb`), so
/// later it is fast, the publisher is not asked on every visit, and it outlives the publisher's
/// server. Only an address some feed or item names as its artwork, and only an image up to
/// 5 MB, so the route cannot be pointed at anything else. It began as a way round http-only
/// artwork on the https page (#90).
async fn art(State(state): State<WebState>, Query(q): Query<ArtQuery>) -> Response {
const MAX: usize = 5 << 20;
if !q.u.starts_with("http://") || !state.ctx.db.names_image(&q.u).await.unwrap_or(false) {
let web = q.u.starts_with("http://") || q.u.starts_with("https://");
if !web || !state.ctx.db.names_image(&q.u).await.unwrap_or(false) {
return (StatusCode::NOT_FOUND, "no feed names that artwork").into_response();
}
let keep = state.ctx.cfg().general.art_cache_mb > 0;
let file = crate::art::path(&q.u);
if keep && let Some((kind, body)) = crate::art::read(&file) {
return art_response(kind, body);
}
let got = async {
let mut r = state.ctx.client.get(&q.u).send().await?.error_for_status()?;
let mut r = state.ctx.client.get(&q.u).timeout(crate::feed::FEED_TIMEOUT).send().await?.error_for_status()?;
let kind = r.headers().get(header::CONTENT_TYPE).and_then(|v| v.to_str().ok()).unwrap_or("").to_owned();
anyhow::ensure!(kind.starts_with("image/"), "not an image: {kind}");
let mut body = Vec::new();
@@ -1737,12 +1905,30 @@ async fn art(State(state): State<WebState>, Query(q): Query<ArtQuery>) -> Respon
};
match got.await {
Ok((kind, body)) => {
([(header::CONTENT_TYPE, kind), (header::CACHE_CONTROL, "private, max-age=86400".into())], body).into_response()
if keep && let Err(e) = crate::art::write(&file, &kind, &body) {
tracing::debug!(error = %e, "could not keep the artwork");
}
art_response(kind, body)
}
Err(e) => (StatusCode::BAD_GATEWAY, format!("{e:#}")).into_response(),
}
}
/// An image from someone else's server, served from iPX's own address: it must stay an image.
/// An SVG opened on its own would otherwise run its script as iPX's page, with its cookies.
fn art_response(kind: String, body: Vec<u8>) -> Response {
(
[
(header::CONTENT_TYPE, kind),
(header::CACHE_CONTROL, "private, max-age=2592000".into()),
(header::X_CONTENT_TYPE_OPTIONS, "nosniff".into()),
(header::CONTENT_SECURITY_POLICY, "default-src 'none'; style-src 'unsafe-inline'; sandbox".into()),
],
body,
)
.into_response()
}
#[derive(Deserialize)]
struct Position {
secs: i64,
@@ -1804,8 +1990,10 @@ fn five() -> i64 {
async fn download_latest(
State(state): State<WebState>,
Path(id): Path<String>,
user: crate::db::User,
Json(body): Json<HowMany>,
) -> Result<Json<serde_json::Value>, ApiError> {
subscribed(&state, &user, &id).await?;
let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100)).await?;
for enc in &ids {
state.ctx.db.requeue(*enc).await?;
@@ -1908,6 +2096,7 @@ struct Settings {
download_dir: String,
max_total_gb: f64,
max_age_days: u64,
art_cache_mb: u64,
}
async fn get_settings(State(state): State<WebState>) -> Json<Settings> {
@@ -1920,6 +2109,7 @@ async fn get_settings(State(state): State<WebState>) -> Json<Settings> {
download_dir: cfg.general.download_dir.display().to_string(),
max_total_gb: cfg.general.max_total_gb,
max_age_days: cfg.general.max_age_days,
art_cache_mb: cfg.general.art_cache_mb,
})
}
@@ -1930,6 +2120,7 @@ struct SettingsPatch {
media_types: Option<Vec<String>>,
max_total_gb: Option<f64>,
max_age_days: Option<u64>,
art_cache_mb: Option<u64>,
}
async fn patch_settings(
@@ -1966,6 +2157,9 @@ async fn patch_settings(
if let Some(v) = body.max_age_days {
cfg.general.max_age_days = v;
}
if let Some(v) = body.art_cache_mb {
cfg.general.art_cache_mb = v;
}
state.ctx.store_cfg(cfg).await?;
Ok(StatusCode::NO_CONTENT)
}

View File

@@ -49,7 +49,6 @@ const drive = PAGE === 'admin.html' ? [
['prefsModal', () => ctx.prefsModal()],
['opmlModal', () => ctx.opmlModal()],
['selectFeed (directory)', () => ctx.selectFeed(':directory')],
['selectFeed (popular)', () => ctx.selectFeed(':popular')],
['selectFeed (currently listening)', () => ctx.selectFeed(':listening')],
['selectFeed (all subscriptions)', () => ctx.selectFeed(':all')],
['keysModal', () => ctx.keysModal()],

View File

@@ -242,7 +242,7 @@ test('an item with several enclosures lists them all', async ({ page }) => {
await expect(page.locator('#files .encbox').nth(1).locator('.kind[title^="image"]')).toBeVisible();
});
test('Currently Listening, its own place below Popular, resumes an episode you started or forgets it', async ({ page }) => {
test('Currently Listening, its own place below the Directory, resumes an episode you started or forgets it', async ({ page }) => {
// Second Episode (900s) is 42 seconds in and unfinished. An earlier test may have opened it,
// and opening marks it read; it is listed all the same, because read is not finished. This
// test used to set it unread first, which hid exactly the bug in issue #14.
@@ -251,12 +251,12 @@ test('Currently Listening, its own place below Popular, resumes an episode you s
await page.evaluate(() =>
api('/api/entries/test-show/ui-2/position', { method: 'POST', body: JSON.stringify({ secs: 42 }) }));
// Popular lists feeds and nothing else; the episodes have a place of their own under it.
await page.locator('#feedlist .place', { hasText: 'Popular' }).click();
await expect(page.locator('#popular')).toBeVisible();
// The Directory lists feeds and nothing else; the episodes have a place of their own under it.
await page.locator('#feedlist .place', { hasText: 'Directory' }).click();
await expect(page.locator('#directory')).toBeVisible();
await expect(page.locator('#listening')).toHaveCount(0);
const places = await page.locator('#feedlist .place b').allTextContents();
expect(places.indexOf('Currently Listening')).toBe(places.indexOf('Popular') + 1);
expect(places.indexOf('Currently Listening')).toBe(places.indexOf('Directory') + 1);
await page.locator('#feedlist .place', { hasText: 'Currently Listening' }).click();
const row = page.locator('#listening .childrow', { hasText: 'Second Episode' });
@@ -812,7 +812,7 @@ test('ipx import subscribes the admin, and ipx export writes the feeds out', asy
expect(xml).toContain('http://127.0.0.1:8792/two.xml');
});
test('Popular lists what everyone here reads, but never a private feed', async ({ browser }) => {
test('the Directory lists what everyone here reads, the most subscribed first, but never a private feed', async ({ browser, page }) => {
const { execFileSync } = require('child_process');
const setup = require('./global-setup');
const env = {
@@ -833,21 +833,23 @@ test('Popular lists what everyone here reads, but never a private feed', async (
await piper.locator('button[type=submit]').click();
await expect(piper.locator('#feedlist')).toContainText('No feeds.');
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
const offered = piper.locator('#popular .childrow');
await expect(offered.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
// The most subscribed are the Directory's first section; Popular, their own place, is gone (#124).
await expect(piper.locator('#feedlist .place', { hasText: 'Popular' })).toHaveCount(0);
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
const chart = piper.locator('#dchart .rank');
await expect(chart.filter({ hasText: 'Test Show' })).toBeVisible({ timeout: 20_000 });
// An OPML is listed as the feeds inside it, and a key in a URL marks someone's paid feed.
await expect(offered.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
await expect(offered.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
await expect(chart.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
await expect(chart.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
// No URL reaches the page at all, so neither can a key, and the server holds the same line.
const listed = await (await piper.request.get('/api/popular')).text();
const listed = await (await piper.request.get('/api/directory')).text();
expect(listed).not.toContain('secret123');
expect(listed).not.toContain('.xml');
expect((await piper.request.post('/api/popular/paid-show')).status()).toBe(400);
// Popular is the top ten of the directory, and the directory is every listed feed A to Z,
// with an OPML's feeds in place of the OPML in both.
// The chart is /api/popular's top ten, and the directory is every listed feed A to Z, with an
// OPML's feeds in place of the OPML in both.
const dir = await (await piper.request.get('/api/directory')).json();
const top = await (await piper.request.get('/api/popular')).json();
const names = dir.map(p => (p.title || p.id).toLowerCase());
@@ -855,48 +857,92 @@ test('Popular lists what everyone here reads, but never a private feed', async (
const ids = dir.map(p => p.id);
expect(ids).not.toContain('test-subscriptions');
expect(ids).toEqual(expect.arrayContaining(['grouped-show', 'aardvark-radio']));
expect(top.length).toBe(Math.min(10, dir.length));
expect(top.every(t => ids.includes(t.id))).toBe(true);
expect(ids).not.toContain('paid-show');
// Subscribe from the directory this time: the same feeds, as a grid of cover art.
const tiles = piper.locator('#popular .tile');
const pick = (row, name) => piper.locator(`#dirbar .${row} button`, { hasText: new RegExp(`^${name}$`) });
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
await expect(chart).toHaveCount(top.length);
await expect(chart.first().locator('.no')).toHaveText('1');
await expect(piper.locator('#count')).toContainText(`Directory: ${dir.length} feed`);
// Then the categories, as tiles, each a page of its own.
const cat = name => piper.locator('#directory .cat', { hasText: new RegExp(`^${name}\\d`) });
const tiles = piper.locator('#dtiles .tile');
expect(dir.find(p => p.id === 'test-show')).toMatchObject({ podcast: true, category: 'Technology' });
expect(dir.find(p => p.id === 'picture-blog')).toMatchObject({ podcast: false });
await cat('Technology').click();
await expect(piper.locator('#directory h2')).toHaveText('Technology');
await expect(tiles).toHaveCount(dir.filter(p => p.category === 'Technology').length);
await expect(tiles.filter({ hasText: 'Test Show' })).toBeVisible();
// Its way back lands on the tile it went in by.
await piper.locator('#directory .dback').click();
await expect(cat('Technology')).toBeFocused();
// What a feed is holds on every page. No blog here names Technology, so Blogs has no such tile.
const kind = name => piper.locator('#directory .tabs button', { hasText: new RegExp(`^${name}$`) });
await kind('Blogs').click();
await expect(kind('Blogs')).toHaveAttribute('aria-pressed', 'true');
await expect(cat('Technology')).toHaveCount(0);
await piper.locator('#directory .dall').click();
await expect(tiles).toHaveCount(dir.filter(p => !p.podcast).length);
await expect(tiles.filter({ hasText: 'Test Show' })).toHaveCount(0);
await kind('All').click();
await expect(tiles).toHaveCount(dir.length);
await expect(tiles.filter({ hasText: /Grouped Show|grouped-show/ })).toBeVisible();
await expect(tiles.filter({ hasText: /Test Subscriptions/ })).toHaveCount(0);
await expect(tiles.filter({ hasText: /Paid Show|paid-show/ })).toHaveCount(0);
// Sorted by name either way, A to Z as it opens, or by subscribers (#122).
const byName = dir.map(p => p.title || p.id).sort((a, b) => a.localeCompare(b, undefined, { sensitivity: 'base', numeric: true }));
await expect(tiles.first()).toContainText(byName[0]);
await piper.locator('#dirsort').selectOption('za');
await expect(tiles.first()).toContainText(byName[byName.length - 1]);
await expect(tiles).toHaveCount(dir.length);
await piper.locator('#dirsort').selectOption('subscribers');
await expect(tiles).toHaveCount(dir.length);
await piper.locator('#dirsort').selectOption('az');
// The search box looks through the Directory by name (#126), and clearing it goes back.
await piper.locator('#epSearch').fill('test show');
await expect(piper.locator('#directory h2')).toHaveText('Results for “test show”');
await expect(tiles).toHaveCount(dir.filter(p => (p.title || p.id).toLowerCase().includes('test show')).length);
await piper.locator('#epSearch').fill('');
await expect(tiles).toHaveCount(dir.length);
// Two filters that combine: what a feed is, and what it is about.
expect(dir.find(p => p.id === 'test-show')).toMatchObject({ podcast: true, category: 'Technology' });
expect(dir.find(p => p.id === 'picture-blog')).toMatchObject({ podcast: false });
await pick('tabs', 'Blogs').click();
await expect(tiles).toHaveCount(dir.filter(p => !p.podcast).length);
await expect(tiles.filter({ hasText: 'Test Show' })).toHaveCount(0);
// No empty chips: no blog here names Technology, so Blogs does not offer it.
await expect(pick('chips', 'Technology')).toHaveCount(0);
await pick('tabs', 'Podcasts').click();
await pick('chips', 'Technology').click();
await expect(pick('chips', 'Technology')).toHaveAttribute('aria-pressed', 'true');
await expect(tiles).toHaveCount(dir.filter(p => p.podcast && p.category === 'Technology').length);
// A second press lifts the chip and leaves the kind as it was.
await pick('chips', 'Technology').click();
await expect(tiles).toHaveCount(dir.filter(p => p.podcast).length);
await pick('tabs', 'All').click();
// A feed you do not subscribe to opens its page here: what it is and its latest items (#128).
await tiles.filter({ hasText: 'Test Show' }).locator('.txt').click();
await expect(piper.locator('.show h2')).toHaveText('Test Show');
await expect(piper.locator('#dshow .lep', { hasText: 'First Episode' })).toBeVisible();
// What the feed says it is, from its own <description> (#130).
await expect(piper.locator('#dabout p')).toHaveText('A fixture feed.');
await expect(piper.locator('#dabout button')).toBeHidden();
// Back, onto the tile it was opened from, the page under it as it was.
await piper.locator('#directory .dback').click();
await expect(tiles.filter({ hasText: 'Test Show' })).toBeFocused();
await expect(tiles).toHaveCount(dir.length);
// Only a listed feed has one, and it carries no file or its address.
const preview = await (await piper.request.get('/api/directory/test-show')).text();
expect(preview).toContain('First Episode');
expect(JSON.parse(preview).description).toBe('A fixture feed.');
expect(preview).not.toContain('.mp3');
expect((await piper.request.get('/api/directory/paid-show')).status()).toBe(404);
// Nor does anything else give a feed's items or files to someone who does not subscribe (#129).
expect((await piper.request.get('/api/feeds/test-show/entries')).status()).toBe(404);
expect((await piper.request.get('/api/feeds/paid-show/entries')).status()).toBe(404);
const pics = await page.evaluate(() => api('/api/feeds/picture-blog/entries'));
const file = pics.entries.flatMap(e => e.enclosures).find(x => x.path);
expect(file).toBeTruthy();
expect((await page.request.get(`/media/${file.id}`)).status()).toBe(200);
expect((await piper.request.get(`/media/${file.id}`)).status()).toBe(404);
expect((await piper.request.post(`/api/enclosures/${file.id}/download`)).status()).toBe(404);
expect((await piper.request.delete(`/api/enclosures/${file.id}?force=true`)).status()).toBe(404);
expect((await piper.request.post('/api/feeds/picture-blog/download-latest', { data: { count: 1 } })).status()).toBe(404);
// Add a feed is for an address; Popular and Directory are where you browse (issue #30).
// Add a feed is for an address; the Directory is where you browse (issue #30).
await piper.locator('#addFeed').click();
await expect(piper.locator('#nurl')).toBeVisible();
await expect(piper.locator('#modalCard .childrow')).toHaveCount(0);
await expect(tiles).toHaveCount(dir.length);
await expect(piper.locator('#modalCard .tile, #modalCard .rank')).toHaveCount(0);
await piper.locator('#modalCard button[title="Cancel"]').click();
const row = async () =>
(await (await piper.request.get('/api/popular')).json()).find(p => p.id === 'test-show');
(await (await piper.request.get('/api/directory')).json()).find(p => p.id === 'test-show');
const before = await row();
expect(before.subscribed).toBe(false);
await tiles.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]').click();
@@ -904,9 +950,10 @@ test('Popular lists what everyone here reads, but never a private feed', async (
// Everyone counts, you included: it stays listed, marked as yours, with one more subscriber.
expect(await row()).toMatchObject({ subscribed: true, subscribers: before.subscribers + 1 });
await piper.locator('#feedlist .place', { hasText: 'Popular' }).click();
await expect(offered.filter({ hasText: 'Test Show' }).locator('[title^="Subscribed"]')).toBeVisible();
await expect(offered.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]')).toHaveCount(0);
expect((await piper.request.get('/api/feeds/test-show/entries')).status()).toBe(200);
await piper.locator('#feedlist .place', { hasText: 'Directory' }).click();
await expect(chart.filter({ hasText: 'Test Show' }).locator('[title^="Subscribed"]')).toBeVisible();
await expect(chart.filter({ hasText: 'Test Show' }).locator('button[title="Subscribe"]')).toHaveCount(0);
// All Subscriptions is every item from piper's feeds and only those: the admin's Picture
// Blog is not among them.
@@ -956,6 +1003,37 @@ test('reading an item updates its feed\'s count without reloading the list', asy
expect(lists, 'the row came back with the read, not by reloading the list').toEqual([]);
});
test('artwork comes from iPX, kept, and only as an image', async ({ page }) => {
// Every image a feed names is drawn from iPX's address.
expect(await page.evaluate(() => artHTML('https://example.com/a.jpg', 'A'))).toContain('src="/api/art?u=https%3A%2F%2Fexample.com%2Fa.jpg"');
// Multi Show's items name art.jpg.
await expect(page.locator('.feed', { hasText: 'Multi Show' })).toBeVisible({ timeout: 20_000 });
const src = '/api/art?u=' + encodeURIComponent('http://127.0.0.1:8792/art.jpg');
for (const _ of [1, 2]) { // fetched, then kept
const r = await page.request.get(src);
expect(r.status()).toBe(200);
expect(r.headers()['content-type']).toMatch(/^image\//);
expect(r.headers()['content-security-policy']).toContain('sandbox');
}
// An address no feed names is not fetched.
expect((await page.request.get('/api/art?u=' + encodeURIComponent('http://127.0.0.1:8792/show.xml'))).status()).toBe(404);
});
test('an item without a title is named from its text, then its file, then its show and date', async ({ page }) => {
const names = await page.evaluate(() => [
entryName({ title: 'A Title' }),
entryName({ title: ' ', description: '<p>I like the way <b>AI</b> is evolving.</p><img src="x.png">' }),
entryName({ description: '<p>' + 'word '.repeat(40) + '</p>' }).text.length,
entryName({ enclosures: [{ url: 'https://k.example/media/KARTAS691.mp3?x=1' }] }),
entryName({ feed_id: 'nobody', published: 0 }),
]);
expect(names[0]).toEqual({ text: 'A Title', derived: false });
expect(names[1]).toEqual({ text: 'I like the way AI is evolving.', derived: true });
expect(names[2]).toBeLessThanOrEqual(121); // cut at a word, with …
expect(names[3]).toEqual({ text: 'KARTAS691', derived: true });
expect(names[4]).toEqual({ text: 'nobody', derived: true });
});
test('a deleted file looks as if it was never downloaded', async ({ page }) => {
// Other people subscribe to Picture Blog by now, so both prompts come; take them.
page.on('dialog', d => d.accept());
@@ -1140,13 +1218,49 @@ test('keys move through items and places, after Feedly', async ({ page }) => {
await expect(page.locator('#count')).toContainText('All Subscriptions');
});
test('an API token acts as its owner until it is revoked', async ({ page, request }) => {
// Made in Settings, shown once.
await page.locator('#prefs').click();
await page.locator('#tokname').fill('test agent');
await page.locator('#tokadd').click();
const token = await page.locator('#tokval').inputValue();
expect(token).toMatch(/^ipx_[0-9a-f]{64}$/);
await expect(page.locator('.tokrow', { hasText: 'test agent' })).toContainText('last used never');
// Sent as a Bearer header, with no cookie, it is the person who made it.
const as = t => ({ headers: { Authorization: `Bearer ${t}` } });
const me = await (await request.get('/api/me', as(token))).json();
expect(me.name).toBe(await page.evaluate(() => S.me.name));
expect((await request.get('/api/feeds', as(token))).status()).toBe(200);
// A wrong one is turned away, not let through as anyone.
expect((await request.get('/api/me', as('ipx_' + '0'.repeat(64)))).status()).toBe(401);
// Only the hash is kept: the list never carries the token.
const listed = await page.evaluate(() => api('/api/tokens'));
expect(JSON.stringify(listed)).not.toContain(token);
expect(listed.find(t => t.name === 'test agent').last_used).toBeTruthy();
// Revoked, it opens nothing.
await page.locator('.tokrow', { hasText: 'test agent' }).locator('[data-tok]').click();
await expect(page.locator('.tokrow', { hasText: 'test agent' })).toHaveCount(0);
expect((await request.get('/api/me', as(token))).status()).toBe(401);
});
test('an admin can give a blog its Directory category', async ({ page }) => {
const patch = (id, category) => page.evaluate(([id, category]) =>
api(`/api/feeds/${id}`, { method: 'PATCH', body: JSON.stringify({ category }) }), [id, category]);
const listed = async id => (await page.evaluate(() => api('/api/directory'))).find(p => p.id === id);
await patch('picture-blog', 'Visual Arts');
expect(await listed('picture-blog')).toMatchObject({ podcast: false, category: 'Visual Arts' });
// Apple's subcategory, under its category, as the Directory browses them (#118).
expect(await listed('picture-blog')).toMatchObject({ podcast: false, category: 'Arts', subcategory: 'Visual Arts' });
// A category's subcategories are on its own page, as Apple's directory has them.
const pill = name => page.locator('#directory .pills button', { hasText: new RegExp(`^${name}\\d`) });
await page.locator('#feedlist .place', { hasText: 'Directory' }).click();
await expect(page.locator('#directory .pills')).toHaveCount(0);
await page.locator('#directory .cat', { hasText: /^Arts\d/ }).click();
await pill('Visual Arts').click();
await expect(pill('Visual Arts')).toHaveAttribute('aria-pressed', 'true');
await expect(page.locator('#dtiles .tile')).toHaveCount(1);
// A feed's own iTunes category wins over one given here.
await patch('test-show', 'Comedy');
expect((await listed('test-show')).category).toBe('Technology');
@@ -1233,7 +1347,7 @@ test('the favicon is the logo, square, from both pages', async ({ page }) => {
await page.evaluate(() => setTheme('modern', 'dark'));
await expect(page.locator('#favicon')).toHaveAttribute('href', /^\/favicon-dark\.png\?v=[0-9a-f]{12}$/);
// A browser asks for /favicon.ico on its own, signed in or not.
for (const path of ['/favicon.ico', '/favicon.png', '/favicon-dark.png', '/apple-touch-icon.png']) {
for (const path of ['/favicon.ico', '/favicon.png', '/favicon-dark.png', '/apple-touch-icon.png', '/apple-touch-icon-precomposed.png']) {
const r = await page.request.get(path, { headers: { cookie: '' } });
expect(r.status(), path).toBe(200);
expect(r.headers()['content-type'], path).toBe('image/png');

View File

@@ -12,7 +12,10 @@ http.createServer((req, res) => {
fs.readFile(file, (err, body) => {
if (err) { res.writeHead(404).end('no'); return; }
const type = file.endsWith('.mp3') ? 'audio/mpeg'
: file.endsWith('.opml') ? 'text/x-opml' : 'application/xml';
: file.endsWith('.opml') ? 'text/x-opml'
// Artwork as an image, or /api/art refuses it as not one.
: file.endsWith('.jpg') ? 'image/jpeg'
: file.endsWith('.html') ? 'text/html' : 'application/xml';
res.writeHead(200, { 'content-type': type, 'content-length': body.length });
res.end(body);
});

View File

@@ -566,7 +566,14 @@ input:focus,select:focus{outline:0;border-color:var(--accent)}
/* A show sits under its folder's title, a size down, so an open folder reads as one. */
.feed.child{margin-left:11px}
/* Pinned feeds, at the top of the list: a small pin before the name, and a rule under the last. */
.feed .fpin .i{width:10px;height:10px;margin-right:5px;vertical-align:-1px;color:var(--accent)}
/* Pinned: a disc on the artwork's corner, as a failing feed's mark is, in the accent and the ink
the theme pairs with it. A feed both pinned and failing keeps the error below, the pin above. */
.fpin{
position:absolute;right:-5px;bottom:-5px;width:17px;height:17px;border-radius:50%;
display:grid;place-items:center;background:var(--accent);color:var(--ink);border:2px solid var(--bg);
}
.fpin .i{width:9px;height:9px}
.feed.err .fpin{bottom:auto;top:-5px}
.feed.lastpin{margin-bottom:9px}
.feed.lastpin::after{content:"";position:absolute;left:8px;right:8px;bottom:-5px;border-bottom:1px solid var(--line)}
.feed.child .art{width:28px;height:28px;font-size:11px}
@@ -630,24 +637,101 @@ body.scan-this .fhead [data-a=scan] .i,body.scan-any .fhead [data-a=scanall] .i,
/* Quiet buttons: a filled one on every row outshouted the row that is playing. */
#listening [data-a=play]{color:var(--accent)}
#listening [data-a=remove]{color:var(--faint)}
/* Directory's filters: .tabs for what a feed is, as the item filters are, and chips for what it
is about. A picked chip is underlined in --accent2, as the download bar and the now-playing EQ
are; a .badge's fill already means unread in the sidebar. */
.dirbar{display:flex;flex-wrap:wrap;align-items:center;gap:8px 14px;margin-top:4px}
.dirbar .tabs{flex:none}
.chips{display:flex;flex-wrap:wrap;gap:2px 6px;flex:1 1 0;min-width:0}
.chips button{flex:none;padding:4px 6px;font-size:13px;color:var(--dim);white-space:nowrap;border-bottom:2px solid transparent}
.chips button:hover{color:var(--fg)}
.chips button[aria-pressed="true"]{color:var(--fg);border-bottom-color:var(--accent2)}
/* The Directory, laid out as Apple's is: a large title, then sections. Every page carries what
kind of feed (.tabs, as the item filters are) in its title row. */
.dhead{margin-bottom:6px}
.dback{display:inline-flex;align-items:center;gap:5px;margin:0 0 8px -5px;padding:3px 5px;border-radius:6px;font-size:13px;color:var(--accent)}
.dback .i{width:11px;height:11px}
.dback:hover{background:var(--raise)}
.dtitle{display:flex;flex-wrap:wrap;align-items:center;justify-content:space-between;gap:10px 18px}
.dtitle h2{margin:0;font-size:30px;line-height:1.15;font-weight:700;letter-spacing:-.02em;overflow-wrap:anywhere}
.dhead .sub{margin:5px 0 0;color:var(--dim);font-size:13px;font-variant-numeric:tabular-nums}
.dsec{margin-top:28px}
.dsech{display:flex;align-items:baseline;justify-content:space-between;gap:12px;margin-bottom:10px}
.dsech h3{margin:0;font-size:19px;line-height:1.25;font-weight:650;letter-spacing:-.01em}
.dall{font-size:13px;color:var(--accent);flex:none}
.dall:hover{text-decoration:underline}
/* The most subscribed, numbered because they are ranked: down one column, then the next. */
.chart{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));grid-template-rows:repeat(var(--rows,5),auto);grid-auto-flow:column;column-gap:32px}
.rank{display:grid;grid-template-columns:22px 52px minmax(0,1fr) auto;align-items:center;gap:12px;padding:8px 0;border-bottom:1px solid var(--line)}
.rank .no{font-size:17px;font-weight:650;color:var(--faint);text-align:right;font-variant-numeric:tabular-nums}
.rank .art{width:52px;height:52px;font-size:16px}
.rank .txt b{display:-webkit-box;-webkit-box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;overflow-wrap:anywhere;font-weight:500;font-size:14px;line-height:1.3}
.rank .txt small{display:block;margin-top:2px;color:var(--faint);font-size:12px;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
/* A category: its name, and its three most subscribed shows' covers fanned in the corner, so the
shows are what colour it. Not tinted: a tint by the name's hash meant nothing, and the five that
drew --good read as yours, which green means everywhere else (#131). The fan opens a little
under the pointer, the only thing on the page that moves. */
.cats{display:grid;grid-template-columns:repeat(auto-fill,minmax(210px,1fr));gap:12px}
.cat{
position:relative;overflow:hidden;height:116px;padding:13px 15px;border-radius:12px;text-align:left;
display:flex;flex-direction:column;align-items:flex-start;
background:var(--panel);border:1px solid var(--line);
}
.cat:hover{border-color:var(--accent)}
/* The whole width: the fan sits below two lines of name, and a narrower name broke Government
in two in the wider faces, Classic's and Glass's. */
.cat b{font-size:16px;line-height:1.2;font-weight:650;color:var(--fg);overflow-wrap:break-word}
.cat small{margin-top:3px;font-size:12px;color:var(--dim);font-variant-numeric:tabular-nums}
.cat .fan{position:absolute;right:14px;bottom:-12px;width:112px;height:80px;pointer-events:none}
.cat .fan .art{
position:absolute;bottom:0;width:62px;height:62px;font-size:16px;border-radius:8px;box-shadow:0 3px 10px rgba(0,0,0,.3);
/* Each cover on a layer of its own before the pointer arrives. Given one only as the fan began
to open, the browser drew the covers again on the first frames, and again on the way back,
and the fan opened in a jerk (#132). All three move, for as long as the eye can follow, as
one motion that settles rather than stops. */
will-change:transform;
transition:rotate .45s cubic-bezier(.2,.7,.2,1),translate .45s cubic-bezier(.2,.7,.2,1);
}
.cat .fan .art:nth-last-child(1){right:0;rotate:8deg;translate:0 -12px}
.cat .fan .art:nth-last-child(2){right:25px;rotate:-3deg;translate:0 -4px}
.cat .fan .art:nth-last-child(3){right:50px;rotate:-13deg;translate:0 4px}
.cat:hover .fan .art:nth-last-child(1){rotate:13deg;translate:5px -17px}
.cat:hover .fan .art:nth-last-child(2){rotate:-2deg;translate:0 -8px}
.cat:hover .fan .art:nth-last-child(3){rotate:-19deg;translate:-7px 3px}
/* A category's subcategories, each with how many it holds, then the order. A picked one is
drawn as a picked tab is. */
.dtools{display:flex;align-items:center;gap:10px 14px;margin-bottom:14px}
.pills{display:flex;flex-wrap:wrap;gap:6px;flex:1 1 0;min-width:0}
.pills button{flex:none;padding:4px 11px;border:1px solid var(--line);border-radius:999px;font-size:13px;color:var(--dim);white-space:nowrap}
.pills button small{margin-left:6px;color:var(--faint);font-size:11.5px;font-variant-numeric:tabular-nums}
.pills button:hover{border-color:var(--accent);color:var(--fg)}
.pills button.on{background:var(--raise);border-color:var(--raise);color:var(--fg);font-weight:500}
.dtools .grow{flex:1}
#dirsort{flex:none;width:auto;padding:4px 6px;font-size:13px}
/* Square cover art, title underneath: podcast art is made to be known at a glance. The subscribe
button stays visible, since a button that only shows on hover cannot be reached on a phone. */
.tiles{display:grid;grid-template-columns:repeat(auto-fill,minmax(140px,1fr));gap:18px 14px;margin-top:14px}
.tiles{display:grid;grid-template-columns:repeat(auto-fill,minmax(140px,1fr));gap:18px 14px}
.tiles>.hint{grid-column:1/-1}
.tile{display:grid;grid-template-columns:minmax(0,1fr) auto;gap:6px 2px;align-items:start;cursor:pointer}
.tile{display:grid;grid-template-columns:minmax(0,1fr) auto;gap:6px 2px;align-items:start}
.tile .art{grid-column:1/-1;width:100%;height:auto;aspect-ratio:1;font-size:34px}
.tile .txt b{display:-webkit-box;-webkit-box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;overflow-wrap:anywhere;font-weight:500;font-size:13px;line-height:1.3}
.tile .txt small{display:block;color:var(--faint);font-size:11.5px}
.tile:hover .txt b{color:var(--accent)}
/* Every one opens: yours its feed, the rest their page here. */
.tile,.rank{cursor:pointer}
.tile:hover .txt b,.rank:hover .txt b{color:var(--accent)}
/* A feed's page before you subscribe, as Apple's show page: the cover large, what it is and who
here reads it beside it, then what it has published lately. */
.show{display:flex;gap:22px;align-items:flex-end;margin-top:4px}
.show>.art{width:180px;height:180px;font-size:48px;border-radius:12px;box-shadow:var(--shadow)}
.show .meta{min-width:0;flex:1;display:flex;flex-direction:column;align-items:flex-start;gap:5px}
.show h2{margin:0 0 2px;font-size:30px;line-height:1.15;font-weight:700;letter-spacing:-.02em;overflow-wrap:anywhere}
.show .sub{margin:0;color:var(--dim);font-size:13px;font-variant-numeric:tabular-nums}
.show .acts{margin-top:8px}
.dlink{padding:0;font-size:13px;color:var(--accent);text-align:left}
.dlink:hover{text-decoration:underline}
/* What the feed says it is, three lines of it until More. */
.about{max-width:760px;margin-top:18px}
.about p{margin:0;color:var(--dim);font-size:14px;line-height:1.55;display:-webkit-box;-webkit-box-orient:vertical;-webkit-line-clamp:3;overflow:hidden;overflow-wrap:anywhere}
.about.open p{display:block}
.about .dlink{margin-top:4px}
.latest{max-width:760px}
.lep{padding:12px 0;border-bottom:1px solid var(--line)}
.lep b{display:block;font-size:15px;font-weight:600;line-height:1.35;overflow-wrap:anywhere}
.lep b a{color:var(--fg);text-decoration:none}
.lep b a:hover{color:var(--accent)}
.lep p{margin:4px 0 0;color:var(--dim);font-size:13px;line-height:1.5;display:-webkit-box;-webkit-box-orient:vertical;-webkit-line-clamp:3;overflow:hidden}
.lep small{display:flex;align-items:center;gap:7px;margin-top:6px;color:var(--faint);font-size:12px;font-variant-numeric:tabular-nums}
/* Outlined, not filled: on --raise the warning and error colours fell short of AA in most light
themes, where on the row's own ground they clear it. */
.tag{
@@ -676,7 +760,7 @@ body.scan-this .fhead [data-a=scan] .i,body.scan-any .fhead [data-a=scanall] .i,
/* A selected row is --raise too, and the zero count's pill vanished into it. */
.sel .badge.zero{background:var(--bg)}
/* Counts, sizes and times that change in place should not jiggle the text around them. */
.badge,.feed small,.childrow small,.tile small,.ep,.fhead .sub,.dmeta,#status,#seekrow{font-variant-numeric:tabular-nums}
.badge,.feed small,.childrow small,.tile small,.rank small,.ep,.fhead .sub,.dmeta,#status,#seekrow{font-variant-numeric:tabular-nums}
/* ---------- main ---------- */
#main{overflow:hidden;min-height:0;min-width:0;display:flex;flex-direction:column}
@@ -824,7 +908,10 @@ body.adminpage #log{max-width:none}
icon, so it spilled 3px right of centre and off line with its heading. */
.ep .st,.ep .fl{width:22px;height:22px;padding:0;border-radius:5px;display:grid;place-items:center;font-size:11px;color:var(--accent2)}
.ep .fl{color:var(--faint);font-size:13px}
.ep .fl.on{color:var(--fg)}
/* Pinned: the pinned feed's disc (.fpin), the accent and its ink, drawn inside the 22px button so
the row does not move when it changes. */
.ep .fl.on,.ep .fl.on:hover{color:var(--ink);background:radial-gradient(circle,var(--accent) 8.5px,transparent 9px)}
.ep .fl.on .i{width:9px;height:9px}
/* The icon's EQ bars mark what is playing: standing still, and moving only while it plays. A
paused animation was tried first; the frames it held were a pixel apart and read as dots. */
.eq{display:inline-flex;align-items:flex-end;gap:2px;width:13px;height:12px;flex:none}
@@ -837,6 +924,8 @@ body.playing .eq i:nth-child(3){animation-delay:-.6s}
@keyframes eq{from{transform:scaleY(.35)}}
.ep .st:hover,.ep .fl:hover{background:var(--raise)}
.ep .t{font-weight:600;font-size:13.5px;display:block;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
/* An item with no title shows its opening words: set as text, not as a heading. */
.ep .t.notitle{font-weight:400}
.ep.read .t{color:var(--dim);font-weight:500}
.ep .line{display:flex;gap:9px;align-items:center;flex-wrap:wrap;color:var(--faint);font-size:11.5px}
.ep .line:empty{display:none}
@@ -937,6 +1026,9 @@ input[type=range]::-moz-range-thumb{width:12px;height:12px;border:0;border-radiu
.inline{display:flex;gap:6px;align-items:stretch}
.inline input{flex:1;min-width:0}
.inline .btn{white-space:nowrap;flex:none}
/* An API token in Settings: its name and dates, and the button that revokes it. */
.tokrow{display:flex;align-items:center;justify-content:space-between;gap:8px;margin-bottom:6px}
.tokrow .hint{display:block}
.inline select{flex:none;width:auto}
.inline input[type=number]{flex:none;width:90px}
.check input{width:16px;height:16px;accent-color:var(--accent)}
@@ -1043,9 +1135,22 @@ body:has(#player.on) #status{padding-bottom:3px}
#pnow .txt small,#pright #vol{display:none}
.wrap.plain{padding:14px}
.card{padding:16px}
/* The chips get a line of their own and scroll sideways rather than wrap, so they never push
the grid down. */
.chips{flex:1 1 100%;flex-wrap:nowrap;overflow-x:auto}
/* The subcategories get a line of their own and scroll sideways rather than wrap, so they
never push the grid down. */
.dtitle h2{font-size:25px}
.dsec{margin-top:22px}
.dsech h3{font-size:17px}
.chart{grid-template-columns:minmax(0,1fr);grid-template-rows:none;grid-auto-flow:row}
.cats{grid-template-columns:repeat(auto-fill,minmax(150px,1fr));gap:10px}
.cat{height:100px;padding:11px 12px}
.cat b{font-size:15px}
.cat .fan{right:6px;bottom:-16px;transform:scale(.7);transform-origin:bottom right}
.pills{flex:1 1 100%;flex-wrap:nowrap;overflow-x:auto}
.dtools{flex-wrap:wrap}
.show{flex-direction:column;align-items:flex-start;gap:14px}
.show>.art{width:150px;height:150px;font-size:40px}
.show h2{font-size:24px}
#dirsort{margin-left:auto}
.tiles{grid-template-columns:repeat(auto-fill,minmax(104px,1fr));gap:14px 10px}
/* A log line has no room for four columns: keep time and message, wrap as prose. */

View File

@@ -49,6 +49,10 @@ async function drawServer(){
items are never touched.</span></div>
<div class="field"><label>Delete items older than (days, 0 = keep)</label>
<input type="number" id="gage" min="0" value="${g.max_age_days}"></div>
<div class="field"><label>Artwork kept (MB, 0 = none)</label>
<input type="number" id="gart" min="0" value="${g.art_cache_mb}">
<span class="hint">Show and episode artwork is kept here once shown, so it loads from iPX
instead of each publisher. Over this, what has gone longest unshown is dropped first.</span></div>
<div class="field"><label>Download folder</label>
<span class="hint" style="overflow-wrap:anywhere">${esc(g.download_dir)}</span></div>
<div class="cardacts"><button class="btn primary" id="gsave">${ICON.check} Save</button></div>`;
@@ -59,7 +63,8 @@ async function drawServer(){
max_new_per_check: Math.max(0, Number($('#gmax').value) || 0),
media_types: $('#gtypes').value.split(',').map(t => t.trim()).filter(Boolean),
max_total_gb: Number($('#gquota').value) || 0,
max_age_days: Number($('#gage').value) || 0})});
max_age_days: Number($('#gage').value) || 0,
art_cache_mb: Math.max(0, Number($('#gart').value) || 0)})});
toast('Settings saved');
}catch(e){ toast(e.message, true); }
};

View File

@@ -34,97 +34,233 @@ $('#addFeed').onclick=()=>{
};
};
// What everyone here reads, you included, as a place to start. The rows carry an id, never a
// URL, so a key in someone's feed address never reaches this page.
// The rows carry an id, never a URL, so a key in someone's feed address never reaches this page.
const NONE_LISTED='<p class="hint">Nothing yet. Feeds people here subscribe to show up here.</p>';
async function listFeeds(url,box){
let rows=[];
try{ rows=await api(url)||[]; }catch{}
box.innerHTML=rows.length?'':NONE_LISTED;
for(const p of rows) box.appendChild(listedFeed(p,'childrow'));
return rows.length;
}
/// One listed feed: a row in Popular and the Add a feed dialog, a tile in Directory's grid. The
/// parts are the same either way; the class lays them out.
function listedFeed(p,cls){
/// One listed feed: a tile in the Directory's grid or, given its place, a row in its chart. The
/// parts are the same either way; the class lays them out. Yours opens its feed; any other, its
/// page in the Directory, to see what it is before you take it (#128).
function listedFeed(p,cls,rank?: number){
const el=document.createElement('div');
el.className=cls;
el.innerHTML=artHTML(p.image,p.title||p.id)+
`<div class="txt"><b>${esc(p.title||p.id)}</b>`+
`<small class="meta">${p.subscribers} subscriber${p.subscribers===1?'':'s'}</small></div>`+
el.className=cls+(p.subscribed?' mine':'');
el.tabIndex=0; el.dataset.id=p.id;
// As the feed list's rows: Enter or Space opens it, and stops there, or Space would also
// play or pause.
el.onkeydown=ev=>{
if((ev.key==='Enter'||ev.key===' ')&&ev.target===el){ ev.preventDefault(); ev.stopPropagation(); el.click(); }
};
// A chart says what a show is about, as Apple's does; the grid says who here reads it, when
// anyone does. "0 subscribers" under most of the grid said nothing.
const meta=rank?(p.subcategory||p.category):p.subscribers?plural(p.subscribers,'subscriber'):'';
el.innerHTML=(rank?`<span class="no">${rank}</span>`:'')+artHTML(p.image,NAME(p))+
`<div class="txt"><b>${esc(NAME(p))}</b>${meta?`<small class="meta">${esc(meta)}</small>`:''}</div>`+
// Green, as a downloaded file is: it is already yours. Plus, beside it, is the way to get one.
(p.subscribed?`<span class="subbed" title="Subscribed: click to open it" aria-label="Subscribed">${ICON.subbed}</span>`
:`<button class="btn ico" data-a="sub" title="Subscribe" aria-label="Subscribe">${ICON.plus}</button>`);
// Yours already: the row opens it instead.
if(p.subscribed){ el.onclick=()=>{ closeModal(); selectFeed(p.id); }; return el; }
$('[data-a="sub"]',el).onclick=async()=>{
try{
await api(`/api/popular/${encodeURIComponent(p.id)}`,{method:'POST'});
closeModal(); toast(`Subscribed to ${p.title||p.id}`);
await loadFeeds(true); selectFeed(p.id);
}catch(e){ toast(e.message,true); }
};
if(p.subscribed){ el.onclick=()=>selectFeed(p.id); return el; }
el.onclick=()=>{ dirShow=p.id; $('#content').scrollTop=0; drawDirectory(undefined,'.dback'); };
// Its own click, not the tile's as well.
$('[data-a="sub"]',el).onclick=ev=>{ ev.stopPropagation(); subscribeListed(p); };
return el;
}
// Directory's filters. Kept out here because a finished scan redraws the pane, which would
// otherwise clear them.
let dirKind='All', dirCat=null;
const KINDS={All:()=>true,Podcasts:p=>p.podcast,Blogs:p=>!p.podcast};
/// Directory: every listed feed as its cover art, under two filters that combine: what a feed is
/// (Podcasts, anything with audio or video, or Blogs, the rest) and what it is about (its iTunes
/// category, as chips). Both filter in place, without asking the server again.
async function renderDirectory(url,box){
let rows=[];
try{ rows=await api(url)||[]; }catch{}
if(!rows.length){ box.innerHTML=NONE_LISTED; return 0; }
const bar=$('#dirbar');
// Only a filter when the server has both kinds.
const both=rows.some(KINDS.Podcasts)&&rows.some(KINDS.Blogs);
const btn=(k,v,on)=>`<button type="button" data-${k}="${esc(v)}" class="${on?'on':''}" aria-pressed="${on}">${esc(v)}</button>`;
const draw=()=>{
if(!both) dirKind='All';
const ofKind=rows.filter(KINDS[dirKind]);
// No empty chips: only the categories among the feeds the kind lets through.
const cats=[...new Set(ofKind.map(p=>p.category).filter(Boolean))].sort();
if(!cats.includes(dirCat)) dirCat=null;
bar.innerHTML=
(both?`<div class="tabs" role="group" aria-label="Kind">${Object.keys(KINDS).map(k=>btn('kind',k,k===dirKind)).join('')}</div>`:'')+
(cats.length?`<div class="chips" role="group" aria-label="Category">${cats.map(c=>btn('cat',c,c===dirCat)).join('')}</div>`:'');
// A picked chip lifts on a second press. Everything is redrawn, so the keyboard goes back to
// the button just pressed.
for(const b of $$('button',bar)) b.onclick=()=>{
const k=b.dataset.kind!=null?'kind':'cat', v=b.dataset[k];
if(k==='kind') dirKind=v; else dirCat=dirCat===v?null:v;
draw(); $(`[data-${k}="${CSS.escape(v)}"]`,bar)?.focus();
};
box.innerHTML='';
for(const p of ofKind.filter(p=>!dirCat||p.category===dirCat)) box.appendChild(listedFeed(p,'tile'));
};
draw();
return rows.length;
async function subscribeListed(p){
try{
await api(`/api/popular/${encodeURIComponent(p.id)}`,{method:'POST'});
toast(`Subscribed to ${NAME(p)}`);
await loadFeeds(true); selectFeed(p.id);
}catch(e){ toast(e.message,true); }
}
/// Directory and Popular open in the main pane, as the original's Directory did.
// Where you are in the Directory: what kind of feed, which category and subcategory, in what
// order. Out here because a search redraws the page, and the kind and order hold across visits.
let dirKind='All', dirCat=null, dirSub=null, dirSort='az', dirRows=[];
// The feed whose page is open over all that, by id; going back leaves the rest as it was.
let dirShow=null;
// The two pages that are not a category: every feed, and the feeds that name none. A category is
// Apple's or one an admin typed, neither of which starts with a colon.
const EVERY=':every', OTHER=':other';
const NAME=p=>(p.title||p.id);
const byName=(a,b)=>NAME(a).localeCompare(NAME(b),undefined,{sensitivity:'base',numeric:true});
const bySubs=(a,b)=>b.subscribers-a.subscribers||byName(a,b);
/// The grid's orders. It opens A to Z, as the server sends it.
const DIR_SORTS={
subscribers:['Most subscribers',bySubs],
az:['Name, A to Z',byName],
za:['Name, Z to A',(a,b)=>byName(b,a)],
};
const KINDS={All:()=>true,Podcasts:p=>p.podcast,Blogs:p=>!p.podcast};
const KIND_WORD={All:'feed',Podcasts:'podcast',Blogs:'blog'};
/// The Directory, as Apple's is laid out: it opens on the most subscribed feeds and the
/// categories, and a category has a page of its own, its subcategories along the top of its
/// grid. What a feed is (Podcasts, anything with audio or video, or Blogs, the rest) holds across
/// every page, and the search box looks through the lot. One fetch draws all of it.
async function renderDirectory(url,box){
// Each visit starts at the front, as going back to a tab does.
dirCat=null; dirSub=null; dirShow=null;
try{ dirRows=await api(url)||[]; }catch{ dirRows=[]; }
drawDirectory(box);
return dirRows.length;
}
/// `focus` is the control to put the keyboard back on, since the whole page is redrawn.
function drawDirectory(box=$('#directory'), focus?: string){
if(!box) return;
if(!dirRows.length){ box.innerHTML=NONE_LISTED; return; }
// Only a choice when the server has both kinds.
const both=dirRows.some(KINDS.Podcasts)&&dirRows.some(KINDS.Blogs);
if(!both) dirKind='All';
const rows=dirRows.filter(KINDS[dirKind]), word=KIND_WORD[dirKind];
const count=n=>`${n.toLocaleString()} ${word}${n===1?'':'s'}`;
const of=c=>p=>c===EVERY||(c===OTHER?!p.category:p.category===c);
// A kind can leave a category with nothing in it; that goes back to the front.
if(dirCat&&!rows.some(of(dirCat))) dirCat=null;
const catName=c=>c===EVERY?`All ${word}s`:c===OTHER?'Other':c;
const top=list=>list.filter(p=>p.subscribers).sort(bySubs).slice(0,10);
const btn=(k,v,label,on)=>`<button type="button" data-${k}="${esc(v)}" aria-pressed="${on}"${on?' class="on"':''}>${label}</button>`;
// back: where its button goes, '' for the front; none on the front itself.
const head=(title,sub,back?)=>`<div class="dhead">${back==null?'':
`<button type="button" class="dback" data-go="${esc(back)}">${ICON.left}${esc(back?catName(back):'Directory')}</button>`}
<div class="dtitle"><h2>${esc(title)}</h2>${both?`<div class="tabs" role="group" aria-label="Kind">${
Object.keys(KINDS).map(k=>btn('kind',k,k,k===dirKind)).join('')}</div>`:''}</div>
<p class="sub">${esc(sub)}</p></div>`;
const sec=(title,body,extra='')=>`<section class="dsec">${title?`<div class="dsech"><h3>${esc(title)}</h3>${extra}</div>`:''}${body}</section>`;
const tools=(pills='')=>`<div class="dtools">${pills||'<span class="grow"></span>'}<select id="dirsort" aria-label="Sort">${
Object.entries(DIR_SORTS).map(([k,[label]])=>`<option value="${k}"${k===dirSort?' selected':''}>${label}</option>`).join('')}</select></div>`;
const q=S.q.trim(), grid='<div class="tiles" id="dtiles"></div>', board='<div class="chart" id="dchart"></div>';
let html, chart=[], tiles=null;
const show=dirShow&&dirRows.find(p=>p.id===dirShow);
if(show){
const n=show.subscribers;
html=`<div class="dhead"><button type="button" class="dback" data-back>${ICON.left}${esc(q?'Results':dirCat?catName(dirCat):'Directory')}</button></div>
<div class="show">${artHTML(show.image,NAME(show))}<div class="meta"><h2>${esc(NAME(show))}</h2>${
// Its category's page is one click away from here.
show.category?`<button type="button" class="dlink" data-go="${esc(show.category)}">${esc(show.subcategory||show.category)}</button>`:''}
<p class="sub">${n?`${n} ${n===1?'person here subscribes':'people here subscribe'}`:'Nobody here subscribes yet'}</p>
<div class="acts"><button class="btn ico primary" data-a="sub" title="Subscribe" aria-label="Subscribe">${ICON.plus}</button></div></div></div>
<div class="about" id="dabout" hidden><p></p><button type="button" class="dlink" aria-expanded="false" hidden>More</button></div>`
+sec(show.podcast?'Latest episodes':'Latest posts','<div class="latest" id="dshow"><p class="hint">Loading…</p></div>');
}else if(q){
// Names only: a category is a tap away, and "news" matching every News feed buried the
// one called that.
tiles=rows.filter(p=>NAME(p).toLocaleLowerCase().includes(q.toLocaleLowerCase()));
html=head(`Results for “${q}”`,tiles.length?count(tiles.length):`No ${word} here has that in its name.`,dirCat||'')+tools()+grid;
}else if(!dirCat){
chart=top(rows);
const cats=new Map<string,any[]>();
for(const p of rows){ const c=p.category||OTHER; if(!cats.has(c)) cats.set(c,[]); cats.get(c).push(p); }
// Other, the feeds with no category, goes last.
const order=[...cats.keys()].sort((a,b)=>Number(a===OTHER)-Number(b===OTHER)||a.localeCompare(b));
html=head('Directory',`${count(rows.length)} on this server. Private feeds are never listed.`)
+(chart.length?sec('Most subscribed',board):'')
+sec('Categories',`<div class="cats">${order.map(c=>catTile(c,catName(c),count(cats.get(c).length),cats.get(c))).join('')}</div>`,
`<button type="button" class="dall" data-go="${EVERY}">See all ${rows.length.toLocaleString()}</button>`);
}else{
const inCat=rows.filter(of(dirCat));
const subs=dirCat===EVERY?[]:[...new Set(inCat.map(p=>p.subcategory).filter(Boolean))].sort();
if(!subs.includes(dirSub)) dirSub=null;
if(dirCat!==EVERY) chart=top(inCat);
tiles=inCat.filter(p=>!dirSub||p.subcategory===dirSub);
const pill=(v,label,n)=>btn('sub',v,`${esc(label)}<small>${n}</small>`,(dirSub||'')===v);
html=head(catName(dirCat),dirCat===OTHER?`${count(inCat.length)} that name no category`:count(inCat.length),'')
+(chart.length?sec('Most subscribed',board):'')
// Its own heading only when the chart is above it, to say where one list ends.
+sec(chart.length?`${word[0].toUpperCase()+word.slice(1)}s`:'',tools(subs.length?`<div class="pills" role="group" aria-label="Subcategory">${
pill('','All',inCat.length)+subs.map(s=>pill(s,s,inCat.filter(p=>p.subcategory===s).length)).join('')}</div>`:'')+grid);
}
box.innerHTML=html;
if(show){ $('.show [data-a="sub"]',box).onclick=()=>subscribeListed(show); fillShow(box,show); }
const c=$('#dchart',box);
if(c){ c.style.setProperty('--rows',String(Math.ceil(chart.length/2))); chart.forEach((p,i)=>c.appendChild(listedFeed(p,'rank',i+1))); }
const t=$('#dtiles',box);
if(t) for(const p of tiles.sort(DIR_SORTS[dirSort][1])) t.appendChild(listedFeed(p,'tile'));
box.onclick=ev=>{
const b=(ev.target as Element).closest<HTMLElement>('[data-go],[data-kind],[data-sub],[data-back]');
if(!b) return;
// Back from a feed's page, onto the tile or row it was opened from.
if(b.dataset.back!=null){ const id=dirShow; dirShow=null; return drawDirectory(box,`[data-id="${CSS.escape(id)}"]`); }
if(b.dataset.kind!=null){ dirKind=b.dataset.kind; return drawDirectory(box,`[data-kind="${CSS.escape(dirKind)}"]`); }
if(b.dataset.sub!=null){ dirSub=b.dataset.sub||null; return drawDirectory(box,`[data-sub="${CSS.escape(b.dataset.sub)}"]`); }
// Going anywhere ends a search.
if(S.q){ S.q=''; $('#epSearch').value=''; }
const from=dirCat;
dirCat=b.dataset.go||null; dirSub=null; dirShow=null;
$('#content').scrollTop=0;
// Into a page, onto its way back; back out, onto the tile you went in by.
drawDirectory(box,dirCat?'.dback':from?`[data-go="${CSS.escape(from)}"]:not(.dback)`:undefined);
};
const s=$('#dirsort',box);
if(s) s.onchange=()=>{ dirSort=s.value; drawDirectory(box,'#dirsort'); };
if(focus) $(focus,box)?.focus();
}
/// A listed feed's page, filled in: what it says it is, its first lines unless you ask for the
/// rest, and its latest items, each one's title, opening its post on the publisher's site where
/// it has one, its first lines, and when and how long.
async function fillShow(box,p){
const list=$('#dshow',box), about=$('#dabout',box);
let got;
try{ got=await api(`/api/directory/${encodeURIComponent(p.id)}`); }
catch(e){ list.innerHTML=`<p class="hint">${esc(e.message)}</p>`; return; }
// Gone back, or on to another, while it loaded.
if(dirShow!==p.id||!list.isConnected) return;
// Some feeds give their title again as their description, which says nothing new.
const said=plainText(got.description);
if(said&&said!==NAME(p)){
const text=$('p',about), more=$('button',about);
text.textContent=said; about.hidden=false;
// Only when there is more than the three lines shown.
more.hidden=text.scrollHeight<=text.clientHeight+1;
more.onclick=()=>{
const open=about.classList.toggle('open');
more.textContent=open?'Less':'More'; more.setAttribute('aria-expanded',String(open));
};
}
const items=got.items;
list.innerHTML=items.length?items.map(e=>{
const name=entryName(e), text=name.derived?'':plainText(e.description).slice(0,500);
const when=[dateOf(e.published),e.duration&&clock(e.duration)].filter(Boolean);
return `<article class="lep"><b>${e.link?`<a href="${esc(e.link)}" target="_blank" rel="noopener">${esc(name.text)}</a>`:esc(name.text)}</b>`
+(text?`<p>${esc(text)}</p>`:'')
+(when.length?`<small>${when.map(s=>`<span>${s}</span>`).join('<span class="dot"></span>')}</small>`:'')+'</article>';
}).join(''):`<p class="hint">Nothing read from it yet. A feed nobody here subscribes to is checked once a day.</p>`;
}
/// A category as a tile: its name, how many feeds, and the covers of its three most subscribed
/// fanned in the corner, so each one is coloured by what is in it, as Apple's are.
function catTile(c,name,count,rows){
// Covers before initials, which are all Other had; the likeliest first, and first is in front:
// the fan is drawn back to front.
const art=[...rows].sort((a,b)=>Number(!a.image)-Number(!b.image)||bySubs(a,b)).slice(0,3).reverse();
return `<button type="button" class="cat" data-go="${esc(c)}"><b>${esc(name)}</b><small>${count}</small>`+
`<span class="fan" aria-hidden="true">${art.map(p=>artHTML(p.image,NAME(p))).join('')}</span></button>`;
}
/// The Directory and Currently Listening open in the main pane, as the original's Directory did.
async function renderListed(v){
const box=$('#content');
box.classList.add('plain');
$('#tbRemove').disabled=true;
syncTools(null);
if(v===VIEWS[':directory']){
$('#epSearch').placeholder='Find a feed…';
box.innerHTML='<div id="directory"><p class="hint">Loading…</p></div>';
$('#count').textContent=v.title;
const n=await renderDirectory(v.url,$('#directory'));
if(VIEWS[S.feed]===v) $('#count').textContent=`${v.title}: ${plural(n,'feed')}`;
return;
}
$('#epSearch').placeholder='Search items…';
const listening=v===VIEWS[':listening'], grid=v===VIEWS[':directory'];
box.innerHTML=`
<div class="fhead slim">
<div class="art">${v.icon}</div>
<div class="meta"><h2>${v.title}</h2>
<div class="sub">${v.blurb}${listening?'':' Everyone counts, you included. Private feeds are never listed.'}</div></div>
<div class="meta"><h2>${v.title}</h2><div class="sub">${v.blurb}</div></div>
</div>
${grid?'<div class="dirbar" id="dirbar"></div>':''}
<div class="${grid?'tiles':'childlist'}" id="${listening?'listening':'popular'}"><p class="hint">Loading…</p></div>`;
<div class="childlist" id="listening"><p class="hint">Loading…</p></div>`;
$('#count').textContent=v.title;
const n=await (listening?renderListening:grid?renderDirectory:listFeeds)(v.url,$(listening?'#listening':'#popular',box));
if(VIEWS[S.feed]===v) $('#count').textContent=`${v.title}: ${plural(n,listening?'episode':'feed')}`;
const n=await renderListening(v.url,$('#listening',box));
if(VIEWS[S.feed]===v) $('#count').textContent=`${v.title}: ${plural(n,'episode')}`;
}
/// Currently Listening: episodes you started and have not finished, across every feed you
@@ -140,7 +276,7 @@ async function renderListening(url,box){
el.className='childrow';
el.entry=e;
el.innerHTML=artHTML(e.image||feedArt(e.feed_id),e.title||'')+
`<div class="txt"><b>${EQ}<span>${esc(e.title||'(untitled)')}</span></b>`+
`<div class="txt"><b>${EQ}<span>${esc(entryName(e).text)}</span></b>`+
`<small><span class="fd">${esc(feedName(e.feed_id))}</span><span class="left"></span></small></div>`+
`<button class="iconbtn" data-a="play"></button>`+
`<button class="iconbtn" data-a="remove" title="Remove from Currently Listening" aria-label="Remove from Currently Listening">${ICON.close}</button>`+
@@ -193,7 +329,9 @@ $('#tbRead').onclick=()=>{ const e=cur(); if(e) epAction('read',e,null); };
$('#tbFlag').onclick=()=>{ const e=cur(); if(e) epAction('flag',e,null); };
let searchT;
$('#epSearch').oninput=ev=>{ clearTimeout(searchT);
searchT=setTimeout(()=>{ S.q=ev.target.value; S.offset=0; loadEntries(); },250); };
// The Directory looks through what it already has; the rest ask the server (#126).
searchT=setTimeout(()=>{ S.q=ev.target.value; S.offset=0;
if(S.feed===':directory'){ dirShow=null; drawDirectory(); } else loadEntries(); },250); };
// Crossing the phone breakpoint moves the files between their pane and the text.
window.matchMedia?.('(max-width:820px)')?.addEventListener?.('change',()=>{ const e=cur(); if(e) showDetail(e); });
@@ -239,12 +377,35 @@ async function prefsModal(){
<span class="hint">Comma separated words or phrases, in every feed you read. An item with
one in its title or text is hidden from you and not downloaded for you. Each feed's
settings can add more.</span></div>
<div class="field"><label for="tokname">API tokens</label>
<div id="tokens"></div>
<div class="inline"><input type="text" id="tokname" placeholder="What it is for">
<button class="btn" id="tokadd" title="Make an API token" aria-label="Make an API token">${ICON.plus} Make</button></div>
<span class="hint">A token lets a script or an agent use iPX as you, adding and removing
your feeds and reading your items: it sends <code>Authorization: Bearer</code> and the token.
Anyone holding one is you here, so revoke one you no longer use.</span></div>
<div class="field"><label>Feeds are checked every</label>
<span class="hint">${everyText(g.every_mins)}, for every feed that does not set its own.
${admin?'This and the rest of the server\'s settings are on the <a href="/admin">admin page</a>.':'Only an admin changes this.'}</span></div>`);
$('#stheme').onchange=e=>setTheme(e.target.value,undefined,true);
$('#smode').onchange=e=>setTheme(undefined,e.target.value,true);
$('#gopml').onclick=opmlModal;
drawTokens();
$('#tokadd').onclick=async()=>{
const name=$('#tokname').value.trim(); if(!name){ $('#tokname').focus(); return; }
try{
const t=await api('/api/tokens',{method:'POST',body:JSON.stringify({name})});
$('#tokname').value='';
await drawTokens();
// Shown this once: only its hash is kept, so a lost token is revoked and made again.
$('#tokens').insertAdjacentHTML('afterbegin',`<div class="field" id="toknew"><span class="hint">Your new token,
${esc(t.name)}. Copy it now: it is not shown again.</span>
<div class="inline"><input type="text" id="tokval" readonly value="${esc(t.token)}">
<button class="btn ico" id="tokcopy" title="Copy" aria-label="Copy">${ICON.copy}</button></div></div>`);
$('#tokcopy').onclick=()=>copyText(t.token,$('#tokcopy'));
$('#tokval').select();
}catch(err){ toast(err.message,true); }
};
$('#sblock').onchange=async e=>{
const blocked=splitWords(e.target.value);
try{
@@ -255,6 +416,19 @@ async function prefsModal(){
};
}
/// Your API tokens, each with when it was made and last used, and a button to revoke it.
async function drawTokens(){
const box=$('#tokens'); if(!box) return;
const list=await api('/api/tokens').catch(()=>[]);
box.innerHTML=list.map(t=>`<div class="tokrow"><span><b>${esc(t.name)}</b>
<span class="hint">made ${dateOf(t.created)}, last used ${ago(t.last_used)}</span></span>
<button class="btn ico" data-tok="${t.id}" title="Revoke ${esc(t.name)}" aria-label="Revoke ${esc(t.name)}">${ICON.trash}</button></div>`).join('');
for(const b of $$('[data-tok]',box)) b.onclick=async()=>{
try{ await api(`/api/tokens/${b.dataset.tok}`,{method:'DELETE'}); toast('Revoked'); drawTokens(); }
catch(err){ toast(err.message,true); }
};
}
const splitWords=(s: string)=>s.split(',').map(w=>w.trim()).filter(Boolean);
function settingsModal(f, newUrl?: string){
@@ -307,7 +481,7 @@ function settingsModal(f, newUrl?: string){
// Offer the categories the Directory already shows, so a blog about games joins Games rather
// than starting a second chip beside it.
if($('#scats')) api('/api/directory').then(rows=>{ $('#scats').innerHTML=[...new Set((rows||[])
.map(p=>p.category).filter(Boolean))].sort().map(c=>`<option value="${esc(c)}">`).join(''); }).catch(()=>{});
.flatMap(p=>[p.category,p.subcategory]).filter(Boolean))].sort().map(c=>`<option value="${esc(c)}">`).join(''); }).catch(()=>{});
$('#ssave').onclick=async()=>{
const max=$('#smax').value;
try{

View File

@@ -28,10 +28,8 @@ const unreadFirst=(a,b)=>Number(b.unread>0)-Number(a.unread>0);
// The original's source list opened with these, above the feeds. They are places, not feeds:
// an id starting with ':' can never be a feed's, since feed ids are slugs.
const VIEWS={
':directory':{title:'Directory',icon:ICON.directory,url:'/api/directory',
blurb:'Every feed anyone on this server subscribes to, A to Z. The feeds inside an OPML are listed one by one, not the OPML.'},
':popular':{title:'Popular',icon:ICON.popular,url:'/api/popular',
blurb:'The ten feeds with the most subscribers here. The feeds inside an OPML count one by one, not the OPML.'},
// The Directory holds the most subscribed feeds too, which had a place of their own, Popular (#124).
':directory':{title:'Directory',icon:ICON.directory,url:'/api/directory'},
':listening':{title:'Currently Listening',icon:ICON.audio,url:'/api/entries?filter=in_progress&limit=50',
blurb:'Episodes you started and have not finished, across every feed you subscribe to. Pick one up where you left off.'},
':all':{title:'All Subscriptions',icon:ICON.all},
@@ -130,8 +128,9 @@ function renderFeeds(){
// The mark sits on the artwork, the thing the eye scans the list by, and the line under
// the name says what is wrong in place of the counts.
`<span class="fart">${mine.length?folderArt(f,mine):artHTML(f.image,f.title||f.id)}`+
(err?`<span class="ferr" role="img" title="${esc(err)}" aria-label="Error: ${esc(err)}">${ICON.alert}</span>`:'')+`</span>`+
`<div class="txt"><b>${f.pinned?`<span class="fpin" title="Pinned">${ICON.pinOn}</span>`:''}${esc(f.title||f.id)}</b><small>`+
(err?`<span class="ferr" role="img" title="${esc(err)}" aria-label="Error: ${esc(err)}">${ICON.alert}</span>`:'')+
(f.pinned?`<span class="fpin" role="img" title="Pinned" aria-label="Pinned">${ICON.pinOn}</span>`:'')+`</span>`+
`<div class="txt"><b>${esc(f.title||f.id)}</b><small>`+
(nbad?`${plural(nbad,'feed')} not updating`
:err?esc(f.failing?.reason||'The last check failed')
:`${mine.length?plural(mine.length,'feed'):plural(eps,'item')} · ${saved} downloaded`)+

View File

@@ -1,7 +1,7 @@
/* ---------------- items ---------------- */
let loadSeq=0;
async function loadEntries(append?: boolean){
// Directory and Popular list feeds, not items.
// The Directory lists feeds, and Currently Listening draws its own episodes.
if(!S.feed || VIEWS[S.feed]?.url) return;
const p=new URLSearchParams({offset:String(S.offset),limit:String(LIMIT),filter:S.filter,sort:S.sort.col,dir:S.sort.dir});
if(S.q) p.set('q',S.q);
@@ -65,13 +65,14 @@ function epEl(e){
el.dataset.guid=e.guid;
const num=[e.season?`S${e.season}`:'',e.episode?`E${e.episode}`:''].filter(Boolean).join('');
const left = e.position>10 && e.duration ? `${clock(e.duration-e.position)} left` : (e.duration?clock(e.duration):'');
const name=entryName(e);
el.innerHTML=`
<button class="st" data-a="read" title="Mark ${e.read?'unread':'read'}">${
player.guid===e.guid?EQ:(e.read?'':'●')}</button>
<button class="fl${e.flagged?' on':''}" data-a="flag" aria-pressed="${!!e.flagged}" title="${
e.flagged?'Unpin':'Pin, so it is never deleted'}">${e.flagged?ICON.pinOn:ICON.pin}</button>
<div class="body">
<span class="t">${esc(e.title||'(untitled)')}</span>
<span class="t${name.derived?' notitle':''}">${esc(name.text)}</span>
<div class="line">${[
num&&`<span>${num}</span>`,
left&&`<span>${left}</span>`,
@@ -136,6 +137,28 @@ function kindIcon(enc){
function feedArt(id=S.feed){ const f=S.feeds.find(x=>x.id===id); return f&&f.image; }
const feedName=id=>{ const f=S.feeds.find(x=>x.id===id); return f?(f.title||f.id):id; };
/// What an item is called. Its title, or for one published without (RSS 2.0 makes it optional,
/// and Scripting News titles almost none of its posts) the opening of its text, then its file's
/// name, then its feed and date: "(untitled)" fifty times down a list said nothing about any of
/// them. `derived` marks a name that is not a title, which the list sets as text, not heading.
/// Feed HTML as plain words. An inert document: nothing in it loads or runs, where a detached
/// element fetches its images. A line break or paragraph's end is a space, or "2010)<br>Recorded"
/// read "2010)Recorded".
const plainText=(html?: string)=>html
? (new DOMParser().parseFromString(html.replace(/<(br|\/p|\/div|\/li|\/h\d)\b[^>]*>/gi,' $&'),'text/html').body.textContent||'')
.replace(/\s+/g,' ').trim() : '';
function entryName(e): {text: string, derived: boolean}{
if(e.title&&e.title.trim()) return {text:e.title, derived:false};
const words=plainText(e.description);
if(words){
const cut=words.length>120 ? words.slice(0,120).replace(/\s+\S*$/,'')+'…' : words;
return {text:cut, derived:true};
}
const file=((e.enclosures||[])[0]?.url||'').split(/[?#]/)[0].split('/').pop().replace(/\.[a-z0-9]{1,5}$/i,'');
if(file){ try{ return {text:decodeURIComponent(file), derived:true}; }catch{ return {text:file, derived:true}; } }
return {text:[feedName(e.feed_id), dateOf(e.published)].filter(Boolean).join(', '), derived:true};
}
/// Selecting an item shows it in the pane below, rather than expanding the row.
/// Replaces one row with a fresh one, leaving the rest of the list and its scroll alone.
function swapRow(e){
@@ -271,7 +294,8 @@ function detailHtml(e){
// description was sanitized server-side with ammonia before it ever reached here
return `
<button class="btn ico" id="dback" title="Back to the items" aria-label="Back to the items">${ICON.left}</button>
<h3 class="dt">${esc(e.title||'(untitled)')}</h3>
${/* A post without a title starts with its text: its own first words as a heading above
themselves would read as a mistake. */ e.title&&e.title.trim() ? `<h3 class="dt">${esc(e.title)}</h3>` : ''}
<div class="dmeta">
${/* Joined, so a missing date or number leaves no stray dot behind. */
[f&&esc(f.title||f.id), num, dateOf(e.published), e.duration&&clock(e.duration)]
@@ -328,7 +352,7 @@ function encBox(x){
// Nothing on disk. For an image or a PDF you usually just want to look at it, so link
// straight to the publisher's copy in a new tab -- no download, and nothing proxied
// through here, which would make ipx a fetch-anything relay.
const viewable = !isPlayable(x) && x.state !== 'pending';
const viewable = !isPlayable(x) && x.state !== 'pending' && x.state !== 'held';
return `<div class="encbox">
${kindIcon(x)}
<span class="meta" style="flex:1">${size}</span>
@@ -346,13 +370,13 @@ async function epAction(a: string, e, el, encId?: number){
const path=`/api/entries/${encodeURIComponent(e.feed_id)}/${encodeURIComponent(e.guid)}`;
try{
if(a==='play') play(e, encId!=null ? enc : undefined);
if(a==='share') await share(e.title||'', encId!=null ? enc.url : e.link, el);
if(a==='share') await share(entryName(e).text, encId!=null ? enc.url : e.link, el);
if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); }
if(a==='read'){ await setRead(e,!e.read); redraw(); }
if(a==='get'){
if(!enc) return;
await api(`/api/enclosures/${enc.id}/download`,{method:'POST'});
toast('Queued: '+(e.title||'item'));
toast('Queued: '+entryName(e).text);
}
if(a==='del'){
const f=S.feeds.find(x=>x.id===e.feed_id);

View File

@@ -77,7 +77,7 @@ function installNativePlayback(){
realLoad.call(audio);
const e = player.entry, f = player.feed;
post({t:'load', url:v, enc:player.enc, feedId:f, guid:player.guid,
title:(e && e.title) || '', feedTitle:feedName(f),
title:e ? entryName(e).text : '', feedTitle:feedName(f),
artwork:(e && e.image) || feedArt(f) || null,
// Where the host starts is not this: the seek to where you left off is player.ts's, on
// loadedmetadata, so one piece of code decides it. This is for the host's now-playing

View File

@@ -32,7 +32,7 @@ function play(e,enc=e.enclosures.find(isPlayable)){
if(e.position>5) audio.addEventListener('loadedmetadata',()=>{audio.currentTime=e.position},{once:true});
// Initials, if it comes to that, are the feed's: the episode's read as "SE" beside the feed's art.
$('#partwrap').innerHTML=artHTML(e.image||feedArt(e.feed_id),feedName(e.feed_id));
$('#ptitle').textContent=e.title||'(untitled)';
$('#ptitle').textContent=entryName(e).text;
const f=S.feeds.find(x=>x.id===e.feed_id);
$('#pfeed').textContent=f?(f.title||f.id):'';
$('#player').classList.add('on');
@@ -45,8 +45,8 @@ function play(e,enc=e.enclosures.find(isPlayable)){
function mediaSession(e,f){
if(!('mediaSession' in navigator)) return;
navigator.mediaSession.metadata=new MediaMetadata({
title:e.title||'', artist:f?(f.title||f.id):'', album:f?(f.title||''):'',
artwork:(e.image||(f&&f.image))?[{src:e.image||f.image,sizes:'512x512'}]:[],
title:entryName(e).text, artist:f?(f.title||f.id):'', album:f?(f.title||''):'',
artwork:(e.image||(f&&f.image))?[{src:artSrc(e.image||f.image),sizes:'512x512'}]:[],
});
const h={play:()=>audio.play(),pause:()=>audio.pause(),
seekbackward:()=>audio.currentTime-=15,seekforward:()=>audio.currentTime+=30};
@@ -135,7 +135,7 @@ document.addEventListener('keydown',ev=>{
// Feedly's keys, vim's j and k among them: a letter to move through items or feeds, g and a
// letter to go somewhere, ? to list them. None fire with Ctrl, Alt or Cmd held, so the browser's
// own shortcuts still work, or while a dialog is open.
const GO={a:':all',d:':directory',p:':popular',l:':listening'};
const GO={a:':all',d:':directory',l:':listening'};
let gAt=0;
function stepEntry(by){
if(VIEWS[S.feed]?.url||!S.entries.length) return;
@@ -176,10 +176,10 @@ function keysModal(){
const k=s=>`<kbd>${esc(s)}</kbd>`, g=c=>k('g')+' '+k(c);
const rows=[
['Go to'],
[g('a'),'All Subscriptions'],[g('d'),'Directory'],[g('p'),'Popular'],
[g('a'),'All Subscriptions'],[g('d'),'Directory'],
[g('l'),'Currently Listening'],[g('s'),'Settings'],
[k('Shift')+' '+k('J'),'Next feed'],[k('Shift')+' '+k('K'),'Previous feed'],
[k('/'),'Search items'],[k('r'),'Refresh'],[k('['),'Show or hide the feed list'],
[k('/'),'Search items, or the Directory'],[k('r'),'Refresh'],[k('['),'Show or hide the feed list'],
['Items'],
[k('j')+' or '+k('n'),'Next item'],[k('k')+' or '+k('p'),'Previous item'],
[k('Shift')+' '+k('A'),'Mark all read'],

View File

@@ -30,7 +30,6 @@ const ICON={
close:fa('0 0 384 512','<path fill="currentColor" d="M55.1 73.4c-12.5-12.5-32.8-12.5-45.3 0s-12.5 32.8 0 45.3L147.2 256 9.9 393.4c-12.5 12.5-12.5 32.8 0 45.3s32.8 12.5 45.3 0L192.5 301.3 329.9 438.6c12.5 12.5 32.8 12.5 45.3 0s12.5-32.8 0-45.3L237.8 256 375.1 118.6c12.5-12.5 12.5-32.8 0-45.3s-32.8-12.5-45.3 0L192.5 210.7 55.1 73.4z"/>'), // solid/xmark
menu:fa('0 0 448 512','<path fill="currentColor" d="M0 96C0 78.3 14.3 64 32 64l384 0c17.7 0 32 14.3 32 32s-14.3 32-32 32L32 128C14.3 128 0 113.7 0 96zM0 256c0-17.7 14.3-32 32-32l384 0c17.7 0 32 14.3 32 32s-14.3 32-32 32L32 288c-17.7 0-32-14.3-32-32zM448 416c0 17.7-14.3 32-32 32L32 448c-17.7 0-32-14.3-32-32s14.3-32 32-32l384 0c17.7 0 32 14.3 32 32z"/>'), // solid/bars
directory:fa('0 0 448 512','<path fill="currentColor" d="M0 96C0 60.7 28.7 32 64 32l320 0c35.3 0 64 28.7 64 64l0 320c0 35.3-28.7 64-64 64L64 480c-35.3 0-64-28.7-64-64L0 96zm64 0l0 64 64 0 0-64-64 0zm320 0l-192 0 0 64 192 0 0-64zM64 224l0 64 64 0 0-64-64 0zm320 0l-192 0 0 64 192 0 0-64zM64 352l0 64 64 0 0-64-64 0zm320 0l-192 0 0 64 192 0 0-64z"/>'), // solid/table-list
popular:fa('0 0 576 512','<path fill="currentColor" d="M309.5-18.9c-4.1-8-12.4-13.1-21.4-13.1s-17.3 5.1-21.4 13.1L193.1 125.3 33.2 150.7c-8.9 1.4-16.3 7.7-19.1 16.3s-.5 18 5.8 24.4l114.4 114.5-25.2 159.9c-1.4 8.9 2.3 17.9 9.6 23.2s16.9 6.1 25 2L288.1 417.6 432.4 491c8 4.1 17.7 3.3 25-2s11-14.2 9.6-23.2L441.7 305.9 556.1 191.4c6.4-6.4 8.6-15.8 5.8-24.4s-10.1-14.9-19.1-16.3L383 125.3 309.5-18.9z"/>'), // solid/star
all:fa('0 0 512 512','<path fill="currentColor" d="M232.5 5.2c14.9-6.9 32.1-6.9 47 0l218.6 101c8.5 3.9 13.9 12.4 13.9 21.8s-5.4 17.9-13.9 21.8l-218.6 101c-14.9 6.9-32.1 6.9-47 0L13.9 149.8C5.4 145.8 0 137.3 0 128s5.4-17.9 13.9-21.8L232.5 5.2zM48.1 218.4l164.3 75.9c27.7 12.8 59.6 12.8 87.3 0l164.3-75.9 34.1 15.8c8.5 3.9 13.9 12.4 13.9 21.8s-5.4 17.9-13.9 21.8l-218.6 101c-14.9 6.9-32.1 6.9-47 0L13.9 277.8C5.4 273.8 0 265.3 0 256s5.4-17.9 13.9-21.8l34.1-15.8zM13.9 362.2l34.1-15.8 164.3 75.9c27.7 12.8 59.6 12.8 87.3 0l164.3-75.9 34.1 15.8c8.5 3.9 13.9 12.4 13.9 21.8s-5.4 17.9-13.9 21.8l-218.6 101c-14.9 6.9-32.1 6.9-47 0L13.9 405.8C5.4 401.8 0 393.3 0 384s5.4-17.9 13.9-21.8z"/>'), // solid/layer-group
unread:fa('0 0 512 512','<path fill="currentColor" d="M48 64c-26.5 0-48 21.5-48 48 0 15.1 7.1 29.3 19.2 38.4l208 156c17.1 12.8 40.5 12.8 57.6 0l208-156c12.1-9.1 19.2-23.3 19.2-38.4 0-26.5-21.5-48-48-48L48 64zM0 196L0 384c0 35.3 28.7 64 64 64l384 0c35.3 0 64-28.7 64-64l0-188-198.4 148.8c-34.1 25.6-81.1 25.6-115.2 0L0 196z"/>'), // solid/envelope: a closed letter, not a record button
audio:fa('0 0 448 512','<path fill="currentColor" d="M64 224c0-88.4 71.6-160 160-160s160 71.6 160 160l0 37.5c-10-3.5-20.8-5.5-32-5.5l-16 0c-26.5 0-48 21.5-48 48l0 128c0 26.5 21.5 48 48 48l16 0c53 0 96-43 96-96l0-160C448 100.3 347.7 0 224 0S0 100.3 0 224L0 384c0 53 43 96 96 96l16 0c26.5 0 48-21.5 48-48l0-128c0-26.5-21.5-48-48-48l-16 0c-11.2 0-22 1.9-32 5.5L64 224z"/>'), // solid/headphones
@@ -145,10 +144,11 @@ const tint=name=>{ let h=0; for(const c of name||'?') h=(h*31+c.charCodeAt(0))>>
function tileHTML(name,cls){
return `<div class="art ini ${cls||''}" style="--tint:${tint(name)}">${esc(initials(name))}</div>`;
}
/// On the https page, the browser upgrades an http:// image to https, and a host that has no
/// https shows nothing (issue #90); ipx fetches those itself.
/// Every image a feed names comes from iPX, which keeps a copy: fast after the first time, and
/// no publisher's server asked on every visit. It began with http-only artwork, which the https
/// page could not load (#90).
function artSrc(url: string){
return location.protocol==='https:'&&/^http:\/\//i.test(url) ? '/api/art?u='+encodeURIComponent(url) : url;
return /^https?:\/\//i.test(url) ? '/api/art?u='+encodeURIComponent(url) : url;
}
function artHTML(url: string | null, name: string, cls?: string){
return url