7 Commits

Author SHA1 Message Date
4f4f2e8d95 Try a feed again in minutes when its host could not be looked up for the moment (#143)
205 of the Directory's 208 feeds without a category had never been read: on 2026-10-05 at 03:24
UTC every one failed with "dns error: ... Temporary failure in name resolution" within a minute,
during an outage on Tower that also stopped iPX's log reaching Loki until Tower rebooted at 11:56.
They are working podcasts, NPR's, Megaphone's, Acast's, which resolve now. A feed nobody
subscribes to is read once a day and a failing one backs off up to a day (#99), so each sat
failing, unread and so uncategorised (#117) until the next day, and explain_failure called every
DNS error "This address no longer resolves; the site is gone.", on its page and since #142 on its
Directory page. A forced check of the 208 read 205; each has its items, its description and its
own category now.

A temporary lookup failure, EAI_AGAIN, is not recorded on the feed at all: no last_error, no
error_since to back off from or call it gone by. The daemon notes it in Ctx::retry and the feed
is due again in ten minutes, which the scan's due check and the sleep until the next scan both
honour; a restart forgets it, which only makes the retry sooner. A name that does not exist is
still the feed's error, explained as "may be gone", and the Directory page, like the feed's own,
names a cause only once a feed has been failing a day; before that its last check failed.

The page also drops "Nobody here subscribes yet", which said nothing worth a line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 00:18:42 +00:00
c1b73b3dbe Between releases, the version says a release is in progress: 0.10.3-dev
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:53:35 +00:00
fc0d6bf98b Release 0.10.2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:50:32 +00:00
a44e2b4235 Bring the docs up to what changed: reload, corrections, flags, durability
An audit after 0.10.1 found the docs behind the code. architecture.md named no reload command
(#120) and still said a changed title was picked up, false since #96 until #141; it now says
what a scan writes again, and what the Directory page's endpoint sends (#142). cli.md lacked
add's --list and --category, and that add, rm and import tell a running daemon to read the
catalogue again. users.md did not mention Currently Listening or its search (#127).
configuration.md says that ipx no longer waits for the disk on each write, on SQLite or Postgres
(#135, #140), and what that can lose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:50:15 +00:00
2098343621 Say on a feed's Directory page why it is failing, and who subscribes (#142)
daily-quests, one subscriber and no items, showed "Nothing read from it yet. A feed nobody here
subscribes to is checked once a day." on its page in the Directory: the second sentence was
written whatever the count, and the feed had been failing since #119 said so, which the page
never showed, /api/directory/{id} sending no error.

It sends `failing` now, the reason a feed's own page gives (explain_failure), or that the last
check failed, never the error itself, which can name the feed's address, and the Directory names
none. The page shows it under the header in the error colour, and says a feed is checked once a
day only when nobody subscribes and nothing is wrong.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:50:15 +00:00
5684a02e79 Write again a stored item the feed now tells differently (#141)
"APPOFENIACS - A Technology Movie about Human Behavior" in Ain't It Cool News showed 66,170
characters: the feed's own HTTP headers and then the whole feed, tags stripped, and every other
item of the feed carried the text of the one before it. The publisher's feed had served it that
way, as only its XML could make a shift like that, and has since been fixed; ipx had read the
fixed feed and kept the broken text. Since 57ab419 (#96) a scan called record_entry only for a
guid it had not stored, so its UPDATE of title, description, image and duration was never
reached, and no correction, retitled episode or fixed length arrived.

Db::changed_items reads back what is stored for the items the feed lists, not the archive, and
names those it now gives a different title, text, artwork, length or number, a field it leaves
out counting as unchanged; the scan writes those again too. A corrected item keeps its read
state. A test serves a feed broken and then fixed from a socket of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:50:15 +00:00
50c3b9db64 Between releases, the version says a release is in progress: 0.10.2-dev
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 19:58:40 +00:00
13 changed files with 268 additions and 29 deletions

View File

@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- A feed's page in the Directory says how many people here subscribe only when someone does, without "Nobody here subscribes yet".
### Fixed
- A moment's DNS failure no longer marks a feed as failing for a day or calls it gone: iPX tries it again in ten minutes. One failed lookup one morning left 205 podcasts unread, uncategorised and labelled gone until the next day. A name that does not resolve is said to be maybe gone, and only once it has been failing a day, on its Directory page as on its own.
## [0.10.2] - 2026-10-05
### Fixed
- A publisher's correction to an item already in iPX, a retitled episode, mended text, new artwork or a fixed length, arrives on the next read. Since 0.9.1 an item kept what it first said: Ain't It Cool News's items each showed the text of the one before, long after the feed was fixed.
- A feed's page in the Directory says when the feed is failing, and why, and no longer says nobody here subscribes to a feed someone does.
## [0.10.1] - 2026-10-05
### Fixed
@@ -751,7 +766,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Torrent enclosures through librqbit, seeding to a ratio or a time, with a stall timeout.
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
[unreleased]: https://git.sdf1.net/rays/ipx/compare/v0.10.1...main
[unreleased]: https://git.sdf1.net/rays/ipx/compare/v0.10.2...main
[0.10.2]: https://git.sdf1.net/rays/ipx/compare/v0.10.1...v0.10.2
[0.10.1]: https://git.sdf1.net/rays/ipx/compare/v0.10.0...v0.10.1
[0.10.0]: https://git.sdf1.net/rays/ipx/compare/v0.9.1...v0.10.0
[0.9.1]: https://git.sdf1.net/rays/ipx/compare/v0.9.0...v0.9.1

2
Cargo.lock generated
View File

@@ -1820,7 +1820,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]]
name = "ipx"
version = "0.10.1"
version = "0.10.3-dev"
dependencies = [
"ammonia",
"anyhow",

View File

@@ -1,6 +1,6 @@
[package]
name = "ipx"
version = "0.10.1"
version = "0.10.3-dev"
edition = "2024"
[dependencies]

View File

@@ -38,7 +38,9 @@ build needs node and `npm ci` run once.
re-derived into the database each scan, never written to config.toml. A Patreon creator link
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
shows come from Patreon's web API and each becomes a derived feed.
4. Record entries. A changed title or description flips the item back to unread.
4. Record entries: the new ones, and those the feed now gives a different title, text, artwork,
length or number, compared with what is stored for the items it lists (#141). A corrected item
keeps its read state.
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
@@ -90,13 +92,15 @@ printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/
```
**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
(`enclosure`), `status`.
(`enclosure`), `status`, `reload` (read the catalogue and settings again from the database, which
`ipx add`, `rm` and `import` send after changing them in a process of their own; answered with
`status`).
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
there. Commands run one at a time, in the order they arrive, except `status`: the socket answers it
straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
there. Commands run one at a time, in the order they arrive, except `status` and `reload`: the
socket answers them straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
answers only the client that asked, since `status` would end any other client's session.
Progress carries the enclosure id, without which a UI cannot tell one download from another and
@@ -128,7 +132,7 @@ else a `401`. A feed's items and files (its entries, `download-latest`, `/api/en
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch` | |
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
| `GET /api/directory/{id}` | a listed feed's description and latest twenty items, for its page before you subscribe: title, link, date, length and text, never a file or its address, and only for a feed the directory lists |
| `GET /api/directory/{id}` | a listed feed's description, why its last check failed when it did (in words, never its error, which can name its address), and latest twenty items, for its page before you subscribe: title, link, date, length and text, never a file or its address, and only for a feed the directory lists |
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours, the feed's iTunes category, whether it carries audio or video; never a URL, never a private feed); subscribe by id |
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |

View File

@@ -13,13 +13,17 @@ processes must never download the same thing. `--local` forces the work to happe
| `ipx list` | Subscriptions and their state |
| `ipx status` | Counts: feeds, pending, downloaded |
| `ipx fetch [FEED] [--force]` | Scan everything, or one feed. `--force` ignores the TTL |
| `ipx add <url> [--folder X] [--keywords a,b]` | Subscribe; the id comes from the feed title |
| `ipx add <url> [--folder X] [--keywords a,b] [--list] [--category C]` | Subscribe; the id comes from the feed title. `--list` puts it in the Directory for anyone to subscribe to instead, and keeps it there when its last subscriber leaves; `--category` files it under one of the Directory's categories. Run for a feed already in the catalogue, these list it or set its category |
| `ipx rm <feed>` | Unsubscribe; downloads and history are kept |
| `ipx import <file.opml>` / `ipx export <file.opml>` | Move subscriptions in or out. Import subscribes the first admin, as the shared web token does; in the web UI it subscribes whoever is signed in |
| `ipx reap [--dry-run]` | Run retention now |
| `ipx user <add\|list\|passwd\|rm>` | Accounts for the web UI |
| `ipx daemon [--web ADDR]` | Scheduler, control socket and web UI |
`add`, `rm` and `import` change the catalogue in a process of their own. With a daemon running they
then tell it to read the catalogue again; without that it kept its own copy and wrote it back at
its next change, undoing them. If it does not answer they say so: restart it.
## Accounts
Passwords are read from **stdin**, so they miss the shell history and any `ps` listing.

View File

@@ -27,6 +27,12 @@ config.toml's default location is `$XDG_CONFIG_HOME/ipx/config.toml`
Postgres database instead. Back SQLite up by copying `state.db` while the daemon is stopped, or
with `sqlite3 state.db .backup`; back Postgres up with `pg_dump`.
ipx does not wait for the disk to confirm each write: SQLite syncs at checkpoints
(`synchronous=NORMAL`), and ipx's own Postgres sessions set `synchronous_commit=off`, which leaves
other databases on the server as they are. Waiting made a scan take most of a second a feed. A
crash of ipx loses nothing; a power cut, or a crash of the Postgres server, can lose the last moment
of changes, a read mark or a saved position, and never corrupts the database.
## `[general]`
```toml
@@ -48,7 +54,8 @@ database as described above; `download_dir`, `socket` and `organize` stay in con
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
be polled *less* often, and a per-feed `schedule` overrides both. A feed that keeps failing
waits as long as it has been failing before the next try, up to a day, and is back on schedule
after its first success. Admin-only from the UI.
after its first success. A moment's failure to look its host up does not count: the network, not
the feed, failed, and it is tried again in ten minutes. Admin-only from the UI.
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
* **`max_total_gb`** — the reaper deletes to get back under this, oldest first, keeping a 50 MB
pad. Kept items are never deleted, and a file only counts as read once every subscriber has

View File

@@ -87,6 +87,10 @@ Patreon or Supercast, which put the key in the path, and any feed inside an OPML
itself. Those are someone's paid subscriptions, and listing them would let anyone here read what
they pay for.
**Currently Listening**, under the Directory, is every episode you started and have not finished,
across all your feeds, with how much is left. A click picks one up where you left off. The search
box looks through it, as it does a feed's items, and through the Directory by name.
An admin can do the same from **Settings → Manage users…**: add someone (with a password, or none
for someone the proxy signs in), tick or untick Admin, or remove an account. Removing one takes its
subscriptions and read state with it; downloaded files stay. The only admin cannot be demoted or

View File

@@ -2059,6 +2059,64 @@ impl Db {
))
}
/// Of these items, the stored ones the feed now tells differently: a title, text, artwork,
/// length or number that is not what is stored, a field the feed leaves out counting as
/// unchanged, as record_entry writes them. A scan inserted only items it had not stored
/// (#96), and with that alone a publisher's correction never arrived: Ain't It Cool News's
/// items kept a broken copy, each with the text of the one before it, after the feed was
/// fixed (#141). Only what the feed lists is read back, so a scan reads as much as it was
/// sent, not the archive.
#[tracing::instrument(skip_all)]
pub async fn changed_items(
&self,
feed_id: &str,
entries: &[crate::feed::Entry],
) -> Result<std::collections::HashSet<String>> {
if entries.is_empty() {
return Ok(Default::default());
}
let mut a = Args::default();
let feed = a.p(feed_id);
let guids: Vec<String> = entries.iter().map(|e| a.p(e.guid.clone())).collect();
let sql = format!(
"SELECT guid, title, description, image, duration, episode, season FROM entries
WHERE feed_id = {feed} AND guid IN ({})",
guids.join(", ")
);
type Stored = (Option<String>, Option<String>, Option<String>, Option<i64>, Option<i64>, Option<i64>);
let mut stored = std::collections::HashMap::<String, Stored>::new();
for r in self.rows(&sql, a.0).await? {
stored.insert(
r.try_get("", "guid")?,
(
r.try_get("", "title")?,
r.try_get("", "description")?,
r.try_get("", "image")?,
r.try_get("", "duration")?,
r.try_get("", "episode")?,
r.try_get("", "season")?,
),
);
}
fn differs<T: PartialEq>(new: &Option<T>, old: &Option<T>) -> bool {
new.is_some() && new != old
}
Ok(entries
.iter()
.filter(|e| {
stored.get(&e.guid).is_some_and(|s| {
differs(&e.title, &s.0)
|| differs(&e.description, &s.1)
|| differs(&e.image, &s.2)
|| differs(&e.duration, &s.3)
|| differs(&e.episode, &s.4)
|| differs(&e.season, &s.5)
})
})
.map(|e| e.guid.clone())
.collect())
}
#[tracing::instrument(skip_all)]
pub async fn skipped_by_filter(&self, feed_id: &str) -> Result<std::collections::HashMap<String, String>> {
self.rows(
@@ -2396,6 +2454,31 @@ mod tests {
assert_eq!(files, ["u1"].map(String::from).into()); // u2 is g's: left to the insert to find
}
#[tokio::test]
async fn a_scan_finds_the_stored_items_the_feed_now_tells_differently() {
let db = Db::memory().await.unwrap();
let item = |guid: &str, title: &str, text: Option<&str>| crate::feed::Entry {
guid: guid.into(),
title: Some(title.into()),
description: text.map(str::to_owned),
..Default::default()
};
db.record_entry("f", &item("a", "A", Some("the review"))).await.unwrap();
db.record_entry("f", &item("b", "B", Some("the next review"))).await.unwrap();
db.record_entry("f", &item("c", "C", Some("as it was"))).await.unwrap();
let changed = db
.changed_items("f", &[
item("a", "A", Some("the review, fixed")), // the text corrected
item("b", "B renamed", Some("the next review")), // retitled
item("c", "C", None), // says nothing of its text: what is stored stands
item("d", "D", Some("new")), // not stored: the insert's, not this
])
.await
.unwrap();
assert_eq!(changed, ["a", "b"].map(String::from).into());
assert!(db.changed_items("f", &[]).await.unwrap().is_empty());
}
#[tokio::test]
async fn only_artwork_a_feed_names_is_fetched_for_the_page() {
let db = Db::memory().await.unwrap();

View File

@@ -196,15 +196,25 @@ pub fn explain_failure(msg: &str) -> Option<Failure> {
if low.contains("http 402") {
return Some(Failure { reason: "The feed now needs a paid plan.", new_url: None });
}
if low.contains("dns error")
|| low.contains("failed to lookup address")
|| low.contains("no address associated")
// Only a name that does not exist. A lookup that failed for the moment says nothing about the
// site, and NPR's and Megaphone's feeds were labelled gone after a DNS outage (#143).
if !is_temporary_lookup_failure(msg)
&& (low.contains("dns error") || low.contains("failed to lookup address") || low.contains("no address associated"))
{
return Some(Failure { reason: "This address no longer resolves; the site is gone.", new_url: None });
return Some(Failure { reason: "This address no longer resolves; the site may be gone.", new_url: None });
}
None
}
/// A host that could not be looked up for the moment -- the resolver's EAI_AGAIN, glibc's
/// "Temporary failure in name resolution", musl's "Try again" -- as against a name that does
/// not exist ("Name or service not known"). It says the network was not answering, not that the
/// feed is gone (#143).
pub fn is_temporary_lookup_failure(msg: &str) -> bool {
let low = msg.to_ascii_lowercase();
low.contains("dns error") && (low.contains("temporary failure in name resolution") || low.contains("try again"))
}
/// True when a body is an OPML document rather than a feed.
///
/// The original matched on the URL ending in ".opml" (iPXClass.py:34), which misses an
@@ -1317,8 +1327,13 @@ mod tests {
explain_failure("the feed's address answered with nothing at all").unwrap().reason,
"This address answers with nothing; the site may be gone."
);
let dns = explain_failure("connecting: dns error: failed to lookup address information").unwrap();
assert_eq!(dns.reason, "This address no longer resolves; the site is gone.");
let dns = explain_failure("connecting: dns error: failed to lookup address information: Name or service not known").unwrap();
assert_eq!(dns.reason, "This address no longer resolves; the site may be gone.");
// A lookup that failed for the moment is not a site gone (#143).
let blip = "connecting: dns error: failed to lookup address information: Temporary failure in name resolution";
assert!(is_temporary_lookup_failure(blip));
assert!(explain_failure(blip).is_none());
assert!(!is_temporary_lookup_failure("connecting: dns error: failed to lookup address information: Name or service not known"));
let moved = explain_failure("got a web page, not a feed; it links https://x/feed as its feed").unwrap();
assert_eq!(moved.new_url.as_deref(), Some("https://x/feed"));
assert!(explain_failure("got a web page, not a feed").unwrap().new_url.is_none());

View File

@@ -141,9 +141,38 @@ pub struct Ctx {
/// stops every feed scan, every HTTP download and every status command behind it.
/// A one-shot CLI run keeps them inline, or the process would exit mid-download.
pub detach_torrents: bool,
/// Feeds whose host could not be looked up for the moment, and when to try them again: a
/// failure that says nothing about the feed, kept off it and out of the database (#143).
/// Forgotten on a restart, which only makes the retry sooner.
pub retry: std::sync::Mutex<std::collections::HashMap<String, i64>>,
}
/// How soon a feed whose host could not be looked up is tried again (#143).
const RETRY_LOOKUP_SECS: i64 = 600;
impl Ctx {
/// When a feed is next due, taking a lookup retry into account: not before it.
fn due(&self, cfg: &config::Config, feed: &config::Feed, id: &str, state: &db::HttpState, subscribed: bool) -> Option<i64> {
let at = due_at(cfg, feed, state, subscribed);
match self.retry.lock().unwrap().get(id) {
Some(&retry) => Some(at.unwrap_or(0).max(retry)),
None => at,
}
}
/// A scan's failure, recorded on the feed, unless it was only a moment's failure to look its
/// host up: on 2026-10-05 a DNS outage on Tower failed 204 feeds in a minute with
/// "Temporary failure in name resolution", and each, read once a day and backed off as a
/// failing feed, sat a day failing, unread and labelled gone (#143). Tried again in minutes
/// instead, with nothing on the feed to say it failed.
async fn feed_failed(&self, id: &str, url: &str, msg: &str) -> Result<()> {
if feed::is_temporary_lookup_failure(msg) {
self.retry.lock().unwrap().insert(id.to_owned(), db::now() + RETRY_LOOKUP_SECS);
return Ok(());
}
self.db.set_feed_error(id, url, msg).await
}
pub fn cfg(&self) -> std::sync::Arc<config::Config> {
self.cfg.read().unwrap().clone()
}
@@ -275,6 +304,7 @@ async fn main() -> Result<()> {
torrent_slots: Arc::new(tokio::sync::Semaphore::new(2)),
config_path: config_path.clone(),
detach_torrents: is_daemon,
retry: Default::default(),
});
let result = match cli.command {
@@ -1093,7 +1123,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
state.last_modified = None;
}
if !force && let Some(at) = due_at(&cfg, &sub.cfg, &state, subscribed.contains_key(id)) {
if !force && let Some(at) = ctx.due(&cfg, &sub.cfg, id, &state, subscribed.contains_key(id)) {
if at > db::now() {
ctx.out.emit(Event::FeedSkip {
feed: id.clone(),
@@ -1165,7 +1195,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
// One bad feed must not end the scan.
let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
ctx.feed_failed(id, &feed_cfg.url, &msg).await?;
}
}
}
@@ -1191,7 +1221,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
Err(e) => {
let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
ctx.feed_failed(id, &feed_cfg.url, &msg).await?;
}
}
}
@@ -1329,7 +1359,9 @@ async fn until_next_scan(ctx: &Ctx) -> std::time::Duration {
.iter()
// A feed with no row yet has never been checked: due now.
.map(|s| {
states.get(&s.id).and_then(|st| due_at(&cfg, &s.cfg, st, subscribed.contains_key(&s.id))).unwrap_or(0)
let never = db::HttpState::default();
let state = states.get(&s.id).unwrap_or(&never);
ctx.due(&cfg, &s.cfg, &s.id, state, subscribed.contains_key(&s.id)).unwrap_or(0)
})
.min(),
)
@@ -1598,13 +1630,17 @@ async fn scan_one(
// changed nothing however often the feed was scanned.
let skipped = ctx.db.skipped_by_filter(id).await?;
let (known_items, known_files) = ctx.db.stored_items(id).await?;
let changed = ctx.db.changed_items(id, &parsed.entries).await?;
let mut scan = Scan::default();
// Its own span: the time a feed spends after its fetch was untraced (#96).
let store = tracing::info_span!("store", items = parsed.entries.len());
tracing::Instrument::instrument(async {
for entry in &parsed.entries {
// Only what is not stored yet is inserted; the insert would find the rest and do nothing.
if !known_items.contains(&entry.guid) && ctx.db.record_entry(id, entry).await? {
// What is not stored yet is inserted, and what the feed has changed since is written
// again (#141); the rest is left alone. A correction keeps the item's read state.
if (!known_items.contains(&entry.guid) || changed.contains(&entry.guid))
&& ctx.db.record_entry(id, entry).await?
{
scan.new_entries += 1;
}
for enc in &entry.enclosures {
@@ -2279,9 +2315,27 @@ mod tests {
torrent_slots: Arc::new(tokio::sync::Semaphore::new(2)),
config_path: PathBuf::new(),
detach_torrents: false,
retry: Default::default(),
}
}
#[tokio::test]
async fn a_moments_lookup_failure_is_tried_again_soon_and_not_held_against_the_feed() {
let ctx = test_ctx(config::Config::default()).await;
let blip = "connecting: dns error: failed to lookup address information: Temporary failure in name resolution";
ctx.feed_failed("npr", "https://feeds.npr.org/x", blip).await.unwrap();
// Nothing on the feed: no error, no run of failures to back off from or call it gone by.
let s = ctx.db.feed_summary("npr").await.unwrap();
assert!(s.last_error.is_none() && s.error_since.is_none(), "{:?}", s.last_error);
// Due again in minutes, though a feed nobody subscribes to is otherwise read once a day.
let state = db::HttpState { last_checked: Some(db::now() - 2 * 86_400), ..Default::default() };
let due = ctx.due(&config::Config::default(), &feed(), "npr", &state, false).unwrap();
assert!((due - db::now() - RETRY_LOOKUP_SECS).abs() <= 2, "{due}");
// A real failure is the feed's.
ctx.feed_failed("gone", "https://x/feed", "HTTP 404 Not Found").await.unwrap();
assert_eq!(ctx.db.feed_summary("gone").await.unwrap().last_error.as_deref(), Some("HTTP 404 Not Found"));
}
#[tokio::test]
async fn a_reload_takes_the_catalogue_another_ipx_wrote() {
let with = |id: &str| {
@@ -2299,7 +2353,8 @@ mod tests {
assert_eq!(urls, vec!["http://x/added-by-the-cli.xml".to_string()]);
}
/// Answers /empty with a 200 and nothing, anything else with a feed of one item.
/// Answers /empty with a 200 and nothing, /fixed with a feed of one item as it should read,
/// anything else with the same item's text broken.
async fn empty_or_feed_server() -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
@@ -2310,9 +2365,13 @@ mod tests {
tokio::spawn(async move {
let mut buf = [0u8; 2048];
let n = sock.read(&mut buf).await.unwrap_or(0);
let empty = String::from_utf8_lossy(&buf[..n]).starts_with("GET /empty");
let body = if empty { "\n" } else {
"<?xml version=\"1.0\"?><rss version=\"2.0\"><channel><title>T</title><item><title>One</title><guid>g1</guid></item></channel></rss>"
let req = String::from_utf8_lossy(&buf[..n]).into_owned();
let body = if req.starts_with("GET /empty") {
"\n"
} else if req.starts_with("GET /fixed") {
"<?xml version=\"1.0\"?><rss version=\"2.0\"><channel><title>T</title><item><title>One</title><guid>g1</guid><description>As it should read.</description></item></channel></rss>"
} else {
"<?xml version=\"1.0\"?><rss version=\"2.0\"><channel><title>T</title><item><title>One</title><guid>g1</guid><description>Somebody else's text.</description></item></channel></rss>"
};
let resp = format!("HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n{body}", body.len());
let _ = sock.write_all(resp.as_bytes()).await;
@@ -2322,6 +2381,20 @@ mod tests {
format!("http://{addr}")
}
#[tokio::test]
async fn a_publishers_correction_reaches_an_item_already_stored() {
let base = empty_or_feed_server().await;
let ctx = Arc::new(test_ctx(config::Config::default()).await);
let at = |path: &str| config::Feed { url: format!("{base}{path}"), ..feed() };
let none = db::HttpState::default();
let text = || ctx.db.strings_for_test("SELECT description FROM entries WHERE feed_id = 'aicn'");
scan_one(&ctx, "aicn", &at("/feed"), &none, false, None).await.unwrap();
assert_eq!(text().await, ["Somebody else's text."]);
// The feed fixed, the item already stored: it was left as it was (#141).
scan_one(&ctx, "aicn", &at("/fixed"), &none, false, None).await.unwrap();
assert_eq!(text().await, ["As it should read."]);
}
#[tokio::test]
async fn nothing_from_a_feed_that_never_posted_is_an_error_and_from_one_that_has_is_not() {
let base = empty_or_feed_server().await;

View File

@@ -1024,9 +1024,22 @@ struct ListedItem {
struct Listed {
/// What the feed says it is (#130), sanitized as its items are.
description: Option<String>,
/// Why its last check failed, when it did (#142), in the words a feed's own page uses.
failing: Option<String>,
items: Vec<ListedItem>,
}
/// A feed's last error as its Directory page says it: explained once it has been failing a day, as
/// the feed's own page does, or else that the check failed; never the error as it stands, which can
/// name the feed's address, and the Directory names none. A day, so that a site down for an hour,
/// or a resolver for a minute, is not called gone (#143).
fn failure_words(err: &str, failing_for: i64) -> String {
Some(err)
.filter(|_| failing_for >= FLAG_AFTER_SECS)
.and_then(crate::feed::explain_failure)
.map_or_else(|| "Its last check failed.".to_owned(), |f| f.reason.to_owned())
}
/// A listed feed's description and latest twenty items. Only a feed the Directory lists, so a
/// guessed id reaches nothing private, as `subscribe_popular` checks.
async fn get_listed(
@@ -1041,6 +1054,9 @@ async fn get_listed(
let rows = state.ctx.db.entries_in(user.id, Some(&id), crate::db::Filter::parse("all"), None, 0, 20, &order).await?;
let mut sanitizer = feed_sanitizer();
let description = state.ctx.db.feed_description(&id).await?.map(|d| clean_description(&mut sanitizer, &d, None));
let summary = state.ctx.db.feed_summary(&id).await?;
let failing_for = summary.error_since.map_or(0, |since| crate::db::now() - since);
let failing = summary.last_error.as_deref().map(|e| failure_words(e, failing_for));
let items = rows
.into_iter()
.map(|e| ListedItem {
@@ -1052,7 +1068,7 @@ async fn get_listed(
duration: e.duration,
})
.collect();
Ok(Json(Listed { description, items }))
Ok(Json(Listed { description, failing, items }))
}
fn sort_name(p: &PopularRow) -> String {
@@ -1146,6 +1162,18 @@ impl IntoResponse for ApiError {
mod tests {
use super::*;
#[test]
fn a_failing_feed_says_why_in_the_directory_without_its_address() {
let day = FLAG_AFTER_SECS;
let nothing = "the feed's address answered with nothing at all";
assert_eq!(failure_words(nothing, day), "This address answers with nothing; the site may be gone.");
// Not before it has been failing a day, as on the feed's own page (#143).
assert_eq!(failure_words(nothing, 3600), "Its last check failed.");
// reqwest names the URL; the Directory names none.
let raw = "error sending request for url (http://daily-quests.com/comic/?feed=atom): connection closed";
assert_eq!(failure_words(raw, day), "Its last check failed.");
}
#[test]
fn a_relative_image_resolves_against_the_post() {
let mut b = feed_sanitizer();

View File

@@ -717,6 +717,7 @@ body.scan-this .fhead [data-a=scan] .i,body.scan-any .fhead [data-a=scanall] .i,
.show .meta{min-width:0;flex:1;display:flex;flex-direction:column;align-items:flex-start;gap:5px}
.show h2{margin:0 0 2px;font-size:30px;line-height:1.15;font-weight:700;letter-spacing:-.02em;overflow-wrap:anywhere}
.show .sub{margin:0;color:var(--dim);font-size:13px;font-variant-numeric:tabular-nums}
.show .dfail{color:var(--bad)}
.show .acts{margin-top:8px}
.dlink{padding:0;font-size:13px;color:var(--accent);text-align:left}
.dlink:hover{text-decoration:underline}

View File

@@ -137,7 +137,8 @@ function drawDirectory(box=$('#directory'), focus?: string){
<div class="show">${artHTML(show.image,NAME(show))}<div class="meta"><h2>${esc(NAME(show))}</h2>${
// Its category's page is one click away from here.
show.category?`<button type="button" class="dlink" data-go="${esc(show.category)}">${esc(show.subcategory||show.category)}</button>`:''}
<p class="sub">${n?`${n} ${n===1?'person here subscribes':'people here subscribe'}`:'Nobody here subscribes yet'}</p>
${n?`<p class="sub">${n} ${n===1?'person here subscribes':'people here subscribe'}</p>`:''}
<p class="sub dfail" id="dfail" hidden></p>
<div class="acts"><button class="btn ico primary" data-a="sub" title="Subscribe" aria-label="Subscribe">${ICON.plus}</button></div></div></div>
<div class="about" id="dabout" hidden><p></p><button type="button" class="dlink" aria-expanded="false" hidden>More</button></div>`
+sec(show.podcast?'Latest episodes':'Latest posts','<div class="latest" id="dshow"><p class="hint">Loading…</p></div>');
@@ -217,6 +218,8 @@ async function fillShow(box,p){
more.textContent=open?'Less':'More'; more.setAttribute('aria-expanded',String(open));
};
}
// A broken feed says so, or it reads as a quiet one (#142).
if(got.failing){ const f=$('#dfail',box); f.textContent=got.failing; f.hidden=false; }
const items=got.items;
list.innerHTML=items.length?items.map(e=>{
const name=entryName(e), text=name.derived?'':plainText(e.description).slice(0,500);
@@ -224,7 +227,8 @@ async function fillShow(box,p){
return `<article class="lep"><b>${e.link?`<a href="${esc(e.link)}" target="_blank" rel="noopener">${esc(name.text)}</a>`:esc(name.text)}</b>`
+(text?`<p>${esc(text)}</p>`:'')
+(when.length?`<small>${when.map(s=>`<span>${s}</span>`).join('<span class="dot"></span>')}</small>`:'')+'</article>';
}).join(''):`<p class="hint">Nothing read from it yet. A feed nobody here subscribes to is checked once a day.</p>`;
}).join(''):`<p class="hint">Nothing read from it yet.${p.subscribers||got.failing?''
:' A feed nobody here subscribes to is checked once a day.'}</p>`;
}
/// A category as a tile: its name, how many feeds, and the covers of its three most subscribed