Compressed answers, a directory twice as patient, fewer whole re-reads (#239, #242, #243, #244)

Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.

- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
  back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
  something is missing a read would fill or the check is forced; and a feed without a schedule
  of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
  failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).

#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 13:54:00 +00:00
parent dbfe7918ac
commit 8f19d88fc9
17 changed files with 221 additions and 18 deletions

View File

@@ -302,8 +302,8 @@ then; a steady stream is worth an issue, since Cloudflare can rate-limit it. A s
`refused_private` is not 0 means an ipx server tried to send a feed that looks private: find which
in that server's log, since the directory refuses it but the sender should not have sent it. Many
`feed check failed` lines for one feed are the publisher's, not the directory's. Each feed is
checked every six hours, 25 at a time, so `feeds checked` over a day should be about four times
the feeds listed; none for over six hours means the refresher is stuck. `could not store a feed
checked every twelve hours (#243), 25 at a time, so `feeds checked` over a day should be about
twice the feeds listed; none for over twelve hours means the refresher is stuck. `could not store a feed
check` or `could not read which feeds are due` is its database (`ipx_directory` on the
`databases` project's Postgres). A config that does not load (`config changed but does not
load`) keeps the old tokens: worth fixing before someone's new token is needed.

67
Cargo.lock generated
View File

@@ -17,6 +17,21 @@ dependencies = [
"memchr",
]
[[package]]
name = "alloc-no-stdlib"
version = "2.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3"
[[package]]
name = "alloc-stdlib"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195"
dependencies = [
"alloc-no-stdlib",
]
[[package]]
name = "allocator-api2"
version = "0.2.21"
@@ -445,6 +460,27 @@ dependencies = [
"syn 3.0.5",
]
[[package]]
name = "brotli"
version = "8.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5cc91aac060a7a1e25823bdccbfb6af1875b88f17c6daac97894eed8207166b3"
dependencies = [
"alloc-no-stdlib",
"alloc-stdlib",
"brotli-decompressor",
]
[[package]]
name = "brotli-decompressor"
version = "5.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583"
dependencies = [
"alloc-no-stdlib",
"alloc-stdlib",
]
[[package]]
name = "bs58"
version = "0.5.1"
@@ -620,9 +656,12 @@ version = "0.4.41"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "58a6d0db8759036a783bc7c3f7a07f8cef3bf9470eb1db3bc86e8bcd1c5d0fe8"
dependencies = [
"brotli",
"compression-core",
"flate2",
"memchr",
"zstd",
"zstd-safe",
]
[[package]]
@@ -5458,3 +5497,31 @@ name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
[[package]]
name = "zstd"
version = "0.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "057cfd910cfac363a0ada849592624b4c9ff2e10bef504c3433810d78ed96f93"
dependencies = [
"zstd-safe",
]
[[package]]
name = "zstd-safe"
version = "8.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bdd44c6a7284e91f3717755b24315a302edd9153a01f753c3cba3d765e8eafac"
dependencies = [
"zstd-sys",
]
[[package]]
name = "zstd-sys"
version = "2.1.1+zstd.1.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aeec9eaf2dffbbd09201e23bd0ffcbaa33bb8e9266a10734fd7ed90a85eca078"
dependencies = [
"cc",
"pkg-config",
]

View File

@@ -27,7 +27,7 @@ opml = "1.1.6"
mimalloc = "0.1.52"
percent-encoding = "2.3.2"
quick-xml = { version = "0.42.0", features = ["escape-html"] }
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] }
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "brotli", "zstd", "stream", "json", "charset", "system-proxy"] }
rss = "2.1.1"
sea-orm = { version = "2.0.3", default-features = false, features = ["sqlx-sqlite", "sqlx-postgres", "runtime-tokio-rustls", "macros", "with-json", "sqlite-use-returning-for-3_35"] }
serde = { version = "1.0.229", features = ["derive"] }

View File

@@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- Each feed is checked every twelve hours instead of six (#243).
- Feeds are fetched accepting brotli and zstd as well as gzip (#244).
## [1.1.1] - 2026-10-15
### Changed

View File

@@ -117,8 +117,8 @@ pub struct Blocked {
pub const MAX_SNAPSHOT: usize = 20_000;
/// How often each feed is fetched again. Metadata changes slowly, and every server subscribed
/// to a feed fetches it itself anyway.
pub const REFRESH_EVERY: i64 = 6 * 3600;
/// to a feed fetches it itself anyway: every six hours was 3.9 GB of feeds a day (#243).
pub const REFRESH_EVERY: i64 = 12 * 3600;
pub struct Directory {
pub store: Store,

View File

@@ -12,6 +12,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- A feed that sends the same body again, as Patreon and Substack do whatever they are asked, is taken as unchanged instead of read again, and, without a schedule of its own, checked less often the longer it stays the same, up to every four hours, back on schedule when it changes (#242).
- Feeds are fetched accepting brotli and zstd as well as gzip (#244).
## [1.2.0] - 2026-10-15
### Added

View File

@@ -95,6 +95,11 @@ async fn create_missing(orm: &sea_orm::DatabaseConnection) -> Result<()> {
orm.execute_unprepared(sql).await.with_context(|| sql.to_owned())?;
}
}
if orm.execute_unprepared("SELECT body_hash FROM feeds LIMIT 0").await.is_err() {
for sql in ["ALTER TABLE feeds ADD COLUMN body_hash TEXT", "ALTER TABLE feeds ADD COLUMN same_since BIGINT"] {
orm.execute_unprepared(sql).await.with_context(|| sql.to_owned())?;
}
}
if orm.execute_unprepared("SELECT pause_ms FROM deadair_prefs LIMIT 0").await.is_err() {
let sql = "ALTER TABLE deadair_prefs ADD COLUMN pause_ms BIGINT";
orm.execute_unprepared(sql).await.with_context(|| sql.to_owned())?;
@@ -294,6 +299,8 @@ pub struct FeedSummary {
pub last_error: Option<String>,
/// When this run of failures began; see the `error_since` column.
pub error_since: Option<i64>,
/// Since when the feed has sent the same body (#242).
pub same_since: Option<i64>,
pub entries: i64,
pub downloaded: i64,
}
@@ -499,6 +506,7 @@ impl Db {
last_error: f.last_error,
orphaned: f.orphaned,
error_since: f.error_since,
same_since: f.same_since,
category: f.category,
},
ttl_mins: f.ttl_mins.map(|t| t.max(0) as u64),
@@ -530,6 +538,9 @@ pub struct HttpState {
pub ttl_mins: Option<u64>,
/// When the feed's current run of failures began, for backing off (`due_after`).
pub error_since: Option<i64>,
/// The last body's hash, and since when it has been that body (#242).
pub body_hash: Option<String>,
pub same_since: Option<i64>,
}
impl From<feeds::Model> for HttpState {
@@ -540,6 +551,8 @@ impl From<feeds::Model> for HttpState {
last_checked: f.last_checked,
ttl_mins: f.ttl_mins.map(|t| t.max(0) as u64),
error_since: f.error_since,
body_hash: f.body_hash,
same_since: f.same_since,
}
}
}
@@ -606,6 +619,16 @@ impl Db {
}
/// 304, or any other poll that produced no new data: only the clock moves.
/// A body read whole that was not the last one: its hash, and that it changed now (#242).
pub async fn set_body(&self, feed_id: &str, hash: &str) -> Result<()> {
self.exec(
"UPDATE feeds SET body_hash = $2, same_since = $3 WHERE id = $1",
vec![feed_id.into(), hash.into(), now().into()],
)
.await?;
Ok(())
}
pub async fn touch_feed(&self, feed_id: &str, url: &str) -> Result<()> {
self.exec(
"INSERT INTO feeds (id, url, last_checked) VALUES ($1, $2, $3)
@@ -2581,6 +2604,23 @@ mod tests {
assert_eq!(db.http_state("f").await.unwrap().etag.as_deref(), Some("\"b\""));
}
#[tokio::test]
async fn a_database_from_before_body_hashes_gains_the_columns() {
let db = Db::memory().await.unwrap();
db.record_feed("f", "https://x/f", Some("F"), None, None, None, None, None, None).await.unwrap();
db.set_body("f", "abc").await.unwrap();
assert_eq!(db.http_state("f").await.unwrap().body_hash.as_deref(), Some("abc"));
// A database from before #242.
for sql in ["ALTER TABLE feeds DROP COLUMN body_hash", "ALTER TABLE feeds DROP COLUMN same_since"] {
db.exec_for_test(sql).await.unwrap();
}
create_missing(&db.orm).await.unwrap();
let st = db.http_state("f").await.unwrap();
assert_eq!((st.body_hash, st.same_since), (None, None), "every feed is read whole once more");
db.set_body("f", "def").await.unwrap();
assert_eq!(db.http_state("f").await.unwrap().body_hash.as_deref(), Some("def"));
}
#[tokio::test]
async fn every_sort_column_runs_and_orders_both_ways() {
let db = Db::memory().await.unwrap();

View File

@@ -46,6 +46,12 @@ pub mod feeds {
/// are re-derived instead. Customising one promotes it to config.
#[sea_orm(default_value = false)]
pub managed: bool,
/// SHA-256 of the body last read, in hex: the same body again is read as "not modified"
/// (#242). Added after the table was, so `create_missing` adds it, and `same_since`.
#[sea_orm(column_type = "Text", nullable)]
pub body_hash: Option<String>,
/// When the body last changed: the longer ago, the less often the feed is checked (#242).
pub same_since: Option<i64>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]

View File

@@ -1531,7 +1531,7 @@ async fn retire_stranded(ctx: &Ctx) -> Result<usize> {
pub fn due_at(cfg: &config::Config, feed: &config::Feed, state: &db::HttpState, subscribed: bool) -> Option<i64> {
let last = state.last_checked?;
let floor = if subscribed { 0 } else { 86_400 };
Some(last + due_after(cfg, feed, state.ttl_mins, state.error_since, last).max(floor) as i64)
Some(last + due_after(cfg, feed, state.ttl_mins, state.error_since, state.same_since, last).max(floor) as i64)
}
/// How long the daemon may sleep before a feed is due: until the earliest one, at least 30 s and
@@ -1577,14 +1577,17 @@ pub fn due_after(
feed: &config::Feed,
ttl_mins: Option<u64>,
error_since: Option<i64>,
same_since: Option<i64>,
last_checked: i64,
) -> u64 {
let mins = match feed.schedule.as_deref().and_then(config::parse_interval) {
Some(explicit) => explicit,
None => ttl_mins.unwrap_or(0).max(cfg.general.interval()),
let explicit = feed.schedule.as_deref().and_then(config::parse_interval);
let usual = match explicit {
Some(explicit) => explicit * 60,
// A schedule someone set is theirs; otherwise a feed long the same is checked less.
None => quiet(ttl_mins.unwrap_or(0).max(cfg.general.interval()) * 60, same_since.map(|s| (last_checked - s).max(0) as u64)),
};
let failing_for = error_since.map(|since| (last_checked - since).max(0) as u64);
backoff(mins * 60, failing_for)
backoff(usual, failing_for)
}
/// A failing feed waits as long as it has been failing, so the wait doubles with each failure
@@ -1592,6 +1595,17 @@ pub fn due_after(
/// never more than a day. Retried hourly, a feed dead for good cost a request, a warning and scan
/// time every hour (#99). The first success clears error_since, and with it the backoff; a
/// refresh someone asks for is not held back by it.
/// A feed whose body has been the same for a while is checked less often, a twelfth of how long
/// it has been the same, up to four hours, and on its schedule again once it changes: Patreon and
/// Substack send the whole feed at every check, ignoring "not modified", 2.6 GB a day (#242).
fn quiet(usual: u64, same_for: Option<u64>) -> u64 {
const CEILING: u64 = 4 * 3600;
match same_for {
Some(f) => usual.max((f / 12).min(CEILING)),
None => usual,
}
}
fn backoff(usual: u64, failing_for: Option<u64>) -> u64 {
const CEILING: u64 = 86_400;
match failing_for {
@@ -1721,6 +1735,18 @@ async fn scan_one(
return Ok(Outcome::Empty);
}
// The same body as last time, from a server that ignores "not modified": nothing to read,
// as a 304 would have said (#242). Not when something is missing that a read would fill, as
// above, nor when someone asked for a full check.
let hash = {
use sha2::Digest as _;
sha2::Sha256::digest(&bytes).iter().map(|b| format!("{b:02x}")).collect::<String>()
};
if !force && state.body_hash.as_deref() == Some(hash.as_str()) && stored.entries > 0 && stored.image.is_some() {
ctx.db.touch_feed(id, &feed_cfg.url).await?;
return Ok(Outcome::NotModified);
}
// A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml"
// URL; sniffing the body also catches one served from a URL without that extension.
if feed::is_opml(&bytes) {
@@ -1792,6 +1818,9 @@ async fn scan_one(
parsed.category.as_deref(),
parsed.description.as_deref(),
).await?;
if state.body_hash.as_deref() != Some(hash.as_str()) {
ctx.db.set_body(id, &hash).await?;
}
// Asked on a feed's first read, nothing of it stored yet, or when someone forces a refresh:
// each call costs, and a feed already here without a category can wait for a forced one.
// The answer becomes the catalogue's category, so a feed is not asked twice.
@@ -2389,6 +2418,40 @@ mod tests {
assert_eq!(at(Some(1000), false), Some(1000 + 86_400)); // listed, nobody subscribed
}
#[test]
fn a_feed_long_the_same_is_checked_less_and_a_schedule_set_wins() {
let (hour, day) = (3600u64, 86_400u64);
assert_eq!(quiet(hour, None), hour);
assert_eq!(quiet(hour, Some(6 * hour)), hour, "a few hours the same changes nothing");
assert_eq!(quiet(hour, Some(day)), 2 * hour, "a day the same: every two hours");
assert_eq!(quiet(hour, Some(30 * day)), 4 * hour, "never past four hours");
assert_eq!(quiet(12 * hour, Some(30 * day)), 12 * hour, "never sooner than its schedule");
let cfg = config::Config::default();
let same = |f: &config::Feed| due_after(&cfg, f, None, None, Some(0), 3 * day as i64);
assert_eq!(same(&feed()), 4 * hour);
let set = config::Feed { schedule: Some("every 1h".into()), ..feed() };
assert_eq!(same(&set), hour, "someone's own schedule wins");
}
#[tokio::test]
async fn the_same_body_again_is_not_modified_unless_a_check_is_forced() {
let base = empty_or_feed_server().await;
let ctx = Arc::new(test_ctx(config::Config::default()).await);
let cfg = config::Feed { url: format!("{base}/feed"), ..feed() };
assert!(matches!(scan_one(&ctx, "same", &cfg, &db::HttpState::default(), false, None).await, Ok(Outcome::Feed(_))));
let state = ctx.db.http_state("same").await.unwrap();
let since = state.same_since.expect("when it was first read");
assert!(state.body_hash.is_some());
// Patreon's way: the whole feed again, no validator honoured (#242).
assert!(matches!(scan_one(&ctx, "same", &cfg, &state, false, None).await, Ok(Outcome::NotModified)));
assert_eq!(ctx.db.http_state("same").await.unwrap().same_since, Some(since), "still the same since then");
assert!(matches!(scan_one(&ctx, "same", &cfg, &state, true, None).await, Ok(Outcome::Feed(_))), "a forced check reads it");
// Changed: read, and the same from now.
let moved = config::Feed { url: format!("{base}/fixed"), ..feed() };
assert!(matches!(scan_one(&ctx, "same", &moved, &state, false, None).await, Ok(Outcome::Feed(_))));
assert_ne!(ctx.db.http_state("same").await.unwrap().body_hash, state.body_hash);
}
#[test]
fn a_listed_feed_nobody_subscribes_to_downloads_nothing() {
let mut f = feed();

View File

@@ -770,11 +770,11 @@ async fn feed_rows(state: &WebState, user: &crate::db::User, only: Option<&str>)
.schedule
.as_deref()
.and_then(crate::config::parse_interval),
every_mins: crate::due_after(&cfg, feed, ttl_mins, None, 0) / 60,
every_mins: crate::due_after(&cfg, feed, ttl_mins, None, None, 0) / 60,
last_checked: s.last_checked,
next_check: s
.last_checked
.map(|t| t + crate::due_after(&cfg, feed, ttl_mins, s.error_since, t) as i64),
.map(|t| t + crate::due_after(&cfg, feed, ttl_mins, s.error_since, s.same_since, t) as i64),
failing: s
.error_since
.filter(|since| crate::db::now() - since >= FLAG_AFTER_SECS)

View File

@@ -60,7 +60,10 @@ and the files, and no page.
## A scan
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
Without a schedule of its own, a feed whose body has been the same for a while waits longer:
a twelfth of how long it has been the same, up to four hours (#242).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there, and so does a
body identical to the last one (`body_hash`), from a server that ignores the validators.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
re-derived into the database each scan, never written to config.toml. A Patreon creator link
(a token, no `show=`) with more than one show is treated the same way, before any fetch: its
@@ -81,7 +84,8 @@ and the files, and no page.
```
feeds id, url, title, image, category, description, etag, last_modified,
last_checked, ttl_mins, last_error, orphaned, group_id, managed
last_checked, ttl_mins, last_error, orphaned, group_id, managed,
body_hash, same_since
entries feed_id, guid, title, link, published, description, first_seen,
image, duration, episode, season PK (feed_id, guid)
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,

View File

@@ -52,7 +52,10 @@ art_cache_mb = 500 # artwork kept on disk. 0 = none
database as described above; `download_dir`, `socket` and `organize` stay in config.toml.
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
be polled *less* often, and a per-feed `schedule` overrides both. A feed that keeps failing
be polled *less* often, and a per-feed `schedule` overrides both. Without a schedule of its own,
a feed that sends the same body check after check (Patreon, Substack, which ignore "not
modified") is checked less often the longer it stays the same, up to every four hours, and is
back on schedule once it changes. A feed that keeps failing
waits as long as it has been failing before the next try, up to a day, and is back on schedule
after its first success. A moment's failure to look its host up does not count: the network, not
the feed, failed, and it is tried again in ten minutes. Admin-only from the UI.

View File

@@ -2,7 +2,7 @@
`ipx-directory` is a catalogue of public podcasts and blogs that ipx servers share (#204). An ipx
server sends it the public feeds its people subscribe to, and how many of them do. The directory
fetches each feed's title, artwork, category, description and latest twenty items every six hours,
fetches each feed's title, artwork, category, description and latest twenty items every twelve hours,
and any server holding a token can browse it. It never downloads a file, and it never stores
one's address: every ipx server still fetches its own feeds.

View File

@@ -279,7 +279,7 @@ row("ipx-directory")
stat("Snapshots", loki(f'sum(count_over_time({DIR} | message="snapshot" [$__range])) or vector(0)', instant=True), 0,
desc="ipx sends one when what it shares changes, and every half hour (#211).")
stat("Feed checks", loki(f'sum(sum_over_time({DIR} | message="feeds checked" | unwrap checked [$__range])) or vector(0)', instant=True), 4,
desc="Each feed every six hours, 25 at a time.")
desc="Each feed every twelve hours, five at a time.")
stat("Failed feed checks", loki(f'sum(count_over_time({DIR} | message="feed check failed" [$__range])) or vector(0)', instant=True), 8,
thresholds=GOOD_BAD, desc="Mostly the publishers': a feed gone or moved.")
stat("Requests", loki(f'sum(count_over_time({DIR_HTTP} [$__range])) or vector(0)', instant=True), 12)

View File

@@ -29,4 +29,4 @@
bind = "0.0.0.0:8100"
database_url_env = "DIRECTORY_DATABASE_URL"</code></pre>
<pre><code>docker exec ipx-directory ipx-directory token new my-server --access rw</code></pre>
<p>That prints the token, to give to the server, and the lines to add to <code>directory.toml</code>, which keeps only its SHA-256. The directory reads the file again within a minute, so adding or revoking a token needs no restart. It checks each podcast every six hours.</p>
<p>That prints the token, to give to the server, and the lines to add to <code>directory.toml</code>, which keeps only its SHA-256. The directory reads the file again within a minute, so adding or revoking a token needs no restart. It checks each podcast every twelve hours.</p>

View File

@@ -11,6 +11,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- The page and the API's answers are sent compressed, zstd or gzip, which the feed list and item lists, over 100 KB each, were not; live updates and audio are not compressed (#239).
## [1.0.2] - 2026-10-15
### Fixed

View File

@@ -57,6 +57,12 @@ http://{$WEB_BIND::8080} {
}
}
# Compressed, zstd or gzip as the browser takes them: the feed list and a page of items went
# out at over 100 KB each (#239). Not the live updates, which a compressor would hold back
# until it had enough to send, nor /media, which is audio and video already compressed.
@compress not path /api/events /media/*
encode @compress zstd gzip
# Before anything else, so no page or API call sees them: only ipx sets these.
@forged not remote_ip {$TUNNEL_PEER:255.255.255.255}
request_header @forged -Cf-Access-Authenticated-User-Email