Apple Podcasts' top charts into the directory weekly, and its categories checked (#252)

With apple_country set, the directory fetches Apple's top 100 podcasts and the top 50 of each of
its 19 categories for that country (itunes.apple.com/{cc}/rss/toppodcasts), looks up each
show's RSS address (/lookup, 100 ids a call, three seconds apart, as Apple asks for no more than
about twenty a minute), and stores them as the snapshot apple-podcasts with no subscribers. The
refresher fetches them like any feed; the most-subscribed chart is unchanged. Apple's primary
genre goes in as the snapshot's category, used only while a feed names none.

The same weekly run reads Apple's genre tree (ws/genres?id=26) and warns when it differs from
ipx_feed's CATEGORIES; on 2026-10-15 they were the same, all 19 and their subcategories.

A syncs table keeps when it last ran: at start if a week has passed, again weekly, an hour after
a failure. keep_instances spares apple-podcasts, which no token holds, and no token may take
the name. A trace a sync, a span each answer.

CLAUDE.md: builds one cargo command at a time, the load looked at before and after, after a
build chained with a test compile took Tower past 200.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-16 13:21:48 +00:00
parent ca73b8136f
commit cd7a9b8530
10 changed files with 362 additions and 2 deletions

View File

@@ -84,6 +84,11 @@ const SCHEMA: &[&str] = &[
PRIMARY KEY (instance, feed_id)
)",
"CREATE INDEX IF NOT EXISTS snapshots_feed ON snapshots (feed_id)",
// When each periodic job last finished: Apple's weekly sync (#252).
"CREATE TABLE IF NOT EXISTS syncs (
name TEXT PRIMARY KEY,
at BIGINT NOT NULL
)",
"CREATE TABLE IF NOT EXISTS blocked (
id TEXT PRIMARY KEY,
url TEXT,
@@ -164,6 +169,23 @@ impl Store {
Ok(self.db.query_all_raw(self.stmt(sql, values)).await?)
}
/// When the job `name` last finished, Unix seconds.
pub async fn synced_at(&self, name: &str) -> Result<Option<i64>> {
Ok(match self.rows("SELECT at FROM syncs WHERE name = $1", vec![name.into()]).await?.first() {
Some(r) => Some(r.try_get("", "at")?),
None => None,
})
}
pub async fn set_synced(&self, name: &str, at: i64) -> Result<()> {
self.exec(
"INSERT INTO syncs (name, at) VALUES ($1, $2) ON CONFLICT (name) DO UPDATE SET at = excluded.at",
vec![name.into(), at.into()],
)
.await?;
Ok(())
}
pub async fn ping(&self) -> Result<()> {
self.rows("SELECT 1", vec![]).await?;
Ok(())
@@ -241,7 +263,8 @@ impl Store {
.iter()
.map(|r| r.try_get("", "instance"))
.collect::<Result<_, _>>()?;
for name in present.iter().filter(|n| !names.contains(n)) {
// Apple's charts are no token's, and stay until the next sync replaces them.
for name in present.iter().filter(|n| !names.contains(n) && n.as_str() != crate::apple::INSTANCE) {
dropped += self.drop_instance(name).await?;
}
Ok(dropped)