Apple Podcasts' top charts into the directory weekly, and its categories checked (#252)
With apple_country set, the directory fetches Apple's top 100 podcasts and the top 50 of each of
its 19 categories for that country (itunes.apple.com/{cc}/rss/toppodcasts), looks up each
show's RSS address (/lookup, 100 ids a call, three seconds apart, as Apple asks for no more than
about twenty a minute), and stores them as the snapshot apple-podcasts with no subscribers. The
refresher fetches them like any feed; the most-subscribed chart is unchanged. Apple's primary
genre goes in as the snapshot's category, used only while a feed names none.
The same weekly run reads Apple's genre tree (ws/genres?id=26) and warns when it differs from
ipx_feed's CATEGORIES; on 2026-10-15 they were the same, all 19 and their subcategories.
A syncs table keeps when it last ran: at start if a week has passed, again weekly, an hour after
a failure. keep_instances spares apple-podcasts, which no token holds, and no token may take
the name. A trace a sync, a span each answer.
CLAUDE.md: builds one cargo command at a time, the load looked at before and after, after a
build chained with a test compile took Tower past 200.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -84,6 +84,11 @@ const SCHEMA: &[&str] = &[
|
||||
PRIMARY KEY (instance, feed_id)
|
||||
)",
|
||||
"CREATE INDEX IF NOT EXISTS snapshots_feed ON snapshots (feed_id)",
|
||||
// When each periodic job last finished: Apple's weekly sync (#252).
|
||||
"CREATE TABLE IF NOT EXISTS syncs (
|
||||
name TEXT PRIMARY KEY,
|
||||
at BIGINT NOT NULL
|
||||
)",
|
||||
"CREATE TABLE IF NOT EXISTS blocked (
|
||||
id TEXT PRIMARY KEY,
|
||||
url TEXT,
|
||||
@@ -164,6 +169,23 @@ impl Store {
|
||||
Ok(self.db.query_all_raw(self.stmt(sql, values)).await?)
|
||||
}
|
||||
|
||||
/// When the job `name` last finished, Unix seconds.
|
||||
pub async fn synced_at(&self, name: &str) -> Result<Option<i64>> {
|
||||
Ok(match self.rows("SELECT at FROM syncs WHERE name = $1", vec![name.into()]).await?.first() {
|
||||
Some(r) => Some(r.try_get("", "at")?),
|
||||
None => None,
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn set_synced(&self, name: &str, at: i64) -> Result<()> {
|
||||
self.exec(
|
||||
"INSERT INTO syncs (name, at) VALUES ($1, $2) ON CONFLICT (name) DO UPDATE SET at = excluded.at",
|
||||
vec![name.into(), at.into()],
|
||||
)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn ping(&self) -> Result<()> {
|
||||
self.rows("SELECT 1", vec![]).await?;
|
||||
Ok(())
|
||||
@@ -241,7 +263,8 @@ impl Store {
|
||||
.iter()
|
||||
.map(|r| r.try_get("", "instance"))
|
||||
.collect::<Result<_, _>>()?;
|
||||
for name in present.iter().filter(|n| !names.contains(n)) {
|
||||
// Apple's charts are no token's, and stay until the next sync replaces them.
|
||||
for name in present.iter().filter(|n| !names.contains(n) && n.as_str() != crate::apple::INSTANCE) {
|
||||
dropped += self.drop_instance(name).await?;
|
||||
}
|
||||
Ok(dropped)
|
||||
|
||||
Reference in New Issue
Block a user