Apple Podcasts' top charts into the directory weekly, and its categories checked (#252)

With apple_country set, the directory fetches Apple's top 100 podcasts and the top 50 of each of
its 19 categories for that country (itunes.apple.com/{cc}/rss/toppodcasts), looks up each
show's RSS address (/lookup, 100 ids a call, three seconds apart, as Apple asks for no more than
about twenty a minute), and stores them as the snapshot apple-podcasts with no subscribers. The
refresher fetches them like any feed; the most-subscribed chart is unchanged. Apple's primary
genre goes in as the snapshot's category, used only while a feed names none.

The same weekly run reads Apple's genre tree (ws/genres?id=26) and warns when it differs from
ipx_feed's CATEGORIES; on 2026-10-15 they were the same, all 19 and their subcategories.

A syncs table keeps when it last ran: at start if a week has passed, again weekly, an hour after
a failure. keep_instances spares apple-podcasts, which no token holds, and no token may take
the name. A trace a sync, a span each answer.

CLAUDE.md: builds one cargo command at a time, the load looked at before and after, after a
build chained with a test compile took Tower past 200.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-16 13:21:48 +00:00
parent ca73b8136f
commit cd7a9b8530
10 changed files with 362 additions and 2 deletions

View File

@@ -181,6 +181,19 @@ npx playwright test # 66 browser tests against a real daemon on fixture fe
node tests/load/run.js # k6: many people at once against a scratch daemon with 1,500 feeds
```
**Builds share Tower with everything else, production included.** On 2026-10-16 a build of the
directory followed in the same command by its tests' compile took Tower's load past 200 and the
session was stopped. So:
* One cargo command at a time: a build, then the tests, as separate commands, never chained or
run in the background beside another, and never two crates' builds at once. Do not tune `-j`.
* Look at the load before and after each (`cat /proc/loadavg`). Above about 24, Tower's twelve
threads twice over, wait for it to come down before starting another.
* Build only the crate the change is in (`cargo build -p ipx-directory`), and run only its tests
until the change is done; the whole workspace's once, at the end.
* Builds and test binaries live under `target/`, a link to `/mnt/fast`; linking through
`/mnt/user` (FUSE) once took the load to 248 on its own.
Things about the browser suite that have cost time:
* It starts **its own daemon and database** under `/tmp/ipx-ui-test`, wiped once per run. Playwright