rays
a8e1f474a6
Streaming download with progress, content sniffing that replaces the long-dead detectFileType(), HTML-body rejection, and placement by rename from an incomplete dir on the same filesystem. Filters are applied once at discovery and recorded in enclosures.state, so the download queue is the table rather than the parse result and max_new_per_check defers work instead of dropping it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPyeapneuXrCdojsaiXGbe
149 lines
8.2 KiB
Markdown
149 lines
8.2 KiB
Markdown
# Progress
|
|
|
|
Running record of what has actually landed. Newest entry first.
|
|
The full design and step list live in the plan file at
|
|
`/config/.claude/plans/i-want-to-create-playful-quiche.md`.
|
|
|
|
## Build order
|
|
|
|
- [x] **1. Repo skeleton** — git init (`main`), `cargo init --name ipx`, deps pinned, LICENSE,
|
|
README, this file.
|
|
- [x] **2. `config.rs` + `db.rs`** — TOML config structs + SQLite schema.
|
|
- [x] **3. `feed.rs`** — conditional GET, RSS-then-Atom parse, persist entries.
|
|
- [x] **4. `download.rs`** — downloads, filters, dedupe.
|
|
- [ ] **5. `retention.rs`** — oldest-first quota + age reaper, `ipx reap [--dry-run]`.
|
|
*Done when:* smoke 6 passes.
|
|
- [ ] **6. `ipc.rs` + daemon** — broadcast event bus, UDS JSON-lines server, TTL scheduler,
|
|
CLI-proxies-to-daemon. *Done when:* smoke 3 passes.
|
|
- [ ] **7. `torrent.rs`** — librqbit, seed to ratio/time, stall abort. *Done when:* smoke 5 passes.
|
|
- [ ] **8. OPML + polish** — import/export, add/rm/status, tracing setup, systemd unit, README.
|
|
|
|
## Smoke tests
|
|
|
|
1. `ipx add <feed>` + `ipx fetch` → file in `download_dir/<Show>/`, row in `enclosures`.
|
|
2. `ipx fetch` again → no re-download, feed skipped for TTL.
|
|
3. `ipx daemon &` + `nc -U $XDG_RUNTIME_DIR/ipx.sock`, send `{"cmd":"fetch"}` → JSON events;
|
|
a concurrent `ipx fetch` proxies to the daemon instead of downloading in parallel.
|
|
4. Delete a downloaded file by hand, `ipx fetch` → NOT re-downloaded.
|
|
5. Torrent enclosure → downloads, moves, stops seeding at the configured ratio/time.
|
|
6. `ipx reap --dry-run` under quota pressure → oldest-first hit list; real run flips rows to
|
|
`reaped`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 4: download.rs
|
|
|
|
`src/download.rs`: streaming download to `<download_dir>/.ipx-incomplete/` (same filesystem as the
|
|
destination, so filing it is a rename, not the original's copy-then-unlink), content sniffing,
|
|
then `place()`. Filename comes from the URL's last path segment, percent-decoded, unless
|
|
`Content-Disposition` names one (RFC 5987 `filename*=` preferred). The sanitizer keeps UTF-8 —
|
|
`latin1_to_ascii` existed because 2004 filesystems demanded ASCII — strips the same characters
|
|
`stringCleaning()` did plus control chars, and adds a real 255-byte cap the Python never had,
|
|
preserving the extension across truncation.
|
|
|
|
Sniffing replaces `detectFileType()`, which called a `typeFile` module that was already missing in
|
|
2008 and so always answered `'data'`. Two rules survive: an HTML body is a failed download (login
|
|
wall/error page), and a torrent body is a torrent whatever the MIME claimed.
|
|
|
|
Filters run once at discovery and are recorded in `enclosures.state`; the download queue is then
|
|
just "everything still `pending`", so an enclosure held back by `max_new_per_check` is picked up by
|
|
the next scan instead of being lost. Keywords are OR'd across keywords and AND'd within one — the
|
|
original's nested loop let a later keyword silently undo an earlier miss.
|
|
|
|
Verified: `cargo test` 16/16. Smoke against a local server, five enclosures, each filter path hit:
|
|
`ep1.mp3 -> done`, `ep2.mp3 -> skipped (explicit)`, `ep2.mp3?v=3 -> skipped (no keyword match)`,
|
|
`paywall.html -> error (HTML page, not media)`, `ep5.torrent -> torrent (deferred to step 7)`.
|
|
Smoke 2 and 4 pass, and because a plain rerun 304s before parsing, dedupe was proved separately by
|
|
clearing the stored etag/last-modified and re-parsing all five entries: 0 downloaded, hand-deleted
|
|
file not refetched, `.ipx-incomplete` left empty.
|
|
|
|
Known wart: `ipx list` counts `path IS NOT NULL`, so a hand-deleted file still reads as downloaded.
|
|
Reconciling rows against the filesystem belongs in step 5.
|
|
Next: step 5 — `retention.rs`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 3: feed.rs
|
|
|
|
`src/feed.rs`: conditional GET (`If-None-Match` + `If-Modified-Since`, optional basic auth) and a
|
|
RSS-first / Atom-fallback parser normalising both into `ParsedFeed`/`Entry`/`Enclosure`. Feed-level
|
|
`itunes:explicit` overrides the entry level, as the original did. `<ttl>` is captured. RSS
|
|
`content:encoded` wins over `description`. Atom enclosures come only from `rel="enclosure"` links.
|
|
|
|
GUID: the original hashed the title or description when no guid existed; here the chain is
|
|
guid → permalink → enclosure URL → title, all stable identifiers, so no hashing and no MD5
|
|
dependency. An entry with none of them has nothing to download and is dropped.
|
|
|
|
`db.rs` gained `http_state`, `record_feed`, `touch_feed`, `set_feed_error`, `record_entry`,
|
|
`record_enclosure`. A changed title/description flips `read` back to 0 — what the original's
|
|
textDiff was ultimately for, minus the `<ins>`/`<del>` markup, which belongs in the UI.
|
|
|
|
`main.rs` gained `ipx fetch [FEED] [--force]`. A failing feed records its error and the scan
|
|
continues.
|
|
|
|
Verified: `cargo test` 10/10. Gate met against a local `python3 -m http.server` serving the
|
|
fixtures — first run inserted 4 entries + 4 enclosures across an RSS and an Atom feed; second run
|
|
showed both skip paths, `atomcast: not modified` (304) and `testcast: not due for 45m` (the feed's
|
|
own ttl=45 beating `interval_mins = 0`).
|
|
|
|
Deferred: nothing downloads yet — enclosure rows land in state `pending`. That is step 4.
|
|
Next: step 4 — `download.rs`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 2: config.rs + db.rs
|
|
|
|
`src/config.rs`: serde structs for `[general]`, `[torrent]` and `[feeds.<id>]` with defaults, `~`
|
|
expansion, `IPX_CONFIG` / `IPX_DATA_DIR` overrides, `save()` at mode 0600, `Feed::password()`
|
|
(`password_env` beats a literal `password`), `Torrent::ports()` parsing `"6881-6889"`. A missing
|
|
config file loads as an empty one so a fresh install works. Retention defaults are 0/0
|
|
(unlimited, keep forever) — nothing gets deleted until Ray asks for it.
|
|
|
|
`src/db.rs`: schema exactly as planned, WAL + `busy_timeout`, `Db::open()` idempotent, connection
|
|
behind a `Mutex`, `feed_summary()` for `list`, `now()` helper. `enclosures.url` is UNIQUE — the
|
|
dedupe key that replaces `history.dat`.
|
|
|
|
`src/main.rs`: clap skeleton with `ipx list`. Only the subcommands that work exist; the rest arrive
|
|
with their steps.
|
|
|
|
Verified: `cargo test` 5/5 green (config defaults, port-range fallback incl. reversed range,
|
|
password_env precedence, schema idempotency, enclosure-url uniqueness). Gate met — `ipx --config
|
|
<scratch> list` printed both feeds and created `state.db` once across two runs.
|
|
|
|
Deferred: nothing. Two dead-code warnings (`Config::save`, `Feed::password`) are expected; steps 3
|
|
and 8 consume them.
|
|
Next: step 3 — `feed.rs`.
|
|
|
|
Also added `chrono` 0.4 (std, clock) for RFC-2822 pubDate parsing in step 3.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 1: repo skeleton
|
|
|
|
Repo created at `/src/ipodderx-rs`, default branch `main`, `cargo init --name ipx` (edition 2024,
|
|
rustc 1.95.0). LICENSE (MIT, carrying the 2010 copyright), README with the lineage note, and this
|
|
file. Hello-world `main.rs` builds.
|
|
|
|
Dependency versions pinned today — the later steps are written against these APIs:
|
|
|
|
| crate | version | notes |
|
|
|---|---|---|
|
|
| tokio | 1.53.1 | rt-multi-thread, macros, fs, io-util, net, sync, time, signal |
|
|
| reqwest | 0.13.5 | `default-features = false`; features **`rustls`** (not `rustls-tls` — renamed in 0.13), http2, gzip, stream, json, charset, **`system-proxy`** (env-var proxy pickup is opt-in in 0.13) |
|
|
| rss | 2.1.1 | default features; the `with-syndication` feature name in my notes does not exist — Atom is handled by the separate crate |
|
|
| atom_syndication | 0.12.10 | |
|
|
| rusqlite | 0.40.2 | `bundled` |
|
|
| librqbit | 9.0.1 | `default-features = false`; features `rust-tls`, `http-api-client`. The default `default-tls` feature pulls reqwest/native-tls **and** an OpenSSL sha1 backend (`crypto-hash`), which fails to build without pkg-config/OpenSSL headers. API not yet exercised — step 7 |
|
|
| serde 1.0.229 / serde_json 1.0.151 / toml 1.1.5 | | |
|
|
| clap | 4.6.6 | derive |
|
|
| infer 0.22.0 / dirs 7.0.0 / anyhow 1.0.104 / opml 1.1.6 | | |
|
|
| tracing 0.1.44 / tracing-subscriber 0.3.23 | | env-filter |
|
|
|
|
Deferred: nothing.
|
|
|
|
Gotcha worth keeping: three feature names in the plan were wrong against current crate versions —
|
|
`reqwest/rustls-tls` is now `rustls`, env-var proxy support moved behind `system-proxy`, and
|
|
`rss/with-syndication` does not exist. `librqbit`'s default features drag in OpenSSL; `rust-tls`
|
|
is the fix. Whole tree is rustls-only now, no C TLS dependency.
|
|
Next: step 2 — `config.rs` + `db.rs`. (done)
|