play() set read=1 the instant playback began. The default view is the Unread tab, so pressing play removed the episode from the list being looked at, which reads as the episode going missing. All four affected rows had position=0: started, never listened to. Read is now set on 'ended' or past 90% of the duration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPyeapneuXrCdojsaiXGbe
490 lines
28 KiB
Markdown
490 lines
28 KiB
Markdown
# Progress
|
|
|
|
Running record of what has actually landed. Newest entry first.
|
|
The full design and step list live in the plan file at
|
|
`/config/.claude/plans/i-want-to-create-playful-quiche.md`.
|
|
|
|
## Build order
|
|
|
|
- [x] **1. Repo skeleton** — git init (`main`), `cargo init --name ipx`, deps pinned, LICENSE,
|
|
README, this file.
|
|
- [x] **2. `config.rs` + `db.rs`** — TOML config structs + SQLite schema.
|
|
- [x] **3. `feed.rs`** — conditional GET, RSS-then-Atom parse, persist entries.
|
|
- [x] **4. `download.rs`** — downloads, filters, dedupe.
|
|
- [x] **5. `retention.rs`** — oldest-first quota + age reaper.
|
|
- [x] **6. `ipc.rs` + daemon** — UDS JSON-lines server, TTL scheduler, CLI-proxies-to-daemon.
|
|
- [x] **7. `torrent.rs`** — librqbit, seed to ratio/time, stall abort. (swarm download unverified —
|
|
see the step 7 entry)
|
|
- [x] **8. OPML + polish** — import/export, add/rm/status, tracing setup, systemd units, README.
|
|
|
|
### Phase 2 — web front end
|
|
|
|
Decided with Ray: axum serving plain HTML/JS (no WASM toolchain), running **inside the daemon**
|
|
process so it reads SQLite and the event bus directly, LAN-bindable with a shared token.
|
|
|
|
- [x] **9. Config hot-reload + web skeleton.** `Ctx.cfg` becomes `RwLock<Arc<Config>>` so the UI can
|
|
edit feeds without a daemon restart. `[web]` config section (enabled/bind/token, token
|
|
auto-generated and saved on first run). axum server started by `ipx daemon`, token checked by
|
|
middleware, `?token=` sets a cookie so `<audio>` requests authenticate too.
|
|
*Done when:* `ipx daemon` serves a page on the configured bind, and a wrong token gets 401.
|
|
- [x] **10. Browsing.** `/api/feeds`, `/api/feeds/:id/entries`, entry detail. Descriptions are
|
|
untrusted feed HTML — sanitized with `ammonia` before they reach the page.
|
|
*Done when:* the Glass Cannon feed's 131 entries browse and read correctly.
|
|
- [x] **11. Media actions.** Range-request audio streaming (`tower-http` ServeFile) so seeking
|
|
works, download-on-demand for a pending enclosure, delete a file, mark read/flagged.
|
|
*Done when:* an episode plays and seeks in a browser, and delete reaps the row.
|
|
- [x] **12. Feed configuration.** Add/remove feeds and edit folder, keywords, allow_explicit,
|
|
auto_download, max_new_per_check from the UI, written back to config.toml and hot-reloaded.
|
|
*Done when:* flipping allow_explicit in the UI takes effect on the next scan with no restart.
|
|
- [x] **13. Live progress + polish.** SSE from the existing broadcast bus so downloads show live.
|
|
README section, screenshot-free usage notes.
|
|
*Done when:* starting a fetch from the UI shows progress advancing without a reload.
|
|
|
|
Note: `read`/`flagged` finally get a writer here. Retention orders by them (see the step 5 entry),
|
|
and until now nothing set them.
|
|
|
|
## Smoke tests
|
|
|
|
1. `ipx add <feed>` + `ipx fetch` → file in `download_dir/<Show>/`, row in `enclosures`.
|
|
2. `ipx fetch` again → no re-download, feed skipped for TTL.
|
|
3. `ipx daemon &` + `nc -U $XDG_RUNTIME_DIR/ipx.sock`, send `{"cmd":"fetch"}` → JSON events;
|
|
a concurrent `ipx fetch` proxies to the daemon instead of downloading in parallel.
|
|
4. Delete a downloaded file by hand, `ipx fetch` → NOT re-downloaded.
|
|
5. Torrent enclosure → downloads, moves, stops seeding at the configured ratio/time.
|
|
6. `ipx reap --dry-run` under quota pressure → oldest-first hit list; real run flips rows to
|
|
`reaped`.
|
|
|
|
---
|
|
|
|
## 2026-09-10 — Fixed: pressing play made an episode vanish from the list
|
|
|
|
Reported as "where did Session Zero go, and why do parts share a number?".
|
|
|
|
Two separate things, one of them a real bug.
|
|
|
|
**The bug.** `play()` marked an episode read the moment playback started. The default view is the
|
|
Unread tab, so pressing play removed the episode from the list you were looking at — indistinguishable
|
|
from it going missing. Confirmed from the data: all four read entries had `position = 0`, meaning
|
|
playback never passed the 10-second save threshold. They had been started, never listened to.
|
|
|
|
Marking read now happens when an episode is actually consumed — on `ended`, or once playback passes
|
|
90% — never on play. Two earlier PROGRESS entries described mark-on-play as a deliberate feature;
|
|
it was a design error, and this supersedes them.
|
|
|
|
Also restored the four affected rows to unread, since their read flag was purely an artifact of the
|
|
bug (`UPDATE entries SET read=0 WHERE read=1 AND position=0` — no genuinely-played episode could
|
|
match, since playing one writes a position).
|
|
|
|
**Not a bug: repeated episode numbers.** Each story part ships as two items — the main episode and a
|
|
shorter `Junk in the Trunk` companion — with distinct GUIDs, durations and files, and the publisher
|
|
gives both the same `itunes:episode`. Zero duplicated titles across all 131 entries. `Session Zero`
|
|
additionally carries a season with no episode number and drops the "Part N" naming, so it does not
|
|
match the pattern its siblings follow.
|
|
|
|
Verified: the arc shows 11/11 items under the default Unread view again.
|
|
|
|
---
|
|
|
|
## 2026-09-10 — Titles: RSS `<title>` is the only source
|
|
|
|
Confirmed against the live feed rather than assumed: 131 feed items, 131 stored, **0 mismatches** —
|
|
entry titles are byte-for-byte what the RSS `<title>` publishes, separators included. `feed.rs` never
|
|
consults `itunes:title`, and season/episode live in their own columns, rendered as chips rather than
|
|
folded into the title.
|
|
|
|
Pinned with a test: a fixture whose `itunes:title` differs from its `<title>` must still yield the
|
|
RSS title, and an item with `itunes:season` but no `itunes:episode` keeps a null episode instead of
|
|
inventing one.
|
|
|
|
That null-episode case is real in the wild — this feed's `Music from a Darkened Room | Session Zero`
|
|
carries S8 with no episode number, while Parts 1-5 get E1-E5, and it drops the "Part N" convention
|
|
its siblings use. The arc is 11 items: Session Zero plus five parts, each part being a main episode
|
|
and its shorter `Junk in the Trunk` companion.
|
|
|
|
---
|
|
|
|
## 2026-09-10 — Phase 3: full-featured UI
|
|
|
|
Rewrite of `web/index.html` (~714 lines) plus the backend it needed.
|
|
|
|
**New stored metadata**, all of which the real feed carries on all 131 episodes and none of which
|
|
was being captured: feed and per-episode artwork (`itunes:image`), duration (`itunes:duration`,
|
|
parsed from either raw seconds or a `1:34:09` clock), season/episode numbers, and a playback
|
|
`position` so episodes resume.
|
|
|
|
**Schema migration.** `Db::open` now runs `migrate()`: `PRAGMA table_info` then `ALTER TABLE ADD
|
|
COLUMN` for anything missing, since `CREATE TABLE IF NOT EXISTS` does nothing to an existing table.
|
|
Tested on a *copy* of the live database first. Accidentally SIGPIPE'd it partway through (a `head -4`
|
|
on the output) which usefully proved it is idempotent and self-heals: the next run added the
|
|
remaining columns, 131 entries and 6 downloads intact.
|
|
|
|
**New endpoints:** entry filters (all/unread/downloaded/flagged), case-insensitive search over title
|
|
and notes, pagination totals, `POST .../position`, `read-all`, `download-latest`, and OPML
|
|
import/export over HTTP.
|
|
|
|
**UI:** persistent player bar surviving navigation (play/pause, ±15s/30s, 0.8-2.5x speed, volume,
|
|
scrubber, all remembered in localStorage), resume playback via `sendBeacon` every 10s, MediaSession
|
|
for lock-screen controls, keyboard shortcuts (space, arrows, `/`, Esc), artwork everywhere with
|
|
generated-initials fallback, filter tabs, episode search, load-more pagination, live progress bars
|
|
off SSE, real modals, toasts, light/dark toggle, and a mobile layout with a collapsing sidebar.
|
|
|
|
**Bug found and fixed in review: filtering was broken while searching worked.** `count_entries`
|
|
dropped the search clause when no term was given but still bound `?2`; rusqlite rejects a parameter
|
|
the statement never mentions -- *"Wrong number of parameters passed to query. Got 2, needed 1"*. So
|
|
`?q=...` worked and a plain Unread/Downloaded/Flagged filter 500'd. The clause is now always present
|
|
with `?2 = ''` short-circuiting it. Regression test runs all four filters with and without a search
|
|
term and asserts the page and the count agree.
|
|
|
|
**A non-bug worth recording:** artwork looked broken on the live DB (0/131) while a fresh DB filled
|
|
all 131. The parser was fine -- the live DB simply had not been rescanned yet, since backfill happens
|
|
through `record_entry`'s update path. After a forced scan: 131/131 image, duration and season.
|
|
|
|
Verified live: all four filters return sane totals (131/129/10/0), search works (`session zero` -> 2,
|
|
`trunk` -> 10), `/media/1` with a Range header returns 206, index serves 34 KB. `cargo test` 32/32.
|
|
|
|
---
|
|
|
|
## 2026-09-10 — Fixed: the UI's Download button downloaded the wrong episodes
|
|
|
|
Reported from the running instance: clicking Download on *Music from a Darkened Room | Session Zero*
|
|
showed progress, then left the episode at `pending`.
|
|
|
|
**Cause, and it was a design error in step 11, not a glitch.** `POST /api/enclosures/{id}/download`
|
|
requeued the row to `pending` and asked for an ordinary scan, justified at the time as "needed no
|
|
new machinery — the queue is the table". But a scan means *take the lowest-id pending rows, up to
|
|
`max_new_per_check`*. Against a 125-item backlog with a cap of 2, that is ids 7 and 8; the requested
|
|
row 11 was never a candidate. The progress on screen was two other episodes downloading past it.
|
|
|
|
The queue can express what is outstanding but not what was *asked for*. `Command::Download {
|
|
enclosure }` is now its own command: it fetches that one row immediately, ignoring both queue order
|
|
and the per-scan cap, still through the single worker so it cannot overlap a scan. Torrent
|
|
enclosures route to `torrent_one` the same way a scan would.
|
|
|
|
Verified live: enclosure 11 went `pending` -> `done`, 90.3 MB on disk, and nothing else was pulled
|
|
in its place. `cargo test` 30/30, with the new command's wire form pinned.
|
|
|
|
Worth remembering: **the daemon scans immediately on its first tick**, so every restart costs
|
|
`max_new_per_check` episodes on the normal path.
|
|
|
|
---
|
|
|
|
## 2026-09-10 — Phase 2: web front end (steps 9-13)
|
|
|
|
axum in the daemon process, plain HTML/JS in `web/index.html` (embedded with `include_str!`), no
|
|
WASM toolchain. One binary still.
|
|
|
|
**Config hot-reload.** `Ctx.cfg` is now `RwLock<Arc<Config>>`; `ctx.cfg()` hands out a snapshot, so
|
|
no guard is ever held across an await. The UI rewrites config.toml and calls `reload_cfg`, and a
|
|
running daemon picks the change up on its next scan. The CLI mutation commands (`add`/`rm`/`import`)
|
|
now clone a snapshot, edit, and save.
|
|
|
|
**Auth.** `[web] enabled/bind/token`; the token is minted from `/dev/urandom` on first run, written
|
|
back to config.toml, and the URL printed. `?token=` sets a year-long cookie — it has to be a cookie
|
|
because an `<audio>` element cannot send a header. Comparison is constant-time. An empty token makes
|
|
the server refuse to serve rather than serve open.
|
|
|
|
**Endpoints.** `/api/feeds` (GET/POST), `/api/feeds/{id}` (PATCH/DELETE),
|
|
`/api/feeds/{id}/entries`, `/api/entries/{feed}/{guid}/flags`, `/api/enclosures/{id}/download`,
|
|
`/api/enclosures/{id}` (DELETE), `/api/fetch`, `/api/events` (SSE off the existing broadcast bus),
|
|
`/media/{id}` (tower-http `ServeFile`, so Range works).
|
|
|
|
**`read`/`flagged` finally have a writer** — the UI sets them, and playing an episode marks it read.
|
|
Retention has ordered by these since step 5 with nothing to set them.
|
|
|
|
**Download-on-demand needed no new machinery**: requeue the row to `pending` and kick a scan, since
|
|
the queue is the table. That also un-skips an enclosure a filter rejected under older settings.
|
|
|
|
Verified against the live Glass Cannon feed, daemon on 127.0.0.1:8749:
|
|
|
|
- auth: no token 401, wrong token 401, right token 200 + cookie, API then works on the cookie alone;
|
|
`/media/1` unauthenticated is 401.
|
|
- browsing: 131 entries page with enclosures attached.
|
|
- **XSS**: injected `<script>alert(1)</script><img src=x onerror=alert(2)>` into a stored
|
|
description; it reaches the page as `<p>hi</p><img src="x">` — script tag and handler both gone.
|
|
- media: 200 with `accept-ranges: bytes`; `Range: bytes=1000000-1000999` returns 206 with the right
|
|
`content-range`, so seeking works.
|
|
- config: PATCH -> 204, written to config.toml, and the running daemon reports the new values with
|
|
no restart.
|
|
- SSE: 105 events for one download (101 progress), then `download_done`/`feed_done`/`scan_done`.
|
|
- flags: 204, unread count dropped 131 -> 130. Delete: file gone, row `reaped`/`path=NULL`.
|
|
|
|
`cargo test` 30/30.
|
|
|
|
**Caveats.** It is plain HTTP — on a LAN bind the token crosses the network in the clear, and a feed
|
|
URL may itself carry a credential (Patreon's does), which the `/api/feeds` response includes. A
|
|
reverse proxy with TLS is the answer if that matters. There is no per-user anything; the token is
|
|
all-or-nothing access.
|
|
|
|
---
|
|
|
|
## 2026-09-10 — Tested against a real feed (Patreon / Glass Cannon)
|
|
|
|
First run against a live subscriber feed: 131 items, 6.17 GB total, all `audio/mpeg`, no `<ttl>`.
|
|
Parsed clean, titled `Get in the Trunk | Anthology Series | Delta Green`, downloaded a valid MP3
|
|
(`file` confirms ID3v2.3, MPEG layer III). A second scan pulled the *next* episode, confirming
|
|
`max_new_per_check` defers rather than drops. Over the socket: 101 progress events for a 79 MB
|
|
episode -- whole-percent throttling behaving exactly as intended -- then `download_done`,
|
|
`feed_done`, `scan_done`.
|
|
|
|
**Fixed: stripped separators left doubled spaces.** The feed title separates words with `|`, a
|
|
forbidden filename character, so the folder came out `Get in the Trunk Anthology Series Delta
|
|
Green`. The Python had the same wart.
|
|
|
|
Forbidden characters are now split in two. Separators (`/ \\ | :`) become `-`; the rest
|
|
(`? * < > " '`) are simply dropped. A run of dashes and spaces then collapses to `" - "` when the
|
|
run contained whitespace and to a bare `-` when it did not, so:
|
|
|
|
| input | output |
|
|
|---|---|
|
|
| `Get in the Trunk \| Anthology Series \| Delta Green` | `Get in the Trunk - Anthology Series - Delta Green` |
|
|
| `Ep 12: The One` | `Ep 12 - The One` |
|
|
| `AC/DC` | `AC-DC` |
|
|
| `well-known.mp3` | `well-known.mp3` |
|
|
| `../../etc/passwd` | `etc-passwd` |
|
|
|
|
Leading dashes and dots are trimmed too -- a filename starting with `-` trips up CLI tools.
|
|
|
|
**Worth knowing, not a bug:**
|
|
|
|
- The channel-level `itunes:explicit` is `true`, so with the default `allow_explicit = false` all
|
|
131 episodes are skipped. Because filter verdicts are recorded once at discovery, flipping the
|
|
setting afterwards does not re-evaluate enclosures already marked `skipped` -- they would need a
|
|
`UPDATE enclosures SET state='pending'`. Worth a `ipx retry <feed>` command if this bites.
|
|
- Patreon sends neither `etag` nor `last-modified` on a GET (a `last-modified` does appear on HEAD,
|
|
which is what made it look briefly like a storage bug). So this feed can never 304: every poll
|
|
re-fetches ~160 KB and re-parses 131 entries, and `interval_mins` is the only thing limiting the
|
|
rate. Cheap, but it means conditional GET buys nothing here.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 8: OPML and polish
|
|
|
|
`ipx add <url>` fetches the feed to name it from its own title (`Accidental Tech Podcast` ->
|
|
`accidental-tech-podcast`); a feed that cannot be reached is still added, named from its URL, rather
|
|
than refused. `ipx rm` leaves downloads and history alone, so re-adding a feed does not re-pull its
|
|
back catalogue. `ipx import`/`export` walk nested OPML folder outlines and skip URLs already
|
|
subscribed. `tracing` logs to stderr, `IPX_LOG` sets the filter. `contrib/` has a systemd user unit
|
|
for the daemon, plus a timer and one-shot service for the no-daemon style (with the caveat that
|
|
without a daemon there is no socket for a UI).
|
|
|
|
Verified: `cargo test` 27/27. Round-trip — exported 3 feeds with real titles, removed one,
|
|
re-imported: exactly the missing one came back, no duplicates, config still mode 0600. Quickstart
|
|
from a genuinely empty home with no env overrides: `list` on a missing config, `add`, `fetch`
|
|
(1 downloaded, 1 explicit skipped, 1 torrent 404), `list`; every path resolved under `$HOME`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 7: torrent.rs
|
|
|
|
`src/torrent.rs`: a librqbit `Session` started lazily on first torrent (binding ports and starting a
|
|
DHT for a config that has never seen a torrent would be rude). `.torrent` URLs and magnets both go
|
|
through `AddTorrent::from_url`. Progress is polled once a second and reported through the same
|
|
`Progress` events HTTP downloads use.
|
|
|
|
Downloads go **straight into the feed folder** rather than staging and moving. The plan said move
|
|
then seed, which cannot work — seeding serves the files it downloaded, so moving them first breaks
|
|
it. In-place also removes a copy the original had to do.
|
|
|
|
Seeding stops at `seed_ratio` or `seed_time_mins`, whichever comes first, then the torrent is
|
|
released from the session (files kept).
|
|
|
|
**Bug the smoke test caught:** the stall budget only covered the download loop, but resolving a
|
|
magnet's metadata happens inside `add_torrent`, which against a dead swarm never returns — a
|
|
torrent nobody seeds wedged the scan indefinitely. `add_torrent` is now wrapped in the same budget.
|
|
|
|
Verified: `cargo test` 27/27 (ratio incl. the divide-by-zero case, stall_mins = 0 not meaning
|
|
"abort instantly"). Routing: with `enabled = false` a torrent enclosure is marked
|
|
`skipped/torrents disabled` and never attempted. Session startup works here. Stall abort measured
|
|
end to end: with `stall_mins = 1`, a dead magnet failed at 20:58:23 -> 20:59:24, exactly 61s, and
|
|
the row recorded `error / no metadata after 1 minutes, gave up`.
|
|
|
|
**Not verified: an actual successful swarm download.** This sandbox has no reachable peers, so
|
|
smoke 5's happy path — payload lands, seeding stops at the ratio — has not been run. The code paths
|
|
either side of it are tested; the swarm itself needs a real network. Worth running once against a
|
|
live torrent feed before trusting it.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 6: ipc.rs + daemon
|
|
|
|
`src/ipc.rs`: `Event` and `Command` as serde-tagged enums (`{"ev":...}` / `{"cmd":...}`), one JSON
|
|
object per line over a Unix socket. `Emitter` is the single output path — it broadcasts to socket
|
|
clients, prints the human rendering to a terminal, or both, so the scan code no longer knows how it
|
|
is being watched. That replaces `printMSG` and its `;;1;;1;;100.00;;42.31` sentinels.
|
|
|
|
`main.rs` restructured around a `Ctx` (config, db, client, emitter). `ipx daemon` binds the socket,
|
|
serves clients, and runs a 60s ticker that defers to each feed's TTL. Commands from every client
|
|
funnel through one mpsc queue into a single worker, which is what stops two scans overlapping.
|
|
Any CLI subcommand with a wire form probes the socket first and proxies to a running daemon;
|
|
`--local` forces the work to happen in-process. SIGINT/SIGTERM remove the socket on the way out.
|
|
|
|
**Bug the smoke test caught:** `fetch` runs a retention sweep first, and that sweep was emitting the
|
|
terminal `ReapDone`. A UI waiting for its fetch to finish would have stopped reading before the scan
|
|
started. Only a standalone `ipx reap` emits it now.
|
|
|
|
Verified: `cargo test` 24/24. Smoke 3 in full — daemon starts and binds; a raw socket client sending
|
|
`{"cmd":"fetch","force":true}` gets `feed_start` → 14 throttled `progress` events → `download_done`
|
|
→ `feed_done` → `scan_done`; `{"cmd":"status"}` answers `{"ev":"status","feeds":1,...}`. With the
|
|
daemon up, `ipx fetch` from the CLI logged `command from a client cmd=Fetch { .. }` in the daemon
|
|
and rendered the streamed events, so it proxied rather than downloading in parallel. Socket removed
|
|
on SIGTERM; with no daemon the same command runs locally.
|
|
|
|
Deferred: `{"cmd":"cancel","enclosure":N}` from the plan's protocol is **not implemented** —
|
|
downloads run sequentially in one worker, so there is nothing to cancel concurrently yet. It wants
|
|
a per-download cancellation token, which is worth doing when downloads go parallel. Say if you want
|
|
it sooner.
|
|
|
|
Note: the event stream is a broadcast, so a CLI client seeing a busy daemon also sees that other
|
|
work. Fine for a terminal; a UI wanting strict request/response would want per-request ids.
|
|
|
|
Also note: the daemon reads config once at startup — changing `config.toml` needs a restart.
|
|
|
|
Next: step 7 — `torrent.rs`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 5: retention.rs
|
|
|
|
`src/retention.rs`: reconcile pass (rows claiming a file that is gone become `reaped`, fixing the
|
|
step-4 wart), age sweep, quota sweep keeping the original's 50 MB headroom pad, and entry pruning.
|
|
`ipx reap [--dry-run]`; a sweep also runs before every `fetch`, as the Python did per download.
|
|
`pick()` and `aged()` are pure so the ordering rules are testable without touching a disk.
|
|
|
|
**Judgement call worth Ray's eye.** The Python meant to reap only `read = 1 AND flagged = 0` but
|
|
never managed it — a missing `plistlib` import and an `EntreiesData` typo made that filter throw on
|
|
every candidate, so with a `.ipxd` present nothing was ever deleted. Requiring `read = 1` here would
|
|
be equally dead, because nothing marks episodes read until a UI exists. So: **`flagged` is the
|
|
keep-forever marker, and `read` only decides what goes first** (`ORDER BY read DESC, downloaded_at
|
|
ASC`). Quota therefore actually reclaims space headless. Say the word if you would rather unread
|
|
episodes were never touched.
|
|
|
|
Second call: `max_age_days` deletes *files* older than the cutoff, not just fileless entries as the
|
|
plan's wording had it — "keep 30 days of episodes" is what the setting reads like on a NAS.
|
|
|
|
Verified: `cargo test` 21/21, including the two tests encoding the exact bug the Python had —
|
|
flagged files are never offered, and read sort ahead of unread. Smoke 6 with three 30 MB episodes
|
|
against a 0.1 GB quota (52.4 MB ceiling after the pad): dry run listed ep1+ep2 and deleted nothing
|
|
(3 files still on disk), the real run deleted exactly those two oldest, left ep3, flipped both rows
|
|
to `reaped` with `path = NULL`. A full re-parse with the conditional-GET headers cleared then
|
|
re-downloaded nothing.
|
|
|
|
Next: step 6 — `ipc.rs` + daemon.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 4: download.rs
|
|
|
|
`src/download.rs`: streaming download to `<download_dir>/.ipx-incomplete/` (same filesystem as the
|
|
destination, so filing it is a rename, not the original's copy-then-unlink), content sniffing,
|
|
then `place()`. Filename comes from the URL's last path segment, percent-decoded, unless
|
|
`Content-Disposition` names one (RFC 5987 `filename*=` preferred). The sanitizer keeps UTF-8 —
|
|
`latin1_to_ascii` existed because 2004 filesystems demanded ASCII — strips the same characters
|
|
`stringCleaning()` did plus control chars, and adds a real 255-byte cap the Python never had,
|
|
preserving the extension across truncation.
|
|
|
|
Sniffing replaces `detectFileType()`, which called a `typeFile` module that was already missing in
|
|
2008 and so always answered `'data'`. Two rules survive: an HTML body is a failed download (login
|
|
wall/error page), and a torrent body is a torrent whatever the MIME claimed.
|
|
|
|
Filters run once at discovery and are recorded in `enclosures.state`; the download queue is then
|
|
just "everything still `pending`", so an enclosure held back by `max_new_per_check` is picked up by
|
|
the next scan instead of being lost. Keywords are OR'd across keywords and AND'd within one — the
|
|
original's nested loop let a later keyword silently undo an earlier miss.
|
|
|
|
Verified: `cargo test` 16/16. Smoke against a local server, five enclosures, each filter path hit:
|
|
`ep1.mp3 -> done`, `ep2.mp3 -> skipped (explicit)`, `ep2.mp3?v=3 -> skipped (no keyword match)`,
|
|
`paywall.html -> error (HTML page, not media)`, `ep5.torrent -> torrent (deferred to step 7)`.
|
|
Smoke 2 and 4 pass, and because a plain rerun 304s before parsing, dedupe was proved separately by
|
|
clearing the stored etag/last-modified and re-parsing all five entries: 0 downloaded, hand-deleted
|
|
file not refetched, `.ipx-incomplete` left empty.
|
|
|
|
Known wart: `ipx list` counts `path IS NOT NULL`, so a hand-deleted file still reads as downloaded.
|
|
Reconciling rows against the filesystem belongs in step 5.
|
|
Next: step 5 — `retention.rs`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 3: feed.rs
|
|
|
|
`src/feed.rs`: conditional GET (`If-None-Match` + `If-Modified-Since`, optional basic auth) and a
|
|
RSS-first / Atom-fallback parser normalising both into `ParsedFeed`/`Entry`/`Enclosure`. Feed-level
|
|
`itunes:explicit` overrides the entry level, as the original did. `<ttl>` is captured. RSS
|
|
`content:encoded` wins over `description`. Atom enclosures come only from `rel="enclosure"` links.
|
|
|
|
GUID: the original hashed the title or description when no guid existed; here the chain is
|
|
guid → permalink → enclosure URL → title, all stable identifiers, so no hashing and no MD5
|
|
dependency. An entry with none of them has nothing to download and is dropped.
|
|
|
|
`db.rs` gained `http_state`, `record_feed`, `touch_feed`, `set_feed_error`, `record_entry`,
|
|
`record_enclosure`. A changed title/description flips `read` back to 0 — what the original's
|
|
textDiff was ultimately for, minus the `<ins>`/`<del>` markup, which belongs in the UI.
|
|
|
|
`main.rs` gained `ipx fetch [FEED] [--force]`. A failing feed records its error and the scan
|
|
continues.
|
|
|
|
Verified: `cargo test` 10/10. Gate met against a local `python3 -m http.server` serving the
|
|
fixtures — first run inserted 4 entries + 4 enclosures across an RSS and an Atom feed; second run
|
|
showed both skip paths, `atomcast: not modified` (304) and `testcast: not due for 45m` (the feed's
|
|
own ttl=45 beating `interval_mins = 0`).
|
|
|
|
Deferred: nothing downloads yet — enclosure rows land in state `pending`. That is step 4.
|
|
Next: step 4 — `download.rs`.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 2: config.rs + db.rs
|
|
|
|
`src/config.rs`: serde structs for `[general]`, `[torrent]` and `[feeds.<id>]` with defaults, `~`
|
|
expansion, `IPX_CONFIG` / `IPX_DATA_DIR` overrides, `save()` at mode 0600, `Feed::password()`
|
|
(`password_env` beats a literal `password`), `Torrent::ports()` parsing `"6881-6889"`. A missing
|
|
config file loads as an empty one so a fresh install works. Retention defaults are 0/0
|
|
(unlimited, keep forever) — nothing gets deleted until Ray asks for it.
|
|
|
|
`src/db.rs`: schema exactly as planned, WAL + `busy_timeout`, `Db::open()` idempotent, connection
|
|
behind a `Mutex`, `feed_summary()` for `list`, `now()` helper. `enclosures.url` is UNIQUE — the
|
|
dedupe key that replaces `history.dat`.
|
|
|
|
`src/main.rs`: clap skeleton with `ipx list`. Only the subcommands that work exist; the rest arrive
|
|
with their steps.
|
|
|
|
Verified: `cargo test` 5/5 green (config defaults, port-range fallback incl. reversed range,
|
|
password_env precedence, schema idempotency, enclosure-url uniqueness). Gate met — `ipx --config
|
|
<scratch> list` printed both feeds and created `state.db` once across two runs.
|
|
|
|
Deferred: nothing. Two dead-code warnings (`Config::save`, `Feed::password`) are expected; steps 3
|
|
and 8 consume them.
|
|
Next: step 3 — `feed.rs`.
|
|
|
|
Also added `chrono` 0.4 (std, clock) for RFC-2822 pubDate parsing in step 3.
|
|
|
|
---
|
|
|
|
## 2026-09-09 — Step 1: repo skeleton
|
|
|
|
Repo created at `/src/ipodderx-rs`, default branch `main`, `cargo init --name ipx` (edition 2024,
|
|
rustc 1.95.0). LICENSE (MIT, carrying the 2010 copyright), README with the lineage note, and this
|
|
file. Hello-world `main.rs` builds.
|
|
|
|
Dependency versions pinned today — the later steps are written against these APIs:
|
|
|
|
| crate | version | notes |
|
|
|---|---|---|
|
|
| tokio | 1.53.1 | rt-multi-thread, macros, fs, io-util, net, sync, time, signal |
|
|
| reqwest | 0.13.5 | `default-features = false`; features **`rustls`** (not `rustls-tls` — renamed in 0.13), http2, gzip, stream, json, charset, **`system-proxy`** (env-var proxy pickup is opt-in in 0.13) |
|
|
| rss | 2.1.1 | default features; the `with-syndication` feature name in my notes does not exist — Atom is handled by the separate crate |
|
|
| atom_syndication | 0.12.10 | |
|
|
| rusqlite | 0.40.2 | `bundled` |
|
|
| librqbit | 9.0.1 | `default-features = false`; features `rust-tls`, `http-api-client`. The default `default-tls` feature pulls reqwest/native-tls **and** an OpenSSL sha1 backend (`crypto-hash`), which fails to build without pkg-config/OpenSSL headers. API not yet exercised — step 7 |
|
|
| serde 1.0.229 / serde_json 1.0.151 / toml 1.1.5 | | |
|
|
| clap | 4.6.6 | derive |
|
|
| infer 0.22.0 / dirs 7.0.0 / anyhow 1.0.104 / opml 1.1.6 | | |
|
|
| tracing 0.1.44 / tracing-subscriber 0.3.23 | | env-filter |
|
|
|
|
Deferred: nothing.
|
|
|
|
Gotcha worth keeping: three feature names in the plan were wrong against current crate versions —
|
|
`reqwest/rustls-tls` is now `rustls`, env-var proxy support moved behind `system-proxy`, and
|
|
`rss/with-syndication` does not exist. `librqbit`'s default features drag in OpenSSL; `rust-tls`
|
|
is the fix. Whole tree is rustls-only now, no C TLS dependency.
|
|
Next: step 2 — `config.rs` + `db.rs`. (done)
|