Collapse whitespace left by stripped filename separators
A real feed titled with pipe separators produced a folder named "Get in the Trunk Anthology Series Delta Green": removing a forbidden character left the gap around it. Runs of whitespace now collapse, and control characters map to a space rather than vanishing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RPyeapneuXrCdojsaiXGbe
This commit is contained in:
27
PROGRESS.md
27
PROGRESS.md
@@ -30,6 +30,33 @@ The full design and step list live in the plan file at
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-09-10 — Tested against a real feed (Patreon / Glass Cannon)
|
||||||
|
|
||||||
|
First run against a live subscriber feed: 131 items, 6.17 GB total, all `audio/mpeg`, no `<ttl>`.
|
||||||
|
Parsed clean, titled `Get in the Trunk | Anthology Series | Delta Green`, downloaded a valid MP3
|
||||||
|
(`file` confirms ID3v2.3, MPEG layer III). A second scan pulled the *next* episode, confirming
|
||||||
|
`max_new_per_check` defers rather than drops. Over the socket: 101 progress events for a 79 MB
|
||||||
|
episode -- whole-percent throttling behaving exactly as intended -- then `download_done`,
|
||||||
|
`feed_done`, `scan_done`.
|
||||||
|
|
||||||
|
**Fixed: stripped separators left doubled spaces.** The feed title separates words with `|`, a
|
||||||
|
forbidden filename character, so the folder came out `Get in the Trunk Anthology Series Delta
|
||||||
|
Green`. `sanitize()` now collapses runs of whitespace (and maps control characters to a space
|
||||||
|
rather than deleting them). The Python had the same wart.
|
||||||
|
|
||||||
|
**Worth knowing, not a bug:**
|
||||||
|
|
||||||
|
- The channel-level `itunes:explicit` is `true`, so with the default `allow_explicit = false` all
|
||||||
|
131 episodes are skipped. Because filter verdicts are recorded once at discovery, flipping the
|
||||||
|
setting afterwards does not re-evaluate enclosures already marked `skipped` -- they would need a
|
||||||
|
`UPDATE enclosures SET state='pending'`. Worth a `ipx retry <feed>` command if this bites.
|
||||||
|
- Patreon sends neither `etag` nor `last-modified` on a GET (a `last-modified` does appear on HEAD,
|
||||||
|
which is what made it look briefly like a storage bug). So this feed can never 304: every poll
|
||||||
|
re-fetches ~160 KB and re-parses 131 entries, and `interval_mins` is the only thing limiting the
|
||||||
|
rate. Cheap, but it means conditional GET buys nothing here.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 2026-09-09 — Step 8: OPML and polish
|
## 2026-09-09 — Step 8: OPML and polish
|
||||||
|
|
||||||
`ipx add <url>` fetches the feed to name it from its own title (`Accidental Tech Podcast` ->
|
`ipx add <url>` fetches the feed to name it from its own title (`Accidental Tech Podcast` ->
|
||||||
|
|||||||
@@ -15,10 +15,24 @@ const MAX_NAME_BYTES: usize = 255;
|
|||||||
/// Keeps UTF-8: the original transliterated to ASCII via latin1_to_ascii because 2004
|
/// Keeps UTF-8: the original transliterated to ASCII via latin1_to_ascii because 2004
|
||||||
/// filesystems demanded it. Ours do not.
|
/// filesystems demanded it. Ours do not.
|
||||||
pub fn sanitize(name: &str) -> String {
|
pub fn sanitize(name: &str) -> String {
|
||||||
let mut out: String = name
|
let stripped: String = name
|
||||||
.chars()
|
.chars()
|
||||||
.filter(|c| !c.is_control() && !FORBIDDEN.contains(c))
|
.map(|c| if c.is_control() { ' ' } else { c })
|
||||||
|
.filter(|c| !FORBIDDEN.contains(c))
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
|
// Stripping a separator leaves a gap: "Show | Series | Delta" would otherwise become
|
||||||
|
// "Show Series Delta". Collapse runs of whitespace rather than leaving the seams.
|
||||||
|
let mut out = String::with_capacity(stripped.len());
|
||||||
|
for c in stripped.chars() {
|
||||||
|
if c.is_whitespace() {
|
||||||
|
if !out.ends_with(' ') {
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
out.push(c);
|
||||||
|
}
|
||||||
|
}
|
||||||
out = out.trim().trim_matches('.').trim().to_owned();
|
out = out.trim().trim_matches('.').trim().to_owned();
|
||||||
|
|
||||||
if out.len() > MAX_NAME_BYTES {
|
if out.len() > MAX_NAME_BYTES {
|
||||||
@@ -270,6 +284,17 @@ mod tests {
|
|||||||
assert_eq!(sanitize(" spaced.mp3 "), "spaced.mp3");
|
assert_eq!(sanitize(" spaced.mp3 "), "spaced.mp3");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sanitize_collapses_the_gaps_left_by_stripped_separators() {
|
||||||
|
// A real Patreon feed title; the pipes are forbidden characters.
|
||||||
|
assert_eq!(
|
||||||
|
sanitize("Get in the Trunk | Anthology Series | Delta Green"),
|
||||||
|
"Get in the Trunk Anthology Series Delta Green"
|
||||||
|
);
|
||||||
|
assert_eq!(sanitize("Ep 12: The One"), "Ep 12 The One");
|
||||||
|
assert_eq!(sanitize("a b"), "a b");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn sanitize_never_yields_an_empty_or_dot_name() {
|
fn sanitize_never_yields_an_empty_or_dot_name() {
|
||||||
assert_eq!(sanitize(""), "download");
|
assert_eq!(sanitize(""), "download");
|
||||||
|
|||||||
Reference in New Issue
Block a user