DeadAir copes with hundreds of silences, and retries a failure (#174)

Production's ffmpeg, Debian's 5.1, refused the copy of Brad & Will's
359: "Error initializing bitstream filter: bsf_list". The frames to drop
went to it as between(...)+between(...)+..., which nests a level a term,
and 5.1 takes 100 terms and refuses 200; the 6.1 the tests ran on took
all 520. Summed as a balanced tree the same 520 nest about 10 deep, and
5.1 takes 4000 that way, checked in the production container.

The failure was also stored as the file having no copy, so nothing would
have tried it again after this fix. Failures are now remembered until
the next start instead. The cap on cuts drops to 3500, so the argument
stays under Linux's 128 KiB with frame numbers in the millions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
 - DeadAir shortens an episode you have opened but not played yet, instead of passing it over as if you had finished it (#173).
+- DeadAir shortens episodes with more than about a hundred long silences, which is most of an hour's conversation, instead of failing on them, and tries a file it failed on again after the next restart (#174).
 - Cover art up to 20 MB is shown, instead of nothing for anything over 5 MB, as for five of the Glass Cannon Network's feeds.
 - Seeking in an episode while many people listen answers in about 10 ms instead of 40, and files stream about four times as fast: each connection now sends what is ready without waiting for the browser to acknowledge the last part.
 - Artwork that could not be fetched is not fetched again for an hour, instead of on every page load, and the server's log says which image it was and why.

src/deadair.rs

@@ -29,10 +29,11 @@ const MIN_SILENCE: f64 = 0.35;
 /// before (the bit reservoir): the first frame after a cut decodes against the wrong neighbour,
 /// which nobody hears between two silent frames and everyone hears at the start of a word.
 const KEEP: f64 = 0.1;
-/// ffmpeg takes the frames to drop as one argument, and Linux refuses an argument over 128 KiB.
-// shortcut: a file with more silences than this keeps only its longest; about 9 hours of
+/// ffmpeg takes the frames to drop as one argument, and Linux refuses an argument over 128 KiB:
+/// 3500 of them, numbered into the millions, come to about 110 KiB.
+// shortcut: a file with more silences than this keeps only its longest; about 7 hours of
 // conversation at the rate measured. Pass the list some other way if a feed ever gets there.
-const MAX_CUTS: usize = 4000;
+const MAX_CUTS: usize = 3500;
 /// How long the maker waits when there is nothing to do: a download or a feed turned on is
 /// picked up this long after, at most.
 const IDLE: std::time::Duration = std::time::Duration::from_secs(120);
@@ -239,11 +240,14 @@ pub async fn run(ctx: Arc<Ctx>, mut stop: tokio::sync::watch::Receiver<bool>) {
         tracing::warn!("ffmpeg is not installed, so DeadAir makes no copies");
         return;
     }
+    // Files ffmpeg failed on, passed over until the next start. Not stored: a failure recorded
+    // as the file having no copy was never tried again, even once a fix was deployed (#174).
+    let mut failed = std::collections::HashSet::new();
     loop {
         // Dropping the pass on a signal kills the ffmpeg it is running (`kill_on_drop`).
         let made = tokio::select! {
             _ = stop.changed() => return,
-            made = next(&ctx) => made,
+            made = next(&ctx, &mut failed) => made,
         };
         match made {
             Ok(true) => continue,
@@ -271,13 +275,16 @@ async fn have_ffmpeg() -> bool {
 /// copy has already heard through gets none: finished episodes are rarely played again.
 // shortcut: those stay on the list and are looked at again each pass, one query each; record
 // them as passed over if a large library makes the pass slow.
-async fn next(ctx: &Ctx) -> Result<bool> {
+async fn next(ctx: &Ctx, failed: &mut std::collections::HashSet<i64>) -> Result<bool> {
     // Nobody with DeadAir on for anything, which is most passes: nothing more to ask the database.
     let wanting = Wanting::load(ctx, None).await?;
     if wanting.people.is_empty() {
         return Ok(false);
     }
     for t in ctx.db.deadair_todo().await? {
+        if failed.contains(&t.id) {
+            continue;
+        }
         let users: Vec<i64> =
             wanting.people.keys().copied().filter(|&u| wanting.wants(u, &t.feed_id)).collect();
         if users.is_empty() || !ctx.db.subscribed_any(&users, &t.feed_id).await? {
@@ -294,9 +301,9 @@ async fn next(ctx: &Ctx) -> Result<bool> {
             Ok(Some((path, cuts))) => (Some(path.display().to_string()), cuts),
             Ok(None) => (None, vec![]),
             Err(e) => {
-                // Recorded as having none, so a file ffmpeg cannot read is not tried every pass.
                 tracing::warn!(path = %src.display(), error = %format!("{e:#}"), "DeadAir could not make a copy");
-                (None, vec![])
+                failed.insert(t.id);
+                return Ok(true);
             }
         };
         let saved_ms = saved(&cuts);
@@ -340,7 +347,7 @@ pub async fn make(src: &Path) -> Result<Option<(PathBuf, Vec<Cut>)>> {
     ));
     ffmpeg(&[
         "-v", "error", "-y", "-i", &src.display().to_string(), "-map", "0:a:0", "-c", "copy",
-        "-bsf:a", &format!("noise=drop='{}',setts=ts=N*DURATION", drop.join("+")),
+        "-bsf:a", &format!("noise=drop='{}',setts=ts=N*DURATION", any(&drop)),
         &tmp.display().to_string(),
     ])
     .await
@@ -349,6 +356,19 @@ pub async fn make(src: &Path) -> Result<Option<(PathBuf, Vec<Cut>)>> {
     Ok(Some((dst, cuts)))
 }
 
+/// Terms summed as a balanced tree, `((a+b)+(c+d))`. A plain `a+b+c+...` nests a level a term,
+/// and Debian's ffmpeg 5.1 refuses an expression nested much past 100 (#174).
+fn any(terms: &[String]) -> String {
+    match terms {
+        [] => "0".into(),
+        [one] => one.clone(),
+        _ => {
+            let (a, b) = terms.split_at(terms.len() / 2);
+            format!("({}+{})", any(a), any(b))
+        }
+    }
+}
+
 /// When the first audio frame starts and how long each lasts, in seconds, for a file DeadAir
 /// can copy; None for one it cannot. A cover picture in an MP3 is not video.
 async fn probe(src: &Path) -> Result<Option<(f64, f64)>> {
@@ -457,6 +477,21 @@ mod tests {
         assert!(!wants(&p(&[("g", true), ("a", false)]), "a", Some("g")), "kept out of its group's");
     }
 
+    #[test]
+    fn the_frames_to_drop_nest_shallowly_and_fit_one_argument() {
+        let terms: Vec<String> = (0..MAX_CUTS as u64)
+            .map(|i| format!("between(n\\,{}\\,{})", 1_000_000 + i * 300, 1_000_000 + i * 300 + 250))
+            .collect();
+        let e = any(&terms);
+        let deepest = e.chars().scan(0i32, |d, c| {
+            *d += match c { '(' => 1, ')' => -1, _ => 0 };
+            Some(*d)
+        }).max().unwrap();
+        assert!(deepest <= 14, "{deepest} deep");
+        assert!(e.len() < 128 * 1024, "{} bytes", e.len());
+        assert_eq!(any(&terms[..3]), format!("({}+({}+{}))", terms[0], terms[1], terms[2]));
+    }
+
     #[test]
     fn the_copy_sits_beside_the_original() {
         assert_eq!(copy_path(Path::new("/d/show/ep 1.mp3")), Path::new("/d/show/ep 1.deadair.mp3"));