rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

DeadAir on the server: copies with long silences shortened (#171)

Browse Source 
For anyone who turns DeadAir on, a background task makes each episode a
copy with every silence quieter than -38 dB for over 0.35 s shortened to
0.2 s. Nothing is re-encoded: ffmpeg copies the original's frames and
leaves the silent ones out, about 11 s for an hour of MP3 in a release
build. On Brad & Will's episode 359 that is 520 cuts and 3.3 minutes
saved, the copy decodes clean, and it has no isolated spikes in its
silences that the original lacks.

Cuts are made from the middle of each silence, keeping 0.1 s at each
end, so the frames either side of a cut are silent: decoding blends a
frame with the one before it, and an MP3 frame can begin in the bytes of
the frame before, so the first frame after a cut decodes against the
wrong neighbour. Inside a silence nobody hears that; at a word they would.

Positions stay in the original's time and the server converts them both
ways. The iOS and Android apps save positions themselves from their own
player, even while the page is asleep, so converting in the page would
have stored the copy's times as the original's. /media/{id}/deadair
serves the copy at its own address and records that this person is
playing it; a position saved meanwhile is converted from the copy's
clock, as is the length a player measures.

The copy is made by a task of its own, not the worker: a first pass over
a library is hours of ffmpeg, and scans must not wait behind it. It
checks every two minutes, which covers new downloads, files already
downloaded, a switch turned on and a file downloaded again with one rule.
Episodes everyone who would hear the copy has finished get none.
download::remove deletes a copy with its original, and the quota counts
copies.

Settings: the switch is PATCH /api/me {deadair}, a feed kept out is
PATCH /api/feeds/{id} {deadair: false}. GET /api/me gives the switch and
deadair_saved, the seconds saved on episodes heard 75% or more on a copy,
each counted once. Three new tables, so no ALTER.

The image gains Debian's ffmpeg, about 480 MB, in a layer that stays
cached until the base image changes. The player's switch, the feed
setting and the stat come next, after mockups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-09 17:36:49 +00:00
parent 8687e4ff1b
commit 63e192767d
10 changed files with 968 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

520
src/deadair.rs Normal file
View File

@@ -0,0 +1,520 @@
//! DeadAir (issue #171): podcasts play from a copy with their long silences shortened, so they
//! finish sooner without sounding sped up.
//!
//! The copy is the original's own frames with the silent ones left out, copied by ffmpeg with
//! nothing re-encoded: about 10 s an hour of audio on one core, finding the silences included.
//! It is made in the background, one file at a time, for files in feeds someone has DeadAir on
//! for, and kept beside the original (`copy_path`) until `download::remove` takes both.
//!
//! Positions stay in the original's time. What the copy left out is stored as `Cut`s of the
//! original's time, and the server converts both ways (`to_copy`, `to_original`): the iOS and
//! Android apps save positions themselves, from their own player, so converting in the page
//! would miss them.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use anyhow::{Context, Result, bail};
use serde::{Deserialize, Serialize};
use crate::Ctx;
/// Quieter than this for longer than `MIN_SILENCE` is a silence: what the measurements on the
/// issue used. A podcast's room tone sits well below it and speech well above.
const THRESHOLD: &str = "-38dB";
const MIN_SILENCE: f64 = 0.35;
/// Kept at each end of a silence, so the frames on both sides of a cut are silent. Decoding
/// blends a frame with the one before it, and an MP3 frame can begin in the bytes of the frame
/// before (the bit reservoir): the first frame after a cut decodes against the wrong neighbour,
/// which nobody hears between two silent frames and everyone hears at the start of a word.
const KEEP: f64 = 0.1;
/// ffmpeg takes the frames to drop as one argument, and Linux refuses an argument over 128 KiB.
// shortcut: a file with more silences than this keeps only its longest; about 9 hours of
// conversation at the rate measured. Pass the list some other way if a feed ever gets there.
const MAX_CUTS: usize = 4000;
/// How long the maker waits when there is nothing to do: a download or a switch turned on is
/// picked up this long after, at most.
const IDLE: std::time::Duration = std::time::Duration::from_secs(120);
/// A span of the original, in milliseconds, that the copy leaves out.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub struct Cut(pub i64, pub i64);
/// Where the copy of `original` goes: beside it, `name.deadair.mp3` for `name.mp3`.
pub fn copy_path(original: &Path) -> PathBuf {
let stem = original.file_stem().map(|s| s.to_string_lossy()).unwrap_or_default();
let name = match original.extension() {
Some(ext) => format!("{stem}.deadair.{}", ext.to_string_lossy()),
None => format!("{stem}.deadair"),
};
original.with_file_name(name)
}
/// The silences silencedetect reported, as (start, end) seconds. One still open at the end of
/// the file has no end and is left alone.
pub fn parse_silences(log: &str) -> Vec<(f64, f64)> {
let mut out = Vec::new();
let mut start = None;
for line in log.lines() {
if let Some(v) = after(line, "silence_start: ") {
start = Some(v);
} else if let Some(v) = after(line, "silence_end: ")
&& let Some(s) = start.take()
{
out.push((s, v));
}
}
out
}
fn after(line: &str, key: &str) -> Option<f64> {
let rest = &line[line.find(key)? + key.len()..];
rest.split_whitespace().next()?.parse().ok()
}
/// Which frames to drop for these silences, as inclusive ranges of frame numbers, and the same
/// as `Cut`s. Frame `n` plays from `first + n * frame` seconds. Only frames wholly inside a
/// silence, `KEEP` in from each end, go.
pub fn plan(silences: &[(f64, f64)], first: f64, frame: f64) -> (Vec<(u64, u64)>, Vec<Cut>) {
let mut spans: Vec<(u64, u64)> = silences
.iter()
.filter(|(s, e)| e - s >= MIN_SILENCE)
.filter_map(|&(s, e)| {
let a = ((s + KEEP - first) / frame).ceil().max(0.0) as u64;
let b = ((e - KEEP - first) / frame).floor() as i64 - 1;
(b >= a as i64).then_some((a, b as u64))
})
.collect();
if spans.len() > MAX_CUTS {
spans.sort_by_key(|&(a, b)| std::cmp::Reverse(b - a));
spans.truncate(MAX_CUTS);
spans.sort();
}
let ms = |n: u64| ((first + n as f64 * frame) * 1000.0).round() as i64;
let cuts = spans.iter().map(|&(a, b)| Cut(ms(a), ms(b + 1))).collect();
(spans, cuts)
}
/// A time in the original, in ms, as the same moment in the copy. A time inside a cut is where
/// the cut was made.
pub fn to_copy(cuts: &[Cut], ms: i64) -> i64 {
let mut out = ms;
for &Cut(s, e) in cuts {
if ms >= e {
out -= e - s;
} else {
if ms > s {
out -= ms - s;
}
break;
}
}
out
}
/// A time in the copy, in ms, as the same moment in the original.
pub fn to_original(cuts: &[Cut], ms: i64) -> i64 {
let mut out = ms;
for &Cut(s, e) in cuts {
if out >= s {
out += e - s;
} else {
break;
}
}
out
}
pub fn saved(cuts: &[Cut]) -> i64 {
cuts.iter().map(|Cut(s, e)| e - s).sum()
}
/// Whether someone with these settings plays a feed from its copies: their switch on, and the
/// feed not kept out, by itself or by the group it came in (an OPML or a Patreon creator).
pub fn wants(prefs: &HashMap<String, bool>, feed: &str, group: Option<&str>) -> bool {
prefs.get("").copied().unwrap_or(false)
&& prefs.get(feed).or_else(|| group.and_then(|g| prefs.get(g))).copied().unwrap_or(true)
}
/// Each person's settings who has the switch on, and each managed feed's group, which is what
/// `wants` needs.
pub struct Wanting {
pub people: HashMap<i64, HashMap<String, bool>>,
pub groups: HashMap<String, String>,
}
impl Wanting {
pub async fn load(ctx: &Ctx, only: Option<i64>) -> Result<Self> {
let mut people: HashMap<i64, HashMap<String, bool>> = HashMap::new();
for (user, feed, wanted) in ctx.db.deadair_prefs(only).await? {
people.entry(user).or_default().insert(feed, wanted);
}
people.retain(|_, p| p.get("").copied().unwrap_or(false));
let groups = if people.is_empty() {
HashMap::new()
} else {
ctx.db.managed_feeds().await?.into_iter().map(|m| (m.id, m.group_id)).collect()
};
Ok(Self { people, groups })
}
pub fn wants(&self, user: i64, feed: &str) -> bool {
self.people
.get(&user)
.is_some_and(|p| wants(p, feed, self.groups.get(feed).map(String::as_str)))
}
}
/// What each person is playing from a copy, by (person, feed, guid): set when `/media` serves
/// them a copy and cleared when it serves the original, so a saved position is converted from
/// the clock it was measured in, whether the page saved it or a native app did.
// shortcut: in memory, so a restart forgets it. A position saved before the player next asks
// for more of the copy is then taken as the original's, which resumes a little early, never late.
pub type Playing = Arc<std::sync::Mutex<HashMap<(i64, String, String), Arc<crate::db::Copy>>>>;
/// Puts a page of entries in the clock of what this person will play: for each file with a
/// copy they play, `deadair` is the seconds it saves, and the entry's position and length are
/// the copy's. The page plays `/media/{id}/deadair` for a file that has it.
pub async fn for_reader(ctx: &Ctx, user: i64, rows: &mut [crate::db::EntryRow]) -> Result<()> {
let wanting = Wanting::load(ctx, Some(user)).await?;
if wanting.people.is_empty() {
return Ok(());
}
let ids: Vec<i64> = rows
.iter()
.filter(|r| wanting.wants(user, &r.feed_id))
.flat_map(|r| r.enclosures.iter().filter(|e| e.path.is_some()).map(|e| e.id))
.collect();
let copies: HashMap<i64, crate::db::Copy> =
ctx.db.copies(&ids).await?.into_iter().map(|c| (c.enclosure_id, c)).collect();
for r in rows.iter_mut() {
let mut converted = false;
for e in &mut r.enclosures {
let Some(c) = copies.get(&e.id) else { continue };
e.deadair = Some(c.saved_ms / 1000);
// One position an item, however many files: the first with a copy sets the clock.
if !std::mem::replace(&mut converted, true) {
r.position = to_copy(&c.cuts, r.position * 1000) / 1000;
r.duration = r.duration.map(|d| to_copy(&c.cuts, d * 1000) / 1000);
}
}
}
Ok(())
}
/// A position and length a player saved, in the clock of what it was playing, as the
/// original's. Past 75% of a copy, the episode also counts towards the time DeadAir has saved
/// this person.
pub async fn on_save(
ctx: &Ctx,
playing: &Playing,
user: i64,
feed: &str,
guid: &str,
secs: i64,
duration: Option<i64>,
) -> Result<(i64, Option<i64>)> {
let copy = playing.lock().unwrap().get(&(user, feed.to_owned(), guid.to_owned())).cloned();
let Some(c) = copy else { return Ok((secs, duration)) };
let original = |s: i64| to_original(&c.cuts, s * 1000) / 1000;
let (secs, duration) = (original(secs), duration.map(original));
let len = match duration {
Some(d) => Some(d),
None => ctx.db.entry_duration(user, feed, guid).await?,
};
if let Some(len) = len
&& len > 0
&& secs * 4 >= len * 3
{
ctx.db.count_deadair(user, feed, guid, c.saved_ms).await?;
}
Ok((secs, duration))
}
/// Makes copies until stopped: the next one anyone wants, or a wait of `IDLE` when there is
/// none.
pub async fn run(ctx: Arc<Ctx>, mut stop: tokio::sync::watch::Receiver<bool>) {
if !have_ffmpeg().await {
tracing::warn!("ffmpeg is not installed, so DeadAir makes no copies");
return;
}
loop {
// Dropping the pass on a signal kills the ffmpeg it is running (`kill_on_drop`).
let made = tokio::select! {
_ = stop.changed() => return,
made = next(&ctx) => made,
};
match made {
Ok(true) => continue,
Ok(false) => {}
Err(e) => tracing::warn!(error = %format!("{e:#}"), "DeadAir could not make a copy"),
}
tokio::select! {
_ = stop.changed() => return,
_ = tokio::time::sleep(IDLE) => {}
}
}
}
async fn have_ffmpeg() -> bool {
tokio::process::Command::new("ffmpeg")
.arg("-version")
.stdout(std::process::Stdio::null())
.status()
.await
.is_ok_and(|s| s.success())
}
/// Makes one copy someone wants, if there is one to make. A file everyone who would play the
/// copy has already finished gets none: finished episodes are rarely played again.
// shortcut: those stay on the list and are looked at again each pass, one query each; record
// them as passed over if a large library makes the pass slow.
async fn next(ctx: &Ctx) -> Result<bool> {
// Nobody with the switch on, which is most passes: nothing more to ask the database.
let wanting = Wanting::load(ctx, None).await?;
if wanting.people.is_empty() {
return Ok(false);
}
for t in ctx.db.deadair_todo().await? {
let users: Vec<i64> =
wanting.people.keys().copied().filter(|&u| wanting.wants(u, &t.feed_id)).collect();
if users.is_empty() || !ctx.db.subscribed_any(&users, &t.feed_id).await? {
continue;
}
let readers = ctx.db.readers(&t.feed_id, &t.guid).await?;
if users.iter().all(|u| readers.contains(u)) {
continue;
}
let src = crate::download::playable(Path::new(&t.path));
let started = std::time::Instant::now();
let made = make(&src).await;
let (path, cuts) = match made {
Ok(Some((path, cuts))) => (Some(path.display().to_string()), cuts),
Ok(None) => (None, vec![]),
Err(e) => {
// Recorded as having none, so a file ffmpeg cannot read is not tried every pass.
tracing::warn!(path = %src.display(), error = %format!("{e:#}"), "DeadAir could not make a copy");
(None, vec![])
}
};
let saved_ms = saved(&cuts);
ctx.db.store_deadair(t.id, path.as_deref(), &serde_json::to_string(&cuts)?, saved_ms).await?;
tracing::info!(
ev = "deadair",
feed = t.feed_id,
path = %src.display(),
cuts = cuts.len(),
saved_s = saved_ms / 1000,
took_ms = started.elapsed().as_millis() as u64,
"DeadAir copy made"
);
return Ok(true);
}
Ok(false)
}
/// Makes the copy of one file. None for a file it does not apply to: video, anything but MP3
/// and AAC, or one with no silence long enough to cut.
pub async fn make(src: &Path) -> Result<Option<(PathBuf, Vec<Cut>)>> {
let Some((first, frame)) = probe(src).await? else { return Ok(None) };
let log = ffmpeg(&[
"-hide_banner", "-nostats", "-i", &src.display().to_string(), "-map", "0:a:0",
"-af", &format!("silencedetect=n={THRESHOLD}:d={MIN_SILENCE}"), "-f", "null", "-",
])
.await
.context("finding the silences")?;
let (frames, cuts) = plan(&parse_silences(&log), first, frame);
if cuts.is_empty() {
return Ok(None);
}
let drop: Vec<String> = frames.iter().map(|(a, b)| format!("between(n\\,{a}\\,{b})")).collect();
let dst = copy_path(src);
// Written under another name and moved into place, so a copy cut short by a restart is
// never served.
let tmp = dst.with_file_name(format!(
"{}.part.{}",
dst.file_stem().unwrap_or_default().to_string_lossy(),
dst.extension().unwrap_or_default().to_string_lossy()
));
ffmpeg(&[
"-v", "error", "-y", "-i", &src.display().to_string(), "-map", "0:a:0", "-c", "copy",
"-bsf:a", &format!("noise=drop='{}',setts=ts=N*DURATION", drop.join("+")),
&tmp.display().to_string(),
])
.await
.context("copying the frames kept")?;
tokio::fs::rename(&tmp, &dst).await.with_context(|| format!("moving into {}", dst.display()))?;
Ok(Some((dst, cuts)))
}
/// When the first audio frame starts and how long each lasts, in seconds, for a file DeadAir
/// can copy; None for one it cannot. A cover picture in an MP3 is not video.
async fn probe(src: &Path) -> Result<Option<(f64, f64)>> {
let path = src.display().to_string();
let out = run_out("ffprobe", &[
"-v", "error", "-show_entries", "stream=codec_type,codec_name:stream_disposition=attached_pic",
"-of", "json", &path,
])
.await?;
let v: serde_json::Value = serde_json::from_str(&out).context("reading ffprobe")?;
let streams = v["streams"].as_array().cloned().unwrap_or_default();
let video = streams
.iter()
.any(|s| s["codec_type"] == "video" && s["disposition"]["attached_pic"] != 1);
let audio = streams.iter().find(|s| s["codec_type"] == "audio");
if video || !audio.is_some_and(|a| a["codec_name"] == "mp3" || a["codec_name"] == "aac") {
return Ok(None);
}
let out = run_out("ffprobe", &[
"-v", "error", "-select_streams", "a:0", "-show_entries", "packet=pts_time,duration_time",
"-read_intervals", "%+#1", "-of", "csv=p=0", &path,
])
.await?;
let mut f = out.trim().split(',').map(|x| x.trim().parse::<f64>());
match (f.next(), f.next()) {
(Some(Ok(first)), Some(Ok(frame))) if frame > 0.0 => Ok(Some((first, frame))),
_ => bail!("no audio frame in {path}"),
}
}
/// Runs ffmpeg and gives back what it wrote to stderr, where silencedetect reports.
async fn ffmpeg(args: &[&str]) -> Result<String> {
let out = command("ffmpeg").args(["-nostdin", "-threads", "1"]).args(args).output().await?;
let log = String::from_utf8_lossy(&out.stderr).into_owned();
if !out.status.success() {
bail!("ffmpeg failed: {}", log.lines().rev().take(3).collect::<Vec<_>>().join(" / "));
}
Ok(log)
}
async fn run_out(prog: &str, args: &[&str]) -> Result<String> {
let out = command(prog).args(args).output().await.with_context(|| format!("running {prog}"))?;
if !out.status.success() {
bail!("{prog} failed: {}", String::from_utf8_lossy(&out.stderr).trim());
}
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
fn command(prog: &str) -> tokio::process::Command {
let mut c = tokio::process::Command::new(prog);
c.kill_on_drop(true).stdin(std::process::Stdio::null());
c
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn silences_are_read_from_the_log_and_an_open_one_is_left() {
let log = "[silencedetect @ 0x1] silence_start: 1\n\
[silencedetect @ 0x1] silence_end: 3.00002 | silence_duration: 2.00002\n\
[silencedetect @ 0x1] silence_start: 10.5\n";
assert_eq!(parse_silences(log), vec![(1.0, 3.00002)]);
}
#[test]
fn cuts_leave_silent_frames_either_side_and_skip_short_silences() {
// MP3 frames, 26.122 ms; one silence of 2 s, one too short to touch.
let frame = 1152.0 / 44100.0;
let (frames, cuts) = plan(&[(1.0, 3.0), (5.0, 5.3)], 0.0, frame);
assert_eq!(frames, vec![(43, 110)]);
let Cut(s, e) = cuts[0];
assert!(s >= 1100 && e <= 2900, "{cuts:?}");
assert!(s - 1100 < 27 && 2900 - e < 27, "no more than a frame kept beyond KEEP: {cuts:?}");
// AAC in MP4 starts a frame early, for the encoder's priming.
let (frames, _) = plan(&[(1.0, 3.0)], -0.02322, 1024.0 / 44100.0);
assert_eq!(frames, vec![(49, 124)]);
// A silence the margins use up entirely is left alone.
assert!(plan(&[(1.0, 1.21)], 0.0, frame).0.is_empty());
}
#[test]
fn positions_convert_both_ways() {
let cuts = [Cut(10_000, 20_000), Cut(30_000, 35_000)];
assert_eq!(to_copy(&cuts, 5_000), 5_000);
assert_eq!(to_copy(&cuts, 15_000), 10_000, "inside a cut is where it was made");
assert_eq!(to_copy(&cuts, 25_000), 15_000);
assert_eq!(to_copy(&cuts, 40_000), 25_000);
for t in [0, 5_000, 10_000, 15_000, 19_999, 20_000, 25_000, 40_000] {
let back = to_original(&cuts, to_copy(&cuts, t));
// A time inside a cut comes back as the cut's end: the copy has nothing between.
let expect = if (10_000..20_000).contains(&t) { 20_000 } else { t };
assert_eq!(back, expect, "{t}");
}
assert_eq!(saved(&cuts), 15_000);
}
#[test]
fn a_feed_kept_out_by_itself_or_its_group() {
let p = |v: &[(&str, bool)]| v.iter().map(|(k, w)| (k.to_string(), *w)).collect::<HashMap<_, _>>();
assert!(!wants(&p(&[]), "a", None), "off until turned on");
assert!(wants(&p(&[("", true)]), "a", None));
assert!(!wants(&p(&[("", true), ("a", false)]), "a", None));
assert!(!wants(&p(&[("", true), ("g", false)]), "a", Some("g")));
assert!(wants(&p(&[("", true), ("g", false), ("a", true)]), "a", Some("g")), "let back in");
assert!(!wants(&p(&[("", false), ("a", true)]), "a", None), "the switch rules");
}
#[test]
fn the_copy_sits_beside_the_original() {
assert_eq!(copy_path(Path::new("/d/show/ep 1.mp3")), Path::new("/d/show/ep 1.deadair.mp3"));
}
/// The whole of it, on a clip ffmpeg makes: a second of tone, two of silence, and so on.
/// Skipped where ffmpeg is not installed.
#[tokio::test]
async fn a_copy_is_shorter_by_what_was_cut_and_plays() {
if !have_ffmpeg().await {
eprintln!("no ffmpeg; skipped");
return;
}
let dir = std::env::temp_dir().join(format!("ipx-deadair-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
for (ext, codec) in [("mp3", "libmp3lame"), ("m4a", "aac")] {
let src = dir.join(format!("t.{ext}"));
ffmpeg(&[
"-v", "error", "-y", "-f", "lavfi", "-i",
"aevalsrc='if(lt(mod(t,3),1),0.5*sin(2*PI*440*t),0)':d=12:s=44100",
"-c:a", codec, &src.display().to_string(),
])
.await
.unwrap();
let (dst, cuts) = make(&src).await.unwrap().expect("a copy");
assert_eq!(cuts.len(), 4, "{ext}: {cuts:?}");
// About 1.8 s from each of the four 2 s silences.
let cut = saved(&cuts);
assert!((7_000..7_300).contains(&cut), "{ext}: {cut}");
let len = |p: &Path| {
let p = p.display().to_string();
async move {
run_out("ffprobe", &["-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", &p])
.await
.unwrap()
.trim()
.parse::<f64>()
.unwrap()
}
};
let short = len(&src).await - len(&dst).await;
assert!((short * 1000.0 - cut as f64).abs() < 100.0, "{ext}: {short} s shorter, {cut} ms cut");
// Decodes without a complaint.
assert!(ffmpeg(&["-v", "error", "-i", &dst.display().to_string(), "-f", "null", "-"]).await.unwrap().is_empty());
}
std::fs::remove_dir_all(&dir).unwrap();
}
}
/// A real episode, for trying the cut on something other than a test tone:
/// `IPX_DEADAIR_FILE=/path/ep.mp3 cargo test real_file -- --ignored --nocapture`. The copy is left
/// beside it to listen to.
#[cfg(test)]
#[tokio::test]
#[ignore]
async fn real_file() {
let src = PathBuf::from(std::env::var("IPX_DEADAIR_FILE").expect("IPX_DEADAIR_FILE"));
let t = std::time::Instant::now();
let (dst, cuts) = make(&src).await.unwrap().expect("a copy");
eprintln!("{} cuts, {:.1} min saved, {:.1?} -> {}", cuts.len(), saved(&cuts) as f64 / 60_000.0, t.elapsed(), dst.display());
}