rays/ipx
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Artwork panels: how full, the hit rate and evictions (#198)

Browse Source 
Kept bytes against the limit was a flat line once the cache filled. Now a
percentage of the limit beside the share served from disk, and the images each
sweep evicts. The counter shipped minutes ago as ipx_art_trimmed_total is
renamed ipx_art_evicted_total before anything came to read it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-11 20:17:56 +00:00
parent f032d50ed3
commit 8dc6dd4ce6
6 changed files with 25 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -85,7 +85,7 @@ from 2026-10-11 15:15 UTC. All gauges:
| `ipx_deadair_backlog`, `ipx_deadair_busy_seconds` | files of shows with DeadAir on not yet looked at (episodes everyone heard through stay counted); how long it has been on its file | | `ipx_deadair_backlog`, `ipx_deadair_busy_seconds` | files of shows with DeadAir on not yet looked at (episodes everyone heard through stay counted); how long it has been on its file |
| `ipx_feed_retries` | feeds waiting to retry a failed name lookup | | `ipx_feed_retries` | feeds waiting to retry a failed name lookup |
| `ipx_art_cache_bytes`, `ipx_art_cache_limit_bytes` | artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design | | `ipx_art_cache_bytes`, `ipx_art_cache_limit_bytes` | artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design |
| `ipx_art_trimmed_total` | counter: images the sweep threw out to stay under the limit; many at every sweep means the limit is too small | | `ipx_art_evicted_total`, `ipx_art_requests_total{result="hit"\|"miss"}` | counters: images the sweep threw out to stay under the limit, and artwork served from disk or fetched from the publisher; many evicted at every sweep with a falling hit rate means the limit is too small |
| `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the healthcheck last counted it (every 30 s) | | `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the healthcheck last counted it (every 30 s) |
## The checks ## The checks

2
CHANGELOG.md
View File

@@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added ### Added
- With `[web] metrics_bind` set, `/metrics` gives Prometheus, on a thread and port of its own, what the daemon holds between requests: database connections in use against the pool's size, commands waiting for the worker, live-update listeners, the runtime's tasks, memory and open files. So are the worker's job and how long it has run, jobs done and their time by kind, when a scan last finished, web requests, feed fetches and downloads in progress, download speed, live updates queued and missed, DeadAir's backlog and the file it is on, feeds waiting to retry, the artwork kept and the library's counts. So is how much artwork the sweep throws out to stay under its limit. The iPX dashboard graphs them in a Daemon row (#194, #195, #198). - With `[web] metrics_bind` set, `/metrics` gives Prometheus, on a thread and port of its own, what the daemon holds between requests: database connections in use against the pool's size, commands waiting for the worker, live-update listeners, the runtime's tasks, memory and open files. So are the worker's job and how long it has run, jobs done and their time by kind, when a scan last finished, web requests, feed fetches and downloads in progress, download speed, live updates queued and missed, DeadAir's backlog and the file it is on, feeds waiting to retry, the artwork kept and the library's counts. So are how full the artwork kept is, how much of it is served from disk, and how much the sweep throws out to stay under its limit. The iPX dashboard graphs them in a Daemon row (#194, #195, #198).
- The page reports its errors, the player's, how long play takes to start and how long it stalls to the server's log, and the requests one tap sets off share a trace, the playlist, audio and saved place of what it played included (#190, #192). - The page reports its errors, the player's, how long play takes to start and how long it stalls to the server's log, and the requests one tap sets off share a trace, the playlist, audio and saved place of what it played included (#190, #192).
- Saving an episode from a show with DeadAir on saves it without its dead air, at your breathing room (#186). - Saving an episode from a show with DeadAir on saves it without its dead air, at your breathing room (#186).
- Choose the pause DeadAir leaves where it cuts a silence, from 0.2 to 0.5 seconds, for each show in its settings, as its breathing room, on a slider from brisk to relaxed. It is 0.3 seconds unless you change it, up from 0.2, which made speech sound rushed (#182, #183). - Choose the pause DeadAir leaves where it cuts a silence, from 0.2 to 0.5 seconds, for each show in its settings, as its breathing room, on a slider from brisk to relaxed. It is 0.3 seconds unless you change it, up from 0.2, which made speech sound rushed (#182, #183).

15
grafana/dashboard.py
View File

@@ -185,12 +185,17 @@ ts("Backlogs", [prom('max(ipx_deadair_backlog{job="ipx"})', legend="DeadAir to l
prom('sum(increase(ipx_events_missed_total{job="ipx"}[$__rate_interval]))', ref="D", legend="live updates missed")], prom('sum(increase(ipx_events_missed_total{job="ipx"}[$__rate_interval]))', ref="D", legend="live updates missed")],
16, w=8) 16, w=8)
y += 8 y += 8
ts("DeadAir on a file", [prom('max(ipx_deadair_busy_seconds{job="ipx"})', legend="for")], 0, w=12, unit="s", ts("DeadAir on a file", [prom('max(ipx_deadair_busy_seconds{job="ipx"})', legend="for")], 0, w=8, unit="s",
desc="How long DeadAir has been on the file it is analysing; a few seconds each.") desc="How long DeadAir has been on the file it is analysing; a few seconds each.")
ts("Artwork thrown out", [prom('sum(increase(ipx_art_trimmed_total{job="ipx"}[$__rate_interval]))', legend="images")], ts("Artwork", [prom('max(ipx_art_cache_bytes{job="ipx"}) / max(ipx_art_cache_limit_bytes{job="ipx"})', legend="full"),
12, w=12, bars=True, prom('sum(increase(ipx_art_requests_total{job="ipx", result="hit"}[$__rate_interval])) / '
desc="What is kept sits at its limit (500 MiB unless changed), since each sweep trims the least recently used to fit. " 'sum(increase(ipx_art_requests_total{job="ipx"}[$__rate_interval]))', ref="B", legend="served from disk")],
"Many thrown out at every sweep means the limit is too small and artwork is fetched again.") 8, w=8, unit="percentunit",
desc="Full sits near 100% once the cache has filled, since each sweep trims it to fit. Served from disk is the hit rate: "
"the rest was fetched from the publisher.")
ts("Artwork evicted", [prom('sum(increase(ipx_art_evicted_total{job="ipx"}[$__rate_interval]))', legend="images")],
16, w=8, bars=True,
desc="Images each sweep threw out to stay under the limit. Many at every sweep, with a falling hit rate, means the limit is too small.")
y += 8 y += 8
# ---- Traces # ---- Traces

2
src/art.rs
View File

@@ -98,7 +98,7 @@ pub fn trim(dir: &Path, limit: u64) -> usize {
} }
} }
crate::metrics::STATS.art_cache_bytes.store(total, std::sync::atomic::Ordering::Relaxed); crate::metrics::STATS.art_cache_bytes.store(total, std::sync::atomic::Ordering::Relaxed);
crate::metrics::STATS.art_trimmed.fetch_add(gone as u64, std::sync::atomic::Ordering::Relaxed); crate::metrics::STATS.art_evicted.fetch_add(gone as u64, std::sync::atomic::Ordering::Relaxed);
gone gone
} }

13
src/metrics.rs
View File

@@ -32,7 +32,10 @@ pub struct Stats {
pub art_cache_bytes: AtomicU64, pub art_cache_bytes: AtomicU64,
/// Images the sweep threw out to keep under the limit: kept bytes sit at the limit by design, /// Images the sweep threw out to keep under the limit: kept bytes sit at the limit by design,
/// so this is what says whether the limit is too small. /// so this is what says whether the limit is too small.
pub art_trimmed: AtomicU64, pub art_evicted: AtomicU64,
/// Artwork served from disk, and fetched from the publisher because it was not there.
pub art_hits: AtomicU64,
pub art_misses: AtomicU64,
/// The library as the healthcheck last counted it, every 30 s; -1 until it has. /// The library as the healthcheck last counted it, every 30 s; -1 until it has.
pub feeds: AtomicI64, pub feeds: AtomicI64,
pub pending: AtomicI64, pub pending: AtomicI64,
@@ -55,7 +58,9 @@ impl Stats {
deadair_backlog: AtomicI64::new(0), deadair_backlog: AtomicI64::new(0),
deadair_busy_since: AtomicI64::new(0), deadair_busy_since: AtomicI64::new(0),
art_cache_bytes: AtomicU64::new(0), art_cache_bytes: AtomicU64::new(0),
art_trimmed: AtomicU64::new(0), art_evicted: AtomicU64::new(0),
art_hits: AtomicU64::new(0),
art_misses: AtomicU64::new(0),
feeds: AtomicI64::new(-1), feeds: AtomicI64::new(-1),
pending: AtomicI64::new(-1), pending: AtomicI64::new(-1),
downloaded: AtomicI64::new(-1), downloaded: AtomicI64::new(-1),
@@ -209,7 +214,9 @@ fn work(out: &mut String, st: &Stats, event_queue: usize, retries: usize, art_li
gauge(out, "ipx_feed_retries", "Feeds waiting to try a failed name lookup again.", &[("", retries as f64)]); gauge(out, "ipx_feed_retries", "Feeds waiting to try a failed name lookup again.", &[("", retries as f64)]);
gauge(out, "ipx_art_cache_bytes", "Artwork kept on disk, as of the last sweep.", &[("", st.art_cache_bytes.load(Relaxed) as f64)]); gauge(out, "ipx_art_cache_bytes", "Artwork kept on disk, as of the last sweep.", &[("", st.art_cache_bytes.load(Relaxed) as f64)]);
gauge(out, "ipx_art_cache_limit_bytes", "The most artwork kept.", &[("", art_limit as f64)]); gauge(out, "ipx_art_cache_limit_bytes", "The most artwork kept.", &[("", art_limit as f64)]);
counter(out, "ipx_art_trimmed_total", "Images thrown out to keep the artwork under its limit.", &[("", st.art_trimmed.load(Relaxed) as f64)]); counter(out, "ipx_art_evicted_total", "Images thrown out to keep the artwork under its limit.", &[("", st.art_evicted.load(Relaxed) as f64)]);
counter(out, "ipx_art_requests_total", "Artwork asked for: served from disk, or fetched from the publisher.",
&[("result=\"hit\"", st.art_hits.load(Relaxed) as f64), ("result=\"miss\"", st.art_misses.load(Relaxed) as f64)]);
for (name, help, v) in [ for (name, help, v) in [
("ipx_feeds", "Feeds in the catalogue, as the healthcheck last counted.", &st.feeds), ("ipx_feeds", "Feeds in the catalogue, as the healthcheck last counted.", &st.feeds),
("ipx_downloads_pending", "Files a scan will download on its own, as the healthcheck last counted.", &st.pending), ("ipx_downloads_pending", "Files a scan will download on its own, as the healthcheck last counted.", &st.pending),

2
src/web.rs
View File

@@ -2115,11 +2115,13 @@ async fn art(State(state): State<WebState>, Query(q): Query<ArtQuery>) -> Respon
let keep = state.ctx.cfg().general.art_cache_mb > 0; let keep = state.ctx.cfg().general.art_cache_mb > 0;
let file = crate::art::path(&q.u); let file = crate::art::path(&q.u);
if keep && let Some((kind, body)) = crate::art::read(&file) { if keep && let Some((kind, body)) = crate::art::read(&file) {
crate::metrics::STATS.art_hits.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
return art_response(kind, body); return art_response(kind, body);
} }
if let Some(why) = crate::art::failed(&q.u) { if let Some(why) = crate::art::failed(&q.u) {
return (StatusCode::BAD_GATEWAY, why).into_response(); return (StatusCode::BAD_GATEWAY, why).into_response();
} }
crate::metrics::STATS.art_misses.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let got = async { let got = async {
let mut r = state.ctx.client.get(&q.u).timeout(crate::feed::FEED_TIMEOUT).send().await?.error_for_status()?; let mut r = state.ctx.client.get(&q.u).timeout(crate::feed::FEED_TIMEOUT).send().await?.error_for_status()?;
let kind = r.headers().get(header::CONTENT_TYPE).and_then(|v| v.to_str().ok()).unwrap_or("").to_owned(); let kind = r.headers().get(header::CONTENT_TYPE).and_then(|v| v.to_str().ok()).unwrap_or("").to_owned();