rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Twenty database connections on Postgres, and a panel that does not cry wolf (#199, #200)

Browse Source 
The Database connections panel drew idle connections as their own line, which
sits on the max for ten minutes after any burst and read as the pool maxing out,
while at most two were ever in use. It now draws in use against the max, with
open dashed behind them. The pool itself goes from sqlx's ten to twenty: the
server allows 100 and had about 13 open across every app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-11 20:27:21 +00:00
parent 8dc6dd4ce6
commit c28accb8e4
4 changed files with 16 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -172,7 +172,7 @@ a count says something happened, the lines and traces say why.
$Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h
```
`up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is
unset. Connections in use at the pool's max (10 on Postgres) means something is waiting for
unset. Connections in use at the pool's max (20 on Postgres) means something is waiting for
one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only
grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue
above 0 for more than a scrape or two means every thread is busy: something is blocking a

3
CHANGELOG.md
View File

@@ -9,13 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- With `[web] metrics_bind` set, `/metrics` gives Prometheus, on a thread and port of its own, what the daemon holds between requests: database connections in use against the pool's size, commands waiting for the worker, live-update listeners, the runtime's tasks, memory and open files. So are the worker's job and how long it has run, jobs done and their time by kind, when a scan last finished, web requests, feed fetches and downloads in progress, download speed, live updates queued and missed, DeadAir's backlog and the file it is on, feeds waiting to retry, the artwork kept and the library's counts. So are how full the artwork kept is, how much of it is served from disk, and how much the sweep throws out to stay under its limit. The iPX dashboard graphs them in a Daemon row (#194, #195, #198).
- With `[web] metrics_bind` set, `/metrics` gives Prometheus, on a thread and port of its own, what the daemon holds between requests: database connections in use against the pool's size, commands waiting for the worker, live-update listeners, the runtime's tasks, memory and open files. So are the worker's job and how long it has run, jobs done and their time by kind, when a scan last finished, web requests, feed fetches and downloads in progress, download speed, live updates queued and missed, DeadAir's backlog and the file it is on, feeds waiting to retry, the artwork kept and the library's counts. So are how full the artwork kept is, how much of it is served from disk, and how much the sweep throws out to stay under its limit. The iPX dashboard graphs them in a Daemon row (#194, #195, #198, #199).
- The page reports its errors, the player's, how long play takes to start and how long it stalls to the server's log, and the requests one tap sets off share a trace, the playlist, audio and saved place of what it played included (#190, #192).
- Saving an episode from a show with DeadAir on saves it without its dead air, at your breathing room (#186).
- Choose the pause DeadAir leaves where it cuts a silence, from 0.2 to 0.5 seconds, for each show in its settings, as its breathing room, on a slider from brisk to relaxed. It is 0.3 seconds unless you change it, up from 0.2, which made speech sound rushed (#182, #183).
### Changed
- On Postgres, ipx keeps up to 20 connections to the database instead of 10, for the bursts of a page load asking for many covers at once (#200).
- Traces of the disk sweep, episode lists and DeadAir's work on each file show each step, and the sweep's trace says what it removed (#187).
- DeadAir is in the same box as a show's download settings, below them, and stays there when automatic downloads are off (#185).
- The container is built on Debian 13 (trixie) instead of 12, so DeadAir uses ffmpeg 7.1 instead of 5.1 (#177).

14
grafana/dashboard.py
View File

@@ -147,10 +147,16 @@ y += 8
# ---- Daemon, from Prometheus: what the daemon holds between requests (#194)
row("Daemon")
ts("Database connections", [prom('ipx_db_connections{job="ipx", state="in_use"}', legend="in use"),
prom('ipx_db_connections{job="ipx", state="idle"}', ref="B", legend="idle"),
prom('ipx_db_connections_max{job="ipx"}', ref="C", legend="most the pool opens")],
0, w=8, desc="In use reaching the most the pool opens means requests and scans are waiting for a connection.")
# Open connections stay open, idle, for ten minutes after a burst, so open at the max is not
# pressure; drawn as a line of its own it read as the pool maxing out (#199).
ts("Database connections", [prom('max(max_over_time(ipx_db_connections{job="ipx", state="in_use"}[$__interval]))', legend="in use"),
prom('sum(ipx_db_connections{job="ipx"})', ref="B", legend="open"),
prom('max(ipx_db_connections_max{job="ipx"})', ref="C", legend="most the pool opens")],
0, w=8, desc="In use reaching the most the pool opens means requests and scans are waiting for a connection. "
"Open connections stay open, idle, for ten minutes after a burst.")["fieldConfig"]["overrides"] = [
{"matcher": {"id": "byName", "options": "open"},
"properties": [{"id": "custom.lineStyle", "value": {"fill": "dash", "dash": [4, 4]}},
{"id": "custom.fillOpacity", "value": 0}, {"id": "color", "value": {"mode": "fixed", "fixedColor": "#8e8e8e"}}]}]
ts("Waiting", [prom('ipx_worker_queue{job="ipx"}', legend="commands for the worker"),
prom('ipx_runtime_queue{job="ipx"}', ref="B", legend="tasks for a thread"),
prom('ipx_downloads_pending{job="ipx"}', ref="C", legend="downloads")],

7
src/db.rs
View File

@@ -169,10 +169,9 @@ async fn connect(location: &str) -> Result<sea_orm::DatabaseConnection> {
// line for it at 50 requests a second (#136). In WAL readers run beside the one writer. Each
// transaction here writes first, so it takes the write lock or waits out the busy timeout for
// it; SQLite refuses at once only a transaction that read a snapshot a write has since moved
// past. Postgres keeps sqlx's ten.
if !is_postgres(location) {
opts.max_connections(8);
}
// past. Postgres gets twenty, where sqlx's ten filled in bursts: a page load asks for dozens
// of covers at once, each with a lookup (#200). The server allows 100, shared with other apps.
opts.max_connections(if is_postgres(location) { 20 } else { 8 });
sea_orm::Database::connect(opts)
.await
.with_context(|| format!("opening {}", redact(location)))