#241: on production, All Subscriptions' query runs in 31 ms, unread oldest-first in 30, the feed list's count in 5, generic plans included (EXPLAIN ANALYZE, BUFFERS, read only), against 0.3-1.6 s in ipx. A new Postgres connection takes 80-330 ms to open, and a page load asks for more than are idle: four of six slow loads came as the pool grew (3 to 6, 2 to 6, 3 to 5, 6 to 8), and 20 requests at once took a median 414 ms while the pool grew from 8 to 20. The daemon and the directory's server now keep their whole pool open (min = max); a command such as `ipx status`, and the tests, which share the server's 100 connections, open as they need. #246: the page reports list_shown, the ms from asking for a list to drawing it, with the view and filter, under the click's trace; background refreshes are not timed. Timings skip report()'s once-a-minute limit, which is for errors in a loop. A dashboard panel and table, and the prod check, read it. #245: the load test's browse scenario subscribes by OPML and leaves the Directory, which is ipx-directory's since #204, to it. CLAUDE.md records the owner's leave to EXPLAIN (ANALYZE, BUFFERS) any query on production, read only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -63,7 +63,7 @@ lines had no trace id, the access log's time was `ms`, and each event was logged
|
||||
| `ipx::scan` | `ev` and the event's own: `feed`, `new`, `downloaded`, `failed`, `bytes`, `msg`, `url`, `feeds`, `reason`, `from`, `to`; on a failure `error.type` and, from an HTTP error, `http.response.status_code` | the daemon's events, one line each, in words; warnings are feed and download failures |
|
||||
| `ipx::io` | `ev`, `feeds`, `pending`, `downloaded` on the `status` reply | commands arriving (`-> {...}`) and the healthcheck's answer |
|
||||
| `ipx` | message, sometimes fields | start-up, shutdown, account and config messages |
|
||||
| `ipx::client` | `ev`, `user`, `feed`, `enclosure`, `deadair`; `msg`, `src`, `line` on an error; `ms` on a timing | what the page reports from the browser (#190): `page_error`, `rejection` and `player_error` as warnings; `play_start` (ms from play to sound) and `stall` (ms waiting mid-play) as info. Lines carry the trace of the action in the page: its API requests send `traceparent`, one trace id per tap or key |
|
||||
| `ipx::client` | `ev`, `user`, `feed`, `enclosure`, `deadair`; `msg`, `src`, `line` on an error; `ms` on a timing | what the page reports from the browser (#190): `page_error`, `rejection` and `player_error` as warnings; `play_start` (ms from play to sound), `stall` (ms waiting mid-play) and `list_shown` (ms from a click to its list drawn, with `feed` the view and `filter`, #246) as info. Lines carry the trace of the action in the page: its API requests send `traceparent`, one trace id per tap or key |
|
||||
|
||||
Events (`ev`): `feed_start`, `feed_done` (new, downloaded, failed, torrents), `feed_skip` (not due,
|
||||
routine), `feed_error` (msg), `feed_moved` (from, to: a permanent redirect followed, the address
|
||||
@@ -215,11 +215,12 @@ a count says something happened, the lines and traces say why.
|
||||
```
|
||||
$Q metric 'sum by (ev, msg) (count_over_time({container="iPX"} |= "\"target\":\"ipx::client\"" | json | __error__="" | level="WARN" [$range]))' 7d
|
||||
$Q logs '{container="iPX"} |= "\"target\":\"ipx::client\"" | json | ev=~"play_start|stall" | line_format "{{.timestamp}} {{.user}} {{.ev}} {{.ms}}ms deadair={{.deadair}} {{.feed}}"' 24h
|
||||
$Q metric 'quantile_over_time(0.95, {container="iPX"} |= "\"ev\":\"list_shown\"" | json | __error__="" | unwrap ms [$range]) by (feed, filter)' 24h
|
||||
```
|
||||
A `player_error` is playback that failed in front of someone: `hls.js: ...` is DeadAir's
|
||||
playlist (its trace has the playlist and audio requests), `media error 4` a file the browser
|
||||
cannot play. A `page_error` or `rejection` is a bug in the page; `src` and `line` point into
|
||||
the minified `app.js`, so match the message to `web/src`. `play_start` over 2000 ms, or a
|
||||
the minified `app.js`, so match the message to `web/src`. A `list_shown` over 1000 ms is a list someone waited for: its trace says whether the API or the network held it. `play_start` over 2000 ms, or a
|
||||
`stall`, is worth a trace: was it the server, Cloudflare, or the network.
|
||||
|
||||
`docker logs --since 10m iPX 2>&1 | tail -5` shows the newest lines when Loki is behind.
|
||||
@@ -349,8 +350,9 @@ container after a deploy.
|
||||
|
||||
## Rules
|
||||
|
||||
`docker logs` and `docker exec iPX ipx ...` are fine. Do not query production's Postgres
|
||||
directly: ask the user if a question needs the database.
|
||||
`docker logs` and `docker exec iPX ipx ...` are fine. On production's Postgres, `EXPLAIN (ANALYZE,
|
||||
BUFFERS)` of any query is allowed, read only, as CLAUDE.md's "Production's database" shows; any
|
||||
other query, ask the user first.
|
||||
|
||||
## What to do with what you find
|
||||
|
||||
|
||||
Reference in New Issue
Block a user