rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Metrics and traces for what had none, and alerts on them (#365-#377)

Browse Source 
Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.

Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
  word, from SeaORM's metric callback; and the wait for one, for
  hand-written SQL in db.rs, measured as the call less the statement
  (the callback runs in the poll that returns the result, so the time
  read after the await is that statement's) (#367). SeaORM's own queries
  are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
  event passes once: feed failures by error.type, new items, downloads
  saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
  route on the response, or each feed's page was a row) and status;
  feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
  the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
  calls deprecated, because the browser tests' Caddy is 2.6 (#371).

Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
  (#372). Doing so showed sync_group reading every account and its
  subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
  is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
  catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
  ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
  span id in each request's traceparent, where the parent was random and
  pointed at nothing, and reports the tap (ids, start, its last answer)
  when the next begins or the page is hidden. opentelemetry 0.33 gives a
  span builder no way to take ids, so the shared tracer provider has an
  id generator that takes given ids inside with_ids, on one thread.

Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.

The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-25 16:18:06 +00:00
parent c788112190
commit dd91602a49
23 changed files with 768 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
web/CHANGELOG.md
View File

@@ -21,6 +21,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Caddy's own metrics, its requests and their times and the 502s ipx never sees, for Prometheus on a port of their own, `METRICS_BIND` (#371).
- A tap or a key is its trace's root span: the page sends ipx when it began and when its last answer came, so Tempo shows how long it took beside the requests it made (#376).
- Fonts of your choosing in Settings, beside the theme, kept by each browser as the theme is: one for the interface and one for reading an item's title and text, which is the interface's until chosen, with a little more room between the lines in a serif. Inter, your system's, the reading fonts Atkinson Hyperlegible, Lexend, Literata, Source Serif and OpenDyslexic, your device's serif, or monospace. The default keeps each theme's face as it was (#359).
## [1.3.0] - 2026-10-23

12
web/Caddyfile
View File

@@ -6,12 +6,19 @@
# OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_SERVICE_NAME), continuing the page's traceparent
# and passing it to ipx, so one trace runs browser, Caddy, ipx (#213). Off: none,
# and no export failing every few seconds where there is nothing to send to.
# METRICS_BIND where to serve Caddy's metrics for Prometheus (:9180); keep it off the tunnel
# TUNNEL_PEER the one address whose Cf-Access-* headers are passed on: where cloudflared's
# requests come from. Nobody else's are, so nobody else can claim to be signed in
# by Cloudflare. Unset passes none.
{
admin off
auto_https off
# Requests, their times and upstream errors as Prometheus metrics, served below (#371).
# Nested in servers, which Caddy 2.11 calls deprecated, for the browser tests' Caddy, 2.6,
# which knows no other way; the global `metrics` option once both are 2.8 or later.
servers {
metrics
}
# Everything else Caddy logs, its errors among them, which carry the request too: the same
# filter as the access log's below.
log default {
@@ -28,6 +35,11 @@
}
}
# Caddy's own metrics (#371), on a port of their own: the tunnel reaches only WEB_BIND's.
http://{$METRICS_BIND::9180} {
metrics
}
(tracing-on) {
tracing {
# Named for the request, as ipx names its own (#286): one name for all left every span in

28
web/src/util.ts
View File

@@ -103,14 +103,29 @@ function spanText(secs: number){
/// LAN address over plain HTTP does not have. A trace ends after a minute with no request in it:
/// the requests a live update or a woken laptop sets off hours later used to join the last tap's
/// trace, which then lasted nine hours (#219). A play's position saves, every ten seconds, keep it.
let traceId='', traceAt=0;
/// A tap or a key is the trace's root span (#376): its requests name it as their parent, and once
/// the next one begins, or the page is hidden, the page reports it, from the tap to its last
/// answer, for ipx to send on to Tempo with these ids.
let traceId='', traceAt=0, rootId='', rootAt=0, rootKind='', answeredAt=0;
const hex=(n: number)=>Array.from(crypto.getRandomValues(new Uint8Array(n)),b=>b.toString(16).padStart(2,'0')).join('');
for(const t of ['pointerdown','keydown']) window.addEventListener(t,()=>{ traceId=hex(16); traceAt=Date.now(); },{capture:true,passive:true});
function endTap(){
if(rootKind && answeredAt>rootAt)
report('tap',{trace:traceId,span:rootId,start:rootAt,ms:Math.min(answeredAt-rootAt,60_000),msg:rootKind});
rootKind='';
}
function newTrace(kind: string){
endTap();
traceId=hex(16); rootId=hex(8); rootAt=Date.now(); rootKind=kind; answeredAt=0;
}
for(const t of ['pointerdown','keydown']) window.addEventListener(t,()=>{ newTrace(t); traceAt=Date.now(); },{capture:true,passive:true});
window.addEventListener('pagehide',endTap);
document.addEventListener('visibilitychange',()=>{ if(document.visibilityState==='hidden') endTap(); });
function traceparent(){
const now=Date.now();
if(!traceId||now-traceAt>60_000) traceId=hex(16);
// No tap to hang from: a live update's or a timer's requests, a trace with no root of its own.
if(!traceId||now-traceAt>60_000) newTrace('');
traceAt=now;
return `00-${traceId}-${hex(8)}-01`;
return `00-${traceId}-${rootId}-01`;
}
/// Something that went wrong, or how long something took, in the browser, for the server's log
@@ -130,8 +145,11 @@ window.addEventListener('unhandledrejection',e=>report('rejection',{msg:String(e
async function api(url: string, opts?: RequestInit): Promise<any>{
// The browser's "Failed to fetch" said nothing anyone could act on (#304).
const r = await fetch(url,{headers:{'Content-Type':'application/json',traceparent:traceparent()},...opts})
const tp=traceparent(), mine=traceId;
const r = await fetch(url,{headers:{'Content-Type':'application/json',traceparent:tp},...opts})
.catch(()=>{ throw new Error('Unable to reach iPX. Check your connection and try again.'); });
// The tap lasts until its last answer.
if(mine===traceId) answeredAt=Date.now();
if(r.status===401){ location.href='/login'; throw new Error('signed out'); }
if(!r.ok){
// A page in place of ipx's words is Cloudflare or Caddy answering for it: the message became a