The API's coverage is measured, and every suite's is in Grafana (#394)

The cargo tests put web.rs at 17% and daemon.rs at 10%, because the
browser suite, which drives the API, ran a daemon nobody measured.
`sh scripts/coverage.sh api` builds an instrumented ipx and
ipx-directory in the capped coverage container and runs the browser
suite here against them (IPX_API_COVERAGE): web.rs is at 80% and
daemon.rs at 77%. What it does not reach: access.rs (Cloudflare Access,
0%; its unit tests 73%) and the directory's public pages (site.rs, 4%).

The coverage image is Ubuntu 24.04 now, as code-server is: built on
Debian trixie, the binaries needed glibc 2.41 and code-server has 2.39.
In that mode the suite starts those binaries rather than building its
own, its tests' `ipx` commands too (global-setup's IPX_BIN), passes
LLVM_PROFILE_FILE to both servers, and stops them with SIGTERM, since
an instrumented binary writes its profile as it exits and Playwright
otherwise kills them outright.

Reports go to target/coverage/cargo, api and page. Each run ends with
scripts/coverage-metrics.ts, which writes every suite's lines covered
and in all, by crate, to node-exporter's textfile directory; the
monitoring project's node-exporter now reads it (one flag, its compose
file backed up beside it), so Prometheus keeps them as
ipx_coverage_lines, and the iPX dashboard's Coverage row draws them:
each suite now, over time, and by part. The cargo tests' totals hold
their own code, which runs whole, so they read higher than the code
alone; the panels say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-07 03:33:49 +00:00
parent 352e2b77a5
commit 194a9250e7
10 changed files with 226 additions and 47 deletions

View File

@@ -417,6 +417,37 @@ panel("logs", "ipx log", 24, 12, 0,
description="Without the event stream's requests, not-due skips and healthcheck status calls.", options=LOG_OPTS)
y += 12
# ---- Coverage: what the tests reach (#390, #394). scripts/coverage.sh writes each run's numbers
# for node-exporter's textfile collector, so they are job node's, and stay as the last run left them.
COV = 'ipx_coverage_lines{job="node"'
def cov(suite, by=""):
"""Lines covered over lines there are, for a suite, summed over its parts or by `by`."""
group = f" by ({by})" if by else ""
return (f'sum{group} ({COV},suite="{suite}",state="covered"}}) / '
f'sum{group} ({COV},suite="{suite}",state="total"}})')
row("Coverage")
COV_STEPS = [{"color": "red", "value": None}, {"color": "orange", "value": 0.6}, {"color": "green", "value": 0.8}]
stat("Rust, cargo tests", prom(cov("cargo")), 0, unit="percentunit", thresholds=COV_STEPS,
desc="Lines of ipx, ipx-directory and ipx-feed the cargo tests reach: sh scripts/coverage.sh. The tests' own code counts too, as it runs whole, so this reads higher than the code alone would.")
stat("Rust, browser tests", prom(cov("browser")), 4, unit="percentunit", thresholds=COV_STEPS,
desc="Lines of ipx and ipx-directory the browser suite reaches through the API, against an instrumented build: sh scripts/coverage.sh api.")
stat("Page, browser tests", prom(cov("page")), 8, unit="percentunit", thresholds=COV_STEPS,
desc="Lines of web/src the browser suite runs: npm run coverage:page, or coverage.sh api.")
stat("Since the last coverage run", prom('time() - max(ipx_coverage_run_timestamp_seconds{job="node"})'), 12, unit="s",
desc="The newest of the three suites' reports. Each suite's own is in the table beside it.")
table("Last run of each suite", [prom('time() - ipx_coverage_run_timestamp_seconds{job="node"}', instant=True)], 16, w=8, h=4,
rename={"Value": "ago", "suite": "suite"}, unit="s")
y += 4
ts("Lines covered", [prom(cov("cargo"), legend="Rust, cargo tests"), prom(cov("browser"), ref="B", legend="Rust, browser tests"),
prom(cov("page"), ref="C", legend="Page, browser tests")],
0, w=12, unit="percentunit", desc="As each suite's last run found them: a step is a run that found more, or less.")
table("By part", [prom(cov("cargo", "part"), ref="A", instant=True), prom(cov("browser", "part"), ref="B", instant=True),
prom(cov("page", "part"), ref="C", instant=True)],
12, w=12, unit="percentunit", rename={"Value #A": "cargo tests", "Value #B": "browser tests", "Value #C": "page"},
desc="Each crate's lines, and the page's, by the suite that reached them. A crate has more lines under the cargo tests: their own code, which a build for the browser suite leaves out.")
panels[-1]["transformations"].insert(0, {"id": "merge", "options": {}})
y += 8
dash = {
"uid": "ipx", "title": "iPX", "tags": ["ipx"], "timezone": "browser", "schemaVersion": 39,
"time": {"from": "now-24h", "to": "now"}, "refresh": "1m", "editable": True,