The API's coverage is measured, and every suite's is in Grafana (#394)
The cargo tests put web.rs at 17% and daemon.rs at 10%, because the browser suite, which drives the API, ran a daemon nobody measured. `sh scripts/coverage.sh api` builds an instrumented ipx and ipx-directory in the capped coverage container and runs the browser suite here against them (IPX_API_COVERAGE): web.rs is at 80% and daemon.rs at 77%. What it does not reach: access.rs (Cloudflare Access, 0%; its unit tests 73%) and the directory's public pages (site.rs, 4%). The coverage image is Ubuntu 24.04 now, as code-server is: built on Debian trixie, the binaries needed glibc 2.41 and code-server has 2.39. In that mode the suite starts those binaries rather than building its own, its tests' `ipx` commands too (global-setup's IPX_BIN), passes LLVM_PROFILE_FILE to both servers, and stops them with SIGTERM, since an instrumented binary writes its profile as it exits and Playwright otherwise kills them outright. Reports go to target/coverage/cargo, api and page. Each run ends with scripts/coverage-metrics.ts, which writes every suite's lines covered and in all, by crate, to node-exporter's textfile directory; the monitoring project's node-exporter now reads it (one flag, its compose file backed up beside it), so Prometheus keeps them as ipx_coverage_lines, and the iPX dashboard's Coverage row draws them: each suite now, over time, and by part. The cargo tests' totals hold their own code, which runs whole, so they read higher than the code alone; the panels say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -417,6 +417,37 @@ panel("logs", "ipx log", 24, 12, 0,
|
||||
description="Without the event stream's requests, not-due skips and healthcheck status calls.", options=LOG_OPTS)
|
||||
y += 12
|
||||
|
||||
# ---- Coverage: what the tests reach (#390, #394). scripts/coverage.sh writes each run's numbers
|
||||
# for node-exporter's textfile collector, so they are job node's, and stay as the last run left them.
|
||||
COV = 'ipx_coverage_lines{job="node"'
|
||||
def cov(suite, by=""):
|
||||
"""Lines covered over lines there are, for a suite, summed over its parts or by `by`."""
|
||||
group = f" by ({by})" if by else ""
|
||||
return (f'sum{group} ({COV},suite="{suite}",state="covered"}}) / '
|
||||
f'sum{group} ({COV},suite="{suite}",state="total"}})')
|
||||
row("Coverage")
|
||||
COV_STEPS = [{"color": "red", "value": None}, {"color": "orange", "value": 0.6}, {"color": "green", "value": 0.8}]
|
||||
stat("Rust, cargo tests", prom(cov("cargo")), 0, unit="percentunit", thresholds=COV_STEPS,
|
||||
desc="Lines of ipx, ipx-directory and ipx-feed the cargo tests reach: sh scripts/coverage.sh. The tests' own code counts too, as it runs whole, so this reads higher than the code alone would.")
|
||||
stat("Rust, browser tests", prom(cov("browser")), 4, unit="percentunit", thresholds=COV_STEPS,
|
||||
desc="Lines of ipx and ipx-directory the browser suite reaches through the API, against an instrumented build: sh scripts/coverage.sh api.")
|
||||
stat("Page, browser tests", prom(cov("page")), 8, unit="percentunit", thresholds=COV_STEPS,
|
||||
desc="Lines of web/src the browser suite runs: npm run coverage:page, or coverage.sh api.")
|
||||
stat("Since the last coverage run", prom('time() - max(ipx_coverage_run_timestamp_seconds{job="node"})'), 12, unit="s",
|
||||
desc="The newest of the three suites' reports. Each suite's own is in the table beside it.")
|
||||
table("Last run of each suite", [prom('time() - ipx_coverage_run_timestamp_seconds{job="node"}', instant=True)], 16, w=8, h=4,
|
||||
rename={"Value": "ago", "suite": "suite"}, unit="s")
|
||||
y += 4
|
||||
ts("Lines covered", [prom(cov("cargo"), legend="Rust, cargo tests"), prom(cov("browser"), ref="B", legend="Rust, browser tests"),
|
||||
prom(cov("page"), ref="C", legend="Page, browser tests")],
|
||||
0, w=12, unit="percentunit", desc="As each suite's last run found them: a step is a run that found more, or less.")
|
||||
table("By part", [prom(cov("cargo", "part"), ref="A", instant=True), prom(cov("browser", "part"), ref="B", instant=True),
|
||||
prom(cov("page", "part"), ref="C", instant=True)],
|
||||
12, w=12, unit="percentunit", rename={"Value #A": "cargo tests", "Value #B": "browser tests", "Value #C": "page"},
|
||||
desc="Each crate's lines, and the page's, by the suite that reached them. A crate has more lines under the cargo tests: their own code, which a build for the browser suite leaves out.")
|
||||
panels[-1]["transformations"].insert(0, {"id": "merge", "options": {}})
|
||||
y += 8
|
||||
|
||||
dash = {
|
||||
"uid": "ipx", "title": "iPX", "tags": ["ipx"], "timezone": "browser", "schemaVersion": 39,
|
||||
"time": {"from": "now-24h", "to": "now"}, "refresh": "1m", "editable": True,
|
||||
|
||||
Reference in New Issue
Block a user