rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Compression across the board: directory answers, uploads, metrics, traces, the page (#253)

Browse Source 
Measured 2026-10-16, uncompressed until now: the directory's answers (the feed list ipx reads,
603 KB of JSON, ~134 ms of a 159 ms Directory request), ipx's snapshot upload (~100 KB each half
hour), both /metrics (4.7 and 0.9 KB every 15 s) and every trace export to Tempo. Already
compressed: ipx-web's answers (zstd, #239), the home page and the directory's public pages
through Cloudflare (brotli, zstd).

- The directory compresses with brotli, zstd or gzip as asked, and unpacks a gzipped request.
- ipx sends its snapshot gzipped; the directory is deployed first.
- Both /metrics answer gzipped, as Prometheus asks.
- opentelemetry-otlp's gzip-http: OTEL_EXPORTER_OTLP_COMPRESSION=gzip compresses the export, set
  for all three in production's compose file; Caddy's exporter reads the same variable.
- web/build.mjs writes .br (quality 11) and .gz (level 9) beside each text file, and Caddy serves
  them in place (file_server precompressed): app.js 30 KB where zstd on the fly gave 37, app.css
  11 where 14. A .br or .gz asked for by name is refused, or /admin.js.br would be the admin
  page's script without the check that keeps it to admins; in a handle of its own, since a bare
  respond runs after the handles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-16 14:02:39 +00:00
parent 3d94bce4f5
commit 1949f3e493
14 changed files with 117 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
crates/directory/CHANGELOG.md
View File

@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Answers and pages go out compressed, brotli, zstd or gzip as the caller takes them: the feed list ipx asks for was 603 KB of JSON. A snapshot may come gzipped, `/metrics` answers gzipped, and with `OTEL_EXPORTER_OTLP_COMPRESSION=gzip` traces go to the collector compressed (#253).
- With `apple_country` set, Apple Podcasts' top charts for that country (the top 100 and the top 50 of each category) come into the directory every week, beside what servers share, without counting as subscribers; and a warning says when Apple's categories no longer match ipx's (#252).
### Changed

2
crates/directory/Cargo.toml
View File

@@ -26,10 +26,12 @@ serde_json.workspace = true
sha2.workspace = true
tokio.workspace = true
toml.workspace = true
tower-http.workspace = true
tracing.workspace = true
tracing-opentelemetry.workspace = true
tracing-subscriber.workspace = true
url.workspace = true
[dev-dependencies]
flate2.workspace = true
tower.workspace = true

45
crates/directory/src/lib.rs
View File

@@ -268,6 +268,12 @@ pub fn router(dir: Shared) -> Router {
.merge(v1)
// Anyone may browse, no token, read only (#215).
.merge(site::router())
// Brotli, zstd or gzip as the caller takes them: the feed list went to ipx as 603 KB of
// JSON, most of a Directory page's time (#253). Inside the access log, which records what
// was sent.
.layer(tower_http::compression::CompressionLayer::new().br(true).zstd(true).gzip(true))
// And takes a request's body gzipped: ipx sends its snapshot so, about 100 KB as JSON.
.layer(tower_http::decompression::RequestDecompressionLayer::new().gzip(true))
.layer(axum::middleware::from_fn(access_log))
.with_state(dir)
}
@@ -477,6 +483,45 @@ mod tests {
(status, serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null))
}
#[tokio::test]
async fn a_snapshot_may_come_gzipped() {
use std::io::Write as _;
let (app, _) = app().await;
let json = serde_json::json!([{"url": "https://example.com/feed.xml", "subscribers": 2}]).to_string();
let mut gz = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());
gz.write_all(json.as_bytes()).unwrap();
// As ipx sends it (#253).
let req = axum::http::Request::builder()
.method("POST")
.uri("/v1/snapshot")
.header("authorization", "Bearer secret-rw")
.header("content-type", "application/json")
.header("content-encoding", "gzip")
.body(Body::from(gz.finish().unwrap()))
.unwrap();
let resp = app.clone().oneshot(req).await.unwrap();
assert_eq!(resp.status(), 200);
let got: serde_json::Value = serde_json::from_slice(&axum::body::to_bytes(resp.into_body(), 1 << 20).await.unwrap()).unwrap();
assert_eq!(got["accepted"], 1);
}
#[tokio::test]
async fn answers_come_compressed_as_the_caller_asks() {
let (app, _) = app().await;
for (asked, sent) in [("br", Some("br")), ("zstd", Some("zstd")), ("gzip", Some("gzip")), ("identity", None)] {
// A page, not an empty feed list: under 32 bytes is sent as it is.
let req = axum::http::Request::builder()
.uri("/")
.header("accept-encoding", asked)
.body(Body::empty())
.unwrap();
let resp = app.clone().oneshot(req).await.unwrap();
assert_eq!(resp.status(), 200);
let got = resp.headers().get("content-encoding").map(|v| v.to_str().unwrap().to_owned());
assert_eq!(got.as_deref(), sent, "asked for {asked} (#253)");
}
}
async fn get_page(app: &Router, path: &str) -> (u16, String, Option<String>) {
let req = axum::http::Request::builder().uri(path).body(Body::empty()).unwrap();
let resp = app.clone().oneshot(req).await.unwrap();

4
crates/directory/src/metrics.rs
View File

@@ -51,7 +51,9 @@ pub async fn serve(listener: tokio::net::TcpListener) {
axum::routing::get(move || async move {
([(axum::http::header::CONTENT_TYPE, "text/plain; version=0.0.4")], render(started))
}),
);
)
// Prometheus asks for gzip (#253).
.layer(tower_http::compression::CompressionLayer::new().gzip(true));
if let Err(e) = axum::serve(listener, app).await {
tracing::error!(error = %e, "metrics stopped");
}

1
crates/ipx/CHANGELOG.md
View File

@@ -22,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- With `OTEL_EXPORTER_OTLP_COMPRESSION=gzip`, traces go to the collector compressed; `/metrics` answers gzipped, and a snapshot goes to the directory gzipped (#253).
- A Directory request's trace shows the call to the directory whole, with the size of its answer, and the steps that match it against your subscriptions (#250).
- The page's report of how long a list took to show after a click (`list_shown`, with the view and filter) is logged under that click's trace (#246).

1
crates/ipx/Cargo.toml
View File

@@ -9,6 +9,7 @@ ipx-directory.workspace = true
ammonia.workspace = true
anyhow.workspace = true
argon2.workspace = true
flate2.workspace = true
atom_syndication.workspace = true
axum.workspace = true
chrono.workspace = true

11
crates/ipx/src/directory.rs
View File

@@ -65,8 +65,17 @@ async fn call<T: DeserializeOwned>(ctx: &Ctx, method: reqwest::Method, path: &st
if let Some(tp) = ipx_directory::otel::traceparent() {
req = req.header("traceparent", tp);
}
// A snapshot is about 100 KB of JSON, sent gzipped; the directory unpacks it (#253).
if let Some(b) = body {
req = req.json(b);
use std::io::Write as _;
let json = serde_json::to_vec(b).map_err(|e| DirError::Unreachable(format!("{e:#}")))?;
let mut gz = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());
gz.write_all(&json).map_err(|e| DirError::Unreachable(format!("{e:#}")))?;
let packed = gz.finish().map_err(|e| DirError::Unreachable(format!("{e:#}")))?;
req = req
.header(reqwest::header::CONTENT_TYPE, "application/json")
.header(reqwest::header::CONTENT_ENCODING, "gzip")
.body(packed);
}
let resp = req.send().await.map_err(|e| DirError::Unreachable(format!("{e:#}")))?;
let status = resp.status();

4
crates/ipx/src/metrics.rs
View File

@@ -135,7 +135,9 @@ pub fn spawn(bind: &str, s: Sources) -> anyhow::Result<std::net::SocketAddr> {
let s = s.clone();
async move { ([(axum::http::header::CONTENT_TYPE, "text/plain; version=0.0.4")], render(&s)) }
}),
);
)
// Prometheus asks for gzip, every 15 s (#253).
.layer(tower_http::compression::CompressionLayer::new().gzip(true));
match tokio::net::TcpListener::from_std(listener) {
Ok(l) => {
if let Err(e) = axum::serve(l, app).await {