Compression across the board: directory answers, uploads, metrics, traces, the page (#253)

Measured 2026-10-16, uncompressed until now: the directory's answers (the feed list ipx reads,
603 KB of JSON, ~134 ms of a 159 ms Directory request), ipx's snapshot upload (~100 KB each half
hour), both /metrics (4.7 and 0.9 KB every 15 s) and every trace export to Tempo. Already
compressed: ipx-web's answers (zstd, #239), the home page and the directory's public pages
through Cloudflare (brotli, zstd).

- The directory compresses with brotli, zstd or gzip as asked, and unpacks a gzipped request.
- ipx sends its snapshot gzipped; the directory is deployed first.
- Both /metrics answer gzipped, as Prometheus asks.
- opentelemetry-otlp's gzip-http: OTEL_EXPORTER_OTLP_COMPRESSION=gzip compresses the export, set
  for all three in production's compose file; Caddy's exporter reads the same variable.
- web/build.mjs writes .br (quality 11) and .gz (level 9) beside each text file, and Caddy serves
  them in place (file_server precompressed): app.js 30 KB where zstd on the fly gave 37, app.css
  11 where 14. A .br or .gz asked for by name is refused, or /admin.js.br would be the admin
  page's script without the check that keeps it to admins; in a handle of its own, since a bare
  respond runs after the handles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/directory/CHANGELOG.md

@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Answers and pages go out compressed, brotli, zstd or gzip as the caller takes them: the feed list ipx asks for was 603 KB of JSON. A snapshot may come gzipped, `/metrics` answers gzipped, and with `OTEL_EXPORTER_OTLP_COMPRESSION=gzip` traces go to the collector compressed (#253).
 - With `apple_country` set, Apple Podcasts' top charts for that country (the top 100 and the top 50 of each category) come into the directory every week, beside what servers share, without counting as subscribers; and a warning says when Apple's categories no longer match ipx's (#252).
 
 ### Changed

crates/directory/Cargo.toml

@@ -26,10 +26,12 @@ serde_json.workspace = true
 sha2.workspace = true
 tokio.workspace = true
 toml.workspace = true
+tower-http.workspace = true
 tracing.workspace = true
 tracing-opentelemetry.workspace = true
 tracing-subscriber.workspace = true
 url.workspace = true
 
 [dev-dependencies]
+flate2.workspace = true
 tower.workspace = true

crates/directory/src/lib.rs

@@ -268,6 +268,12 @@ pub fn router(dir: Shared) -> Router {
         .merge(v1)
         // Anyone may browse, no token, read only (#215).
         .merge(site::router())
+        // Brotli, zstd or gzip as the caller takes them: the feed list went to ipx as 603 KB of
+        // JSON, most of a Directory page's time (#253). Inside the access log, which records what
+        // was sent.
+        .layer(tower_http::compression::CompressionLayer::new().br(true).zstd(true).gzip(true))
+        // And takes a request's body gzipped: ipx sends its snapshot so, about 100 KB as JSON.
+        .layer(tower_http::decompression::RequestDecompressionLayer::new().gzip(true))
         .layer(axum::middleware::from_fn(access_log))
         .with_state(dir)
 }
@@ -477,6 +483,45 @@ mod tests {
         (status, serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null))
     }
 
+    #[tokio::test]
+    async fn a_snapshot_may_come_gzipped() {
+        use std::io::Write as _;
+        let (app, _) = app().await;
+        let json = serde_json::json!([{"url": "https://example.com/feed.xml", "subscribers": 2}]).to_string();
+        let mut gz = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());
+        gz.write_all(json.as_bytes()).unwrap();
+        // As ipx sends it (#253).
+        let req = axum::http::Request::builder()
+            .method("POST")
+            .uri("/v1/snapshot")
+            .header("authorization", "Bearer secret-rw")
+            .header("content-type", "application/json")
+            .header("content-encoding", "gzip")
+            .body(Body::from(gz.finish().unwrap()))
+            .unwrap();
+        let resp = app.clone().oneshot(req).await.unwrap();
+        assert_eq!(resp.status(), 200);
+        let got: serde_json::Value = serde_json::from_slice(&axum::body::to_bytes(resp.into_body(), 1 << 20).await.unwrap()).unwrap();
+        assert_eq!(got["accepted"], 1);
+    }
+
+    #[tokio::test]
+    async fn answers_come_compressed_as_the_caller_asks() {
+        let (app, _) = app().await;
+        for (asked, sent) in [("br", Some("br")), ("zstd", Some("zstd")), ("gzip", Some("gzip")), ("identity", None)] {
+            // A page, not an empty feed list: under 32 bytes is sent as it is.
+            let req = axum::http::Request::builder()
+                .uri("/")
+                .header("accept-encoding", asked)
+                .body(Body::empty())
+                .unwrap();
+            let resp = app.clone().oneshot(req).await.unwrap();
+            assert_eq!(resp.status(), 200);
+            let got = resp.headers().get("content-encoding").map(|v| v.to_str().unwrap().to_owned());
+            assert_eq!(got.as_deref(), sent, "asked for {asked} (#253)");
+        }
+    }
+
     async fn get_page(app: &Router, path: &str) -> (u16, String, Option<String>) {
         let req = axum::http::Request::builder().uri(path).body(Body::empty()).unwrap();
         let resp = app.clone().oneshot(req).await.unwrap();

crates/directory/src/metrics.rs

@@ -51,7 +51,9 @@ pub async fn serve(listener: tokio::net::TcpListener) {
         axum::routing::get(move || async move {
             ([(axum::http::header::CONTENT_TYPE, "text/plain; version=0.0.4")], render(started))
         }),
-    );
+    )
+    // Prometheus asks for gzip (#253).
+    .layer(tower_http::compression::CompressionLayer::new().gzip(true));
     if let Err(e) = axum::serve(listener, app).await {
         tracing::error!(error = %e, "metrics stopped");
     }