Traces keep feed and file addresses without their query or password (#225)
The fetch span and the download span recorded the whole address, so a private feed's credential (Patreon's ?auth=) went to Tempo with every scan, readable by anyone with Grafana. feed::redact drops the query, fragment and user:password; a token in the path would still go, noted where it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
- With `metrics_bind` set, the directory serves Prometheus metrics on a port of its own: memory, open files, and the feeds it has fetched and their bytes. A feed check's trace says how large the feed was (#224).
|
||||
|
||||
### Security
|
||||
|
||||
- A feed check's trace keeps the feed's address without its query or password (#225).
|
||||
|
||||
### Changed
|
||||
|
||||
- The directory uses mimalloc instead of glibc's allocator, so it gives back memory it has freed instead of holding over a gigabyte while idle (#224).
|
||||
|
||||
@@ -73,7 +73,24 @@ pub static FETCHED_BYTES: std::sync::atomic::AtomicU64 = std::sync::atomic::Atom
|
||||
/// than be redirected on every read. A temporary redirect (302, 307) moves nothing.
|
||||
/// `system-proxy` reads http_proxy/https_proxy.
|
||||
/// `auth` is a username and password, sent only to the feed's own host.
|
||||
#[tracing::instrument(skip_all, fields(url = %feed_url, http.response.body.size))]
|
||||
/// An address as a trace may keep it: without its query or user and password, where a private
|
||||
/// feed carries its credential (Patreon's `?auth=`), which Tempo kept for anyone reading it (#225).
|
||||
/// shortcut: a token in the path (`/feed/<token>.xml`) stays; strip paths too if a feed does that.
|
||||
pub fn redact(url: &str) -> String {
|
||||
match url::Url::parse(url) {
|
||||
Ok(mut u) => {
|
||||
u.set_query(None);
|
||||
u.set_fragment(None);
|
||||
let _ = u.set_username("");
|
||||
let _ = u.set_password(None);
|
||||
u.to_string()
|
||||
}
|
||||
// Not an address ipx could fetch; keep what comes before anything that could be a query.
|
||||
Err(_) => url.split(['?', '#', '@']).next().unwrap_or_default().to_owned(),
|
||||
}
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip_all, fields(url = %redact(feed_url), http.response.body.size))]
|
||||
pub async fn fetch(
|
||||
client: &reqwest::Client,
|
||||
feed_url: &str,
|
||||
@@ -1173,6 +1190,14 @@ const PAID_HOSTS: &[&str] =
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn redact_drops_what_carries_a_credential() {
|
||||
assert_eq!(super::redact("https://www.patreon.com/rss/x?auth=SECRET&show=1"), "https://www.patreon.com/rss/x");
|
||||
assert_eq!(super::redact("https://me:pw@example.com/feed.xml#top"), "https://example.com/feed.xml");
|
||||
assert_eq!(super::redact("https://example.com/feed.xml"), "https://example.com/feed.xml");
|
||||
assert_eq!(super::redact("not a url?token=SECRET"), "not a url");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_address_carrying_someones_access_looks_private() {
|
||||
let private = super::url_looks_private;
|
||||
|
||||
@@ -16,6 +16,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
- `/metrics` counts the feeds fetched and the bytes of their bodies (`ipx_feed_fetches_total`, `ipx_feed_bytes_total`), and a feed's trace says how large its body was (#224).
|
||||
|
||||
### Security
|
||||
|
||||
- Traces no longer keep a feed's or file's address with its query or password, where private feeds carry their credentials, such as Patreon's `auth` (#225).
|
||||
|
||||
### Changed
|
||||
|
||||
- ipx uses mimalloc instead of glibc's allocator, so memory a night of scans needed goes back to the system instead of staying held until a restart (#224).
|
||||
|
||||
@@ -2139,7 +2139,7 @@ fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Pol
|
||||
|
||||
/// Downloads one file and records it. None when it turned out to be a torrent, handed off to run
|
||||
/// on its own, which records it when it finishes.
|
||||
#[tracing::instrument(name = "download", skip_all, fields(feed = feed_id, enclosure = enclosure, url = url))]
|
||||
#[tracing::instrument(name = "download", skip_all, fields(feed = feed_id, enclosure = enclosure, url = %feed::redact(url)))]
|
||||
async fn fetch_one(
|
||||
ctx: &Arc<Ctx>,
|
||||
feed_id: &str,
|
||||
|
||||
Reference in New Issue
Block a user