Traces keep feed and file addresses without their query or password (#225)
The fetch span and the download span recorded the whole address, so a private feed's credential (Patreon's ?auth=) went to Tempo with every scan, readable by anyone with Grafana. feed::redact drops the query, fragment and user:password; a token in the path would still go, noted where it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
- With `metrics_bind` set, the directory serves Prometheus metrics on a port of its own: memory, open files, and the feeds it has fetched and their bytes. A feed check's trace says how large the feed was (#224).
|
- With `metrics_bind` set, the directory serves Prometheus metrics on a port of its own: memory, open files, and the feeds it has fetched and their bytes. A feed check's trace says how large the feed was (#224).
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
- A feed check's trace keeps the feed's address without its query or password (#225).
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
- The directory uses mimalloc instead of glibc's allocator, so it gives back memory it has freed instead of holding over a gigabyte while idle (#224).
|
- The directory uses mimalloc instead of glibc's allocator, so it gives back memory it has freed instead of holding over a gigabyte while idle (#224).
|
||||||
|
|||||||
@@ -73,7 +73,24 @@ pub static FETCHED_BYTES: std::sync::atomic::AtomicU64 = std::sync::atomic::Atom
|
|||||||
/// than be redirected on every read. A temporary redirect (302, 307) moves nothing.
|
/// than be redirected on every read. A temporary redirect (302, 307) moves nothing.
|
||||||
/// `system-proxy` reads http_proxy/https_proxy.
|
/// `system-proxy` reads http_proxy/https_proxy.
|
||||||
/// `auth` is a username and password, sent only to the feed's own host.
|
/// `auth` is a username and password, sent only to the feed's own host.
|
||||||
#[tracing::instrument(skip_all, fields(url = %feed_url, http.response.body.size))]
|
/// An address as a trace may keep it: without its query or user and password, where a private
|
||||||
|
/// feed carries its credential (Patreon's `?auth=`), which Tempo kept for anyone reading it (#225).
|
||||||
|
/// shortcut: a token in the path (`/feed/<token>.xml`) stays; strip paths too if a feed does that.
|
||||||
|
pub fn redact(url: &str) -> String {
|
||||||
|
match url::Url::parse(url) {
|
||||||
|
Ok(mut u) => {
|
||||||
|
u.set_query(None);
|
||||||
|
u.set_fragment(None);
|
||||||
|
let _ = u.set_username("");
|
||||||
|
let _ = u.set_password(None);
|
||||||
|
u.to_string()
|
||||||
|
}
|
||||||
|
// Not an address ipx could fetch; keep what comes before anything that could be a query.
|
||||||
|
Err(_) => url.split(['?', '#', '@']).next().unwrap_or_default().to_owned(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tracing::instrument(skip_all, fields(url = %redact(feed_url), http.response.body.size))]
|
||||||
pub async fn fetch(
|
pub async fn fetch(
|
||||||
client: &reqwest::Client,
|
client: &reqwest::Client,
|
||||||
feed_url: &str,
|
feed_url: &str,
|
||||||
@@ -1173,6 +1190,14 @@ const PAID_HOSTS: &[&str] =
|
|||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
|
#[test]
|
||||||
|
fn redact_drops_what_carries_a_credential() {
|
||||||
|
assert_eq!(super::redact("https://www.patreon.com/rss/x?auth=SECRET&show=1"), "https://www.patreon.com/rss/x");
|
||||||
|
assert_eq!(super::redact("https://me:pw@example.com/feed.xml#top"), "https://example.com/feed.xml");
|
||||||
|
assert_eq!(super::redact("https://example.com/feed.xml"), "https://example.com/feed.xml");
|
||||||
|
assert_eq!(super::redact("not a url?token=SECRET"), "not a url");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn an_address_carrying_someones_access_looks_private() {
|
fn an_address_carrying_someones_access_looks_private() {
|
||||||
let private = super::url_looks_private;
|
let private = super::url_looks_private;
|
||||||
|
|||||||
@@ -16,6 +16,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
- `/metrics` counts the feeds fetched and the bytes of their bodies (`ipx_feed_fetches_total`, `ipx_feed_bytes_total`), and a feed's trace says how large its body was (#224).
|
- `/metrics` counts the feeds fetched and the bytes of their bodies (`ipx_feed_fetches_total`, `ipx_feed_bytes_total`), and a feed's trace says how large its body was (#224).
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
- Traces no longer keep a feed's or file's address with its query or password, where private feeds carry their credentials, such as Patreon's `auth` (#225).
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
- ipx uses mimalloc instead of glibc's allocator, so memory a night of scans needed goes back to the system instead of staying held until a restart (#224).
|
- ipx uses mimalloc instead of glibc's allocator, so memory a night of scans needed goes back to the system instead of staying held until a restart (#224).
|
||||||
|
|||||||
@@ -2139,7 +2139,7 @@ fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Pol
|
|||||||
|
|
||||||
/// Downloads one file and records it. None when it turned out to be a torrent, handed off to run
|
/// Downloads one file and records it. None when it turned out to be a torrent, handed off to run
|
||||||
/// on its own, which records it when it finishes.
|
/// on its own, which records it when it finishes.
|
||||||
#[tracing::instrument(name = "download", skip_all, fields(feed = feed_id, enclosure = enclosure, url = url))]
|
#[tracing::instrument(name = "download", skip_all, fields(feed = feed_id, enclosure = enclosure, url = %feed::redact(url)))]
|
||||||
async fn fetch_one(
|
async fn fetch_one(
|
||||||
ctx: &Arc<Ctx>,
|
ctx: &Arc<Ctx>,
|
||||||
feed_id: &str,
|
feed_id: &str,
|
||||||
|
|||||||
Reference in New Issue
Block a user