rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Fetches refuse internal addresses unless fetch_internal lists them (#320)

Browse Source 
ipx's clients had no destination policy and followed redirects
anywhere. Adding a feed returned the status or first line of whatever
answered at the address; an enclosure declared as audio was downloaded
whole from any URL and served back through /media; /api/art fetched
any image a feed named. Feeds come from publishers and anyone signed
in, so loopback, the Docker network and the LAN were all reachable.

ipx-feed's guard is a connector layer: after the name resolves and the
connection opens, before the request is written, it looks at the
peer's address and refuses anything off the public internet unless
config.toml's [general] fetch_internal lists it. Being on the
connection, it covers every request on ctx.client and feed_client,
each redirect hop, IP literals (which never reach a DNS resolver) and
a name that resolves differently the second time. reqwest's rustls
connection reports the peer address; a test run by hand checks HTTPS
gets through, since were it lost every https fetch would be refused.

The directory is on the server's own network by design
(http://ipx-directory:8100), so it moves to local_client, unguarded,
set by an admin. The browser suite and load tests serve their feeds on
loopback and list 127.0.0.1.

Not covered: librqbit's own connections (trackers, peers), which do
not go through these clients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-23 13:27:40 +00:00
parent 6d71613e19
commit a20062d0f7
12 changed files with 311 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
crates/feed/Cargo.toml
View File

@@ -8,12 +8,15 @@ edition.workspace = true
anyhow.workspace = true
atom_syndication.workspace = true
chrono.workspace = true
http.workspace = true
hyper-util.workspace = true
opml.workspace = true
pulldown-cmark.workspace = true
quick-xml.workspace = true
reqwest.workspace = true
rss.workspace = true
serde_json.workspace = true
tower.workspace = true
tracing.workspace = true
url.workspace = true

269
crates/feed/src/guard.rs Normal file
View File

@@ -0,0 +1,269 @@
//! Keeps fetches off internal addresses (#320). A feed, an episode, its artwork and a .torrent are
//! all at addresses a publisher or any signed-in person names, and redirects go anywhere, so
//! without this the server fetched from its own loopback, its Docker network and the LAN, and
//! handed back what answered.
//!
//! The check is on the connection, after the name is resolved and before a byte of the request
//! is sent, so it covers every request, every redirect, an address written as an IP, and a name
//! that resolves to an internal address only on the second lookup.
use std::future::Future;
use std::net::{IpAddr, Ipv4Addr};
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
use anyhow::{Context as _, Result};
use hyper_util::client::legacy::connect::{Connection, HttpInfo};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
/// Internal addresses and networks a fetch may reach all the same: config.toml's `fetch_internal`,
/// as `192.168.1.20` or `192.168.1.0/24`. Names are not taken: the check sees only the address.
#[derive(Clone, Debug, Default)]
pub struct Allowed(Arc<Vec<(IpAddr, u8)>>);
impl Allowed {
pub fn parse(list: &[String]) -> Result<Self> {
list.iter().map(|s| parse_net(s)).collect::<Result<Vec<_>>>().map(|v| Self(Arc::new(v)))
}
pub fn permits(&self, ip: IpAddr) -> bool {
let ip = canonical(ip);
!is_internal(ip) || self.0.iter().any(|&(net, bits)| contains(net, bits, ip))
}
}
fn parse_net(s: &str) -> Result<(IpAddr, u8)> {
let bad = || format!("fetch_internal: {s:?} is not an address or a network such as 192.168.1.0/24");
let (addr, bits) = s.trim().split_once('/').map_or((s.trim(), None), |(a, b)| (a, Some(b)));
let ip = canonical(addr.parse::<IpAddr>().with_context(bad)?);
let max = if ip.is_ipv4() { 32 } else { 128 };
let bits = match bits {
Some(b) => b.parse::<u8>().ok().filter(|n| *n <= max).with_context(bad)?,
None => max,
};
Ok((ip, bits))
}
/// An IPv4 address written as IPv6 (`::ffff:127.0.0.1`) is the IPv4 one.
fn canonical(ip: IpAddr) -> IpAddr {
match ip {
IpAddr::V6(v6) => v6.to_ipv4_mapped().map_or(ip, IpAddr::V4),
v4 => v4,
}
}
fn contains(net: IpAddr, bits: u8, ip: IpAddr) -> bool {
match (net, ip) {
(IpAddr::V4(n), IpAddr::V4(i)) => {
let mask = u32::MAX.checked_shl(32 - u32::from(bits)).unwrap_or(0);
u32::from(n) & mask == u32::from(i) & mask
}
(IpAddr::V6(n), IpAddr::V6(i)) => {
let mask = u128::MAX.checked_shl(128 - u32::from(bits)).unwrap_or(0);
u128::from(n) & mask == u128::from(i) & mask
}
_ => false,
}
}
/// Not on the public internet: loopback, private, link-local, shared (CGNAT), multicast and
/// reserved ranges, and their IPv6 counterparts. `IpAddr::is_global` would say this, but is
/// unstable.
// shortcut: 6to4 (2002::/16) and Teredo addresses that wrap an internal IPv4 one pass; check the
// embedded address if an IPv6 route to them ever exists here.
pub fn is_internal(ip: IpAddr) -> bool {
match canonical(ip) {
IpAddr::V4(a) => {
let [o0, o1, o2, _] = a.octets();
a.is_private()
|| a.is_loopback()
|| a.is_link_local()
|| a.is_unspecified()
|| a.is_broadcast()
|| a.is_multicast()
|| o0 == 0
|| o0 >= 240
|| (o0 == 100 && o1 & 0xc0 == 64)
|| (o0 == 192 && o1 == 0 && o2 == 0)
}
IpAddr::V6(a) => {
let s = a.segments();
a.is_loopback()
|| a.is_unspecified()
|| a.is_multicast()
|| s[0] & 0xfe00 == 0xfc00
|| s[0] & 0xffc0 == 0xfe80
// NAT64 carries an IPv4 address in its last 32 bits.
|| (s[0] == 0x64 && s[1] == 0xff9b && is_internal(IpAddr::V4(Ipv4Addr::from(u128::from(a) as u32))))
}
}
}
/// A connector layer for `reqwest::ClientBuilder::connector_layer`: a connection to an address
/// `Allowed` does not permit fails before the request is sent.
#[derive(Clone)]
pub struct Guard(Allowed);
impl Guard {
pub fn new(allowed: Allowed) -> Self {
Self(allowed)
}
}
impl<S> tower::Layer<S> for Guard {
type Service = Guarded<S>;
fn layer(&self, inner: S) -> Guarded<S> {
Guarded { inner, allowed: self.0.clone() }
}
}
#[derive(Clone)]
pub struct Guarded<S> {
inner: S,
allowed: Allowed,
}
impl<S, R> tower::Service<R> for Guarded<S>
where
S: tower::Service<R>,
S::Response: Connection,
S::Error: Into<BoxError>,
S::Future: Send + 'static,
{
type Response = S::Response;
type Error = BoxError;
type Future = Pin<Box<dyn Future<Output = Result<S::Response, BoxError>> + Send>>;
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), BoxError>> {
self.inner.poll_ready(cx).map_err(Into::into)
}
fn call(&mut self, req: R) -> Self::Future {
let connecting = self.inner.call(req);
let allowed = self.allowed.clone();
Box::pin(async move {
let conn = connecting.await.map_err(Into::into)?;
let mut extras = http::Extensions::new();
conn.connected().get_extras(&mut extras);
// Refused when the peer is unknown too: a connection this cannot check is one it
// cannot vouch for.
match extras.get::<HttpInfo>().map(|i| i.remote_addr().ip()) {
Some(ip) if allowed.permits(ip) => Ok(conn),
Some(ip) => {
Err(format!("{ip} is an internal address, not fetched (fetch_internal in config.toml)").into())
}
None => Err("the connection's address is unknown, so it is not fetched".into()),
}
})
}
}
#[cfg(test)]
mod tests {
use super::*;
fn ip(s: &str) -> IpAddr {
s.parse().unwrap()
}
#[test]
fn internal_addresses_are_known() {
for a in [
"127.0.0.1",
"10.1.2.3",
"172.16.0.1",
"192.168.1.130",
"169.254.169.254",
"100.64.0.1",
"0.0.0.0",
"224.0.0.1",
"255.255.255.255",
"::1",
"::",
"fd00::1",
"fe80::1",
"::ffff:127.0.0.1",
"64:ff9b::a00:1",
] {
assert!(is_internal(ip(a)), "{a}");
}
for a in ["1.1.1.1", "8.8.8.8", "2606:4700:4700::1111", "64:ff9b::808:808", "100.128.0.1"] {
assert!(!is_internal(ip(a)), "{a}");
}
}
#[test]
fn the_allowlist_lets_through_what_it_names_and_no_more() {
let a = Allowed::parse(&["192.168.1.0/24".into(), "127.0.0.1".into(), "fd00::/8".into()]).unwrap();
assert!(a.permits(ip("192.168.1.20")));
assert!(a.permits(ip("127.0.0.1")));
assert!(a.permits(ip("::ffff:127.0.0.1")), "the same address written as IPv6");
assert!(a.permits(ip("fd12::1")));
assert!(!a.permits(ip("192.168.2.1")));
assert!(!a.permits(ip("127.0.0.2")));
assert!(a.permits(ip("1.1.1.1")), "public addresses need no listing");
assert!(Allowed::default().permits(ip("1.1.1.1")));
assert!(!Allowed::default().permits(ip("10.0.0.1")));
for bad in ["nas.lan", "10.0.0.0/33", "10.0.0.0/x", ""] {
assert!(Allowed::parse(&[bad.into()]).is_err(), "{bad}");
}
}
async fn answers() -> String {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
while let Ok((mut sock, _)) = listener.accept().await {
let _ = sock.read(&mut [0u8; 1024]).await;
let _ = sock.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 6\r\n\r\nsecret").await;
}
});
format!("http://{addr}/")
}
fn client(allowed: &[&str]) -> reqwest::Client {
let allowed = Allowed::parse(&allowed.iter().map(|s| s.to_string()).collect::<Vec<_>>()).unwrap();
reqwest::Client::builder().connector_layer(Guard::new(allowed)).build().unwrap()
}
#[tokio::test]
async fn a_fetch_from_loopback_is_refused_unless_listed() {
let url = answers().await;
let err = client(&[]).get(&url).send().await.unwrap_err();
assert!(format!("{err:?}").contains("internal address"), "{err:?}");
let body = client(&["127.0.0.1"]).get(&url).send().await.unwrap().text().await.unwrap();
assert_eq!(body, "secret");
}
#[tokio::test]
async fn a_redirect_to_loopback_is_refused() {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
let inner = answers().await;
let listener = tokio::net::TcpListener::bind("127.0.0.2:0").await.unwrap();
let addr = listener.local_addr().unwrap();
tokio::spawn(async move {
while let Ok((mut sock, _)) = listener.accept().await {
let _ = sock.read(&mut [0u8; 1024]).await;
let r = format!("HTTP/1.1 302 Found\r\nLocation: {inner}\r\nContent-Length: 0\r\n\r\n");
let _ = sock.write_all(r.as_bytes()).await;
}
});
// 127.0.0.2 stands for a public address: listed, it is let through, and the redirect it
// sends to 127.0.0.1 is not.
let err = client(&["127.0.0.2"]).get(format!("http://{addr}/")).send().await.unwrap_err();
assert!(format!("{err:?}").contains("127.0.0.1 is an internal address"), "{err:?}");
}
/// Run by hand: `cargo test -p ipx-feed guard -- --ignored`. HTTPS gives the peer's address
/// too; were it lost under TLS, every https fetch would be refused.
#[tokio::test]
#[ignore = "reaches the internet"]
async fn https_is_checked_and_let_through() {
let r = client(&[]).get("https://example.com/").send().await.unwrap();
assert!(r.status().is_success());
}
}

2
crates/feed/src/lib.rs
View File

@@ -1,6 +1,8 @@
//! Feed fetching and parsing, shared by ipx and the directory (#202). Replaces
//! FeedData.__getFeed / __getEntries.
pub mod guard;
use anyhow::{Context, Result, anyhow};
use reqwest::StatusCode;
use reqwest::header::{ETAG, IF_MODIFIED_SINCE, IF_NONE_MATCH, LAST_MODIFIED};