diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ffaf73..2f56f6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Add a feed no longer asks for a folder; a feed's folder is its title (#201). - `docker-compose.yml` is a sample to start from, ipx alone on SQLite with its folders beside the file, instead of a copy of the author's server (#208). +- The source is a Cargo workspace: the daemon in `crates/ipx`, feed fetching and parsing in `crates/feed`. `cargo build --release` still makes `target/release/ipx` (#202). ## [0.12.0] - 2026-10-11 diff --git a/CLAUDE.md b/CLAUDE.md index 896925c..3fa868e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -89,7 +89,7 @@ running the command and kills the session (exit 144), which has happened more th There are three pages: the app (`web/index.html`), the admin page (`web/admin.html`, sent to admins only) and sign-in (`web/login.html`). The app and admin pages share one stylesheet, `web/app.css`, and their script is TypeScript in `web/src/`; `web/build.mjs` lists which files -make up each page's script. `build.rs` runs +make up each page's script. `crates/ipx/build.rs` runs `web/build.mjs`, which uses swc to strip the types and minify the script into `app.js` (and `login.js`), and minifies the page, and the results are `include_str!`d into the binary. The page loads its script as `/app.js?v=`, and `/app.css` the same way: the page is @@ -205,7 +205,7 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic * `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a different task. `ipx status` checks the control socket and the database; to see the worker getting through its jobs, watch for `scan complete` in the log. -* **The database goes through SeaORM, and the entities in `src/entity.rs` are the schema.** +* **The database goes through SeaORM, and the entities in `crates/ipx/src/entity.rs` are the schema.** `Db::open` creates any missing table or index from them (`create_missing`), on every `ipx` command, the healthcheck's `ipx status` included, so it must never write when nothing is missing: SeaORM's experimental schema sync dropped and remade an index on every open, the @@ -236,7 +236,7 @@ a feature added and then reworked in the same release is one line saying what it Added; something added and taken out again before the release, or a fix for a bug no release ever had, is not there at all; and "instead of" and "no longer" compare with the last release, not with last week. Issue numbers stay, all of them, on the line that survives. Then rename `[Unreleased]` to -`## [X.Y.Z] - YYYY-MM-DD` and open a new empty `[Unreleased]` above it, set `version` in `Cargo.toml` to `X.Y.Z` (dropping `-dev`), tag the commit +`## [X.Y.Z] - YYYY-MM-DD` and open a new empty `[Unreleased]` above it, set `version` in the root `Cargo.toml` (`[workspace.package]`) to `X.Y.Z` (dropping `-dev`), tag the commit `vX.Y.Z`, and update the compare links at the bottom of the changelog. Then, in the next commit, set `version` to the next patch with `-dev` (after 0.9.0, `0.9.1-dev`), so a build between releases says so in the logo's tooltip instead of claiming to be the last release. The release that follows can diff --git a/Cargo.lock b/Cargo.lock index 48a584f..3854105 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1830,6 +1830,7 @@ dependencies = [ "chrono", "clap", "futures-util", + "ipx-feed", "jsonwebtoken", "librqbit", "opentelemetry", @@ -1854,6 +1855,23 @@ dependencies = [ "url", ] +[[package]] +name = "ipx-feed" +version = "0.12.1-dev" +dependencies = [ + "anyhow", + "atom_syndication", + "chrono", + "opml", + "quick-xml 0.42.0", + "reqwest", + "rss", + "serde_json", + "tokio", + "tracing", + "url", +] + [[package]] name = "is_terminal_polyfill" version = "1.70.2" diff --git a/Cargo.toml b/Cargo.toml index 95bcb02..661d932 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,9 +1,14 @@ -[package] -name = "ipx" +[workspace] +members = ["crates/ipx", "crates/feed"] +resolver = "3" + +# One version for the whole workspace: what a release sets, and what the logo's tooltip shows. +[workspace.package] version = "0.12.1-dev" edition = "2024" -[dependencies] +[workspace.dependencies] +ipx-feed = { path = "crates/feed" } ammonia = "4.1.4" anyhow = "1.0.104" argon2 = "0.6.0" diff --git a/Dockerfile b/Dockerfile index 487a6af..8aca069 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,17 +11,20 @@ WORKDIR /src COPY package.json package-lock.json ./ RUN npm ci --omit=dev -# Dependencies first, so editing the source does not rebuild librqbit every time. +# Dependencies first, so editing the source does not rebuild librqbit every time: each crate of +# the workspace as an empty stub, its manifest real. COPY Cargo.toml Cargo.lock ./ -RUN mkdir src && echo 'fn main(){}' > src/main.rs \ +COPY crates/ipx/Cargo.toml crates/ipx/ +COPY crates/feed/Cargo.toml crates/feed/ +RUN mkdir -p crates/ipx/src crates/feed/src \ + && echo 'fn main(){}' > crates/ipx/src/main.rs && touch crates/feed/src/lib.rs \ && cargo build --release --locked \ - && rm -rf src + && rm -rf crates/ipx/src crates/feed/src -COPY build.rs ./ -COPY src ./src +COPY crates ./crates COPY web ./web # cargo skips a rebuild if mtimes look untouched; make sure it does not. -RUN touch src/main.rs && cargo build --release --locked +RUN touch crates/ipx/src/main.rs crates/feed/src/lib.rs && cargo build --release --locked FROM debian:trixie-slim # ffmpeg is DeadAir's (#171): it finds the silences and copies the frames kept. Debian's adds diff --git a/crates/feed/Cargo.toml b/crates/feed/Cargo.toml new file mode 100644 index 0000000..506fb59 --- /dev/null +++ b/crates/feed/Cargo.toml @@ -0,0 +1,19 @@ +[package] +name = "ipx-feed" +version.workspace = true +edition.workspace = true + +[dependencies] +anyhow.workspace = true +atom_syndication.workspace = true +chrono.workspace = true +opml.workspace = true +quick-xml.workspace = true +reqwest.workspace = true +rss.workspace = true +serde_json.workspace = true +tracing.workspace = true +url.workspace = true + +[dev-dependencies] +tokio.workspace = true diff --git a/src/feed.rs b/crates/feed/src/lib.rs similarity index 96% rename from src/feed.rs rename to crates/feed/src/lib.rs index 5b14200..ad10540 100644 --- a/src/feed.rs +++ b/crates/feed/src/lib.rs @@ -1,11 +1,10 @@ -//! Feed fetching and parsing. Replaces FeedData.__getFeed / __getEntries. +//! Feed fetching and parsing, shared by ipx and the directory (#202). Replaces +//! FeedData.__getFeed / __getEntries. use anyhow::{Context, Result, anyhow}; use reqwest::StatusCode; use reqwest::header::{ETAG, IF_MODIFIED_SINCE, IF_NONE_MATCH, LAST_MODIFIED}; -use crate::config::Feed as FeedCfg; - #[derive(Debug, Default)] pub struct ParsedFeed { pub title: Option, @@ -67,14 +66,16 @@ pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30) /// (301 or 308): a publisher that moved its feed, which the catalogue should follow rather /// than be redirected on every read. A temporary redirect (302, 307) moves nothing. /// `system-proxy` reads http_proxy/https_proxy. -#[tracing::instrument(skip_all, fields(url = %cfg.url))] +/// `auth` is a username and password, sent only to the feed's own host. +#[tracing::instrument(skip_all, fields(url = %feed_url))] pub async fn fetch( client: &reqwest::Client, - cfg: &FeedCfg, + feed_url: &str, + auth: Option<(&str, Option)>, etag: Option<&str>, last_modified: Option<&str>, ) -> Result<(Fetched, Option)> { - let start = reqwest::Url::parse(&cfg.url).context("the feed's address")?; + let start = reqwest::Url::parse(feed_url).context("the feed's address")?; let mut url = start.clone(); let mut permanent = true; for _ in 0..10 { @@ -86,10 +87,10 @@ pub async fn fetch( req = req.header(IF_MODIFIED_SINCE, lm); } // The feed's own host only: a redirect elsewhere must not be handed the password. - if let Some(user) = &cfg.username + if let Some((user, password)) = &auth && url.host_str() == start.host_str() { - req = req.basic_auth(user, cfg.password()); + req = req.basic_auth(user, password.as_deref()); } let resp = req.send().await.context("connecting")?; @@ -231,10 +232,21 @@ pub fn parse_opml(bytes: &[u8]) -> Result> { let doc = opml::OPML::from_str(&text) .map_err(|e| anyhow!("that does not parse as OPML: {e}"))?; let mut out = vec![]; - crate::collect_outlines(&doc.body.outlines, &mut out); + collect_outlines(&doc.body.outlines, &mut out); Ok(out) } +/// OPML nests feeds inside folder outlines, so this walks the whole tree. +pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, String)>) { + for o in outlines { + if let Some(url) = &o.xml_url { + let title = o.title.clone().unwrap_or_else(|| o.text.clone()); + out.push((title, url.clone())); + } + collect_outlines(&o.outlines, out); + } +} + /// The of an OPML document. pub fn opml_title(bytes: &[u8]) -> Option<String> { let text = String::from_utf8_lossy(bytes); @@ -1121,6 +1133,34 @@ fn picked(answer: &serde_json::Value) -> Result<String> { .ok_or_else(|| anyhow!("no category in the answer: {answer}")) } +/// An address that carries a subscriber's own access: a paid-feed host, credentials in the URL, +/// or a key in the query. ipx leaves such a feed out of anything others see, and the directory +/// refuses one sent to it (#202). +/// +/// ponytail: a heuristic. A key hidden in the path of a host not in `PAID_HOSTS` gets +/// through; a per-feed private flag is the upgrade (#205). +pub fn url_looks_private(feed_url: &str) -> bool { + let Ok(u) = url::Url::parse(feed_url) else { return true }; + // Paid-feed services put the subscriber's key in the path as often as in the query, and a + // long path segment alone proves nothing (acast's public show ids look the same). So a + // feed from one of them is private whatever its URL looks like. Supercast is why this + // exists: `feeds.supercast.com/feeds/<key>` reached the directory before it did. + let host = u.host_str().unwrap_or(""); + PAID_HOSTS.iter().any(|h| host == *h || host.ends_with(&format!(".{h}"))) + || !u.username().is_empty() + || u.password().is_some() + || u.query_pairs().any(|(k, _)| { + let k = k.to_ascii_lowercase(); + ["auth", "token", "key", "secret", "pass", "sig", "session", "user", "uid"] + .iter() + .any(|w| k.contains(w)) + }) +} + +/// Services whose feeds are always one subscriber's own. +const PAID_HOSTS: &[&str] = + &["patreon.com", "supercast.com", "supportingcast.fm", "glow.fm", "memberful.com"]; + #[cfg(test)] mod tests { /// A server answering by path: /old moves for good to /new, /tmp for now, /chain for good @@ -1157,9 +1197,9 @@ mod tests { let base = redirecting_server().await; let client = reqwest::Client::builder().redirect(reqwest::redirect::Policy::none()).build().unwrap(); let get = |path: &str| { - let cfg: crate::config::Feed = serde_json::from_value(serde_json::json!({ "url": format!("{base}{path}") })).unwrap(); + let url = format!("{base}{path}"); let client = client.clone(); - async move { super::fetch(&client, &cfg, None, None).await } + async move { super::fetch(&client, &url, None, None, None).await } }; let (got, moved) = get("/old").await.unwrap(); assert!(matches!(got, super::Fetched::Body { .. })); @@ -1175,7 +1215,7 @@ mod tests { #[test] fn parses_rss_with_itunes_extensions() { - let bytes = include_bytes!("../tests/data/rss2.xml"); + let bytes = include_bytes!("../../../tests/data/rss2.xml"); let feed = parse(bytes).unwrap(); assert_eq!(feed.title.as_deref(), Some("Test Cast")); @@ -1381,7 +1421,7 @@ mod tests { #[test] fn parses_atom_enclosure_links() { - let bytes = include_bytes!("../tests/data/atom.xml"); + let bytes = include_bytes!("../../../tests/data/atom.xml"); let feed = parse(bytes).unwrap(); assert_eq!(feed.title.as_deref(), Some("Atom Cast")); @@ -1479,8 +1519,8 @@ mod tests { assert_eq!(feeds[0], ("Alpha".into(), "https://a.example/rss".into())); // A feed must never be mistaken for a subscription list. - assert!(!is_opml(include_bytes!("../tests/data/rss2.xml"))); - assert!(!is_opml(include_bytes!("../tests/data/atom.xml"))); + assert!(!is_opml(include_bytes!("../../../tests/data/rss2.xml"))); + assert!(!is_opml(include_bytes!("../../../tests/data/atom.xml"))); } #[test] diff --git a/crates/ipx/Cargo.toml b/crates/ipx/Cargo.toml new file mode 100644 index 0000000..81ae77b --- /dev/null +++ b/crates/ipx/Cargo.toml @@ -0,0 +1,37 @@ +[package] +name = "ipx" +version.workspace = true +edition.workspace = true + +[dependencies] +ipx-feed.workspace = true +ammonia.workspace = true +anyhow.workspace = true +argon2.workspace = true +atom_syndication.workspace = true +axum.workspace = true +chrono.workspace = true +clap.workspace = true +futures-util.workspace = true +jsonwebtoken.workspace = true +librqbit.workspace = true +opentelemetry.workspace = true +opentelemetry-otlp.workspace = true +opentelemetry_sdk.workspace = true +opml.workspace = true +percent-encoding.workspace = true +quick-xml.workspace = true +reqwest.workspace = true +rss.workspace = true +sea-orm.workspace = true +serde.workspace = true +serde_json.workspace = true +sha2.workspace = true +tokio.workspace = true +toml.workspace = true +tower.workspace = true +tower-http.workspace = true +tracing.workspace = true +tracing-opentelemetry.workspace = true +tracing-subscriber.workspace = true +url.workspace = true diff --git a/build.rs b/crates/ipx/build.rs similarity index 71% rename from build.rs rename to crates/ipx/build.rs index 488e4d0..e156a2e 100644 --- a/build.rs +++ b/crates/ipx/build.rs @@ -1,13 +1,13 @@ //! Builds web/index.html and web/login.html from their TypeScript (web/build.mjs) into OUT_DIR, -//! where src/web.rs include_str!s them. Needs node and `npm ci` run first. +//! where src/web.rs include_str!s them. Needs node and `npm ci` run first, at the repo's root. use std::process::Command; fn main() { - println!("cargo:rerun-if-changed=web"); - println!("cargo:rerun-if-changed=package-lock.json"); + println!("cargo:rerun-if-changed=../../web"); + println!("cargo:rerun-if-changed=../../package-lock.json"); let out = std::env::var("OUT_DIR").unwrap(); let status = Command::new("node") - .args(["web/build.mjs", &out]) + .args(["../../web/build.mjs", &out]) .status() .expect("building the web pages needs node on PATH (and `npm ci` run once)"); assert!(status.success(), "web/build.mjs failed; run `node web/build.mjs` to see why"); diff --git a/src/access.rs b/crates/ipx/src/access.rs similarity index 97% rename from src/access.rs rename to crates/ipx/src/access.rs index 2a4c9a1..a3f5bad 100644 --- a/src/access.rs +++ b/crates/ipx/src/access.rs @@ -137,7 +137,7 @@ mod tests { const AUD: &str = "aud-tag"; fn jwks() -> JwkSet { - serde_json::from_str(include_str!("../tests/data/access-test.jwks.json")).unwrap() + serde_json::from_str(include_str!("../../../tests/data/access-test.jwks.json")).unwrap() } fn token(key: &[u8], alg: Algorithm, claims: serde_json::Value) -> String { @@ -155,8 +155,8 @@ mod tests { }) } - const SIGNER: &[u8] = include_bytes!("../tests/data/access-test.der"); - const FORGER: &[u8] = include_bytes!("../tests/data/access-forger.der"); + const SIGNER: &[u8] = include_bytes!("../../../tests/data/access-test.der"); + const FORGER: &[u8] = include_bytes!("../../../tests/data/access-forger.der"); async fn check(keys: &Keys, t: &str) -> Option<String> { keys.verify_with(TEAM, AUD, t, || async { Ok(jwks()) }).await diff --git a/src/art.rs b/crates/ipx/src/art.rs similarity index 100% rename from src/art.rs rename to crates/ipx/src/art.rs diff --git a/src/auth.rs b/crates/ipx/src/auth.rs similarity index 100% rename from src/auth.rs rename to crates/ipx/src/auth.rs diff --git a/src/config.rs b/crates/ipx/src/config.rs similarity index 99% rename from src/config.rs rename to crates/ipx/src/config.rs index 01e83b1..c72fdac 100644 --- a/src/config.rs +++ b/crates/ipx/src/config.rs @@ -273,6 +273,11 @@ impl Feed { } self.password.clone() } + + /// What `feed::fetch` sends the feed's host: a username, and its password if there is one. + pub fn auth(&self) -> Option<(&str, Option<String>)> { + self.username.as_deref().map(|u| (u, self.password())) + } } impl Config { diff --git a/src/db.rs b/crates/ipx/src/db.rs similarity index 100% rename from src/db.rs rename to crates/ipx/src/db.rs diff --git a/src/deadair.rs b/crates/ipx/src/deadair.rs similarity index 100% rename from src/deadair.rs rename to crates/ipx/src/deadair.rs diff --git a/src/download.rs b/crates/ipx/src/download.rs similarity index 100% rename from src/download.rs rename to crates/ipx/src/download.rs diff --git a/src/entity.rs b/crates/ipx/src/entity.rs similarity index 100% rename from src/entity.rs rename to crates/ipx/src/entity.rs diff --git a/src/ipc.rs b/crates/ipx/src/ipc.rs similarity index 100% rename from src/ipc.rs rename to crates/ipx/src/ipc.rs diff --git a/src/logbuf.rs b/crates/ipx/src/logbuf.rs similarity index 100% rename from src/logbuf.rs rename to crates/ipx/src/logbuf.rs diff --git a/src/main.rs b/crates/ipx/src/main.rs similarity index 99% rename from src/main.rs rename to crates/ipx/src/main.rs index 65bdd39..bb5299c 100644 --- a/src/main.rs +++ b/crates/ipx/src/main.rs @@ -6,7 +6,8 @@ mod db; mod deadair; mod entity; mod download; -mod feed; +// The crate root's import, so `crate::feed` reaches it from every module as before. +use ipx_feed as feed; mod ipc; mod logbuf; mod metrics; @@ -908,7 +909,7 @@ async fn add_one( listed: false, }; - let title = match feed::fetch(&ctx.feed_client, &probe, None, None).await.map(|(got, _)| got) { + let title = match feed::fetch(&ctx.feed_client, &probe.url, None, None, None).await.map(|(got, _)| got) { // An OPML subscription is named from its own <head><title>, not by trying to // parse it as a feed and falling back to the hostname. Ok(feed::Fetched::Body { bytes, .. }) if feed::is_opml(&bytes) => { @@ -999,7 +1000,7 @@ pub async fn subscribe_opml( user_id: i64, ) -> Result<(usize, usize)> { let mut found = vec![]; - collect_outlines(&doc.body.outlines, &mut found); + feed::collect_outlines(&doc.body.outlines, &mut found); let mut cfg = ctx.edit_cfg().await; let known = subscriptions(ctx).await?; @@ -1057,17 +1058,6 @@ pub async fn subscribe_opml( Ok((added, had)) } -/// OPML nests feeds inside folder outlines, so this walks the whole tree. -pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, String)>) { - for o in outlines { - if let Some(url) = &o.xml_url { - let title = o.title.clone().unwrap_or_else(|| o.text.clone()); - out.push((title, url.clone())); - } - collect_outlines(&o.outlines, out); - } -} - /// The configuration ipx runs with: config.toml for where things are and who may sign in, the /// database for the feeds and the server settings (issue #18). The first time a database holds /// neither, it takes them from config.toml, which is then cut down to the rest, the original kept @@ -1264,7 +1254,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String] return None; } let _fetching = metrics::busy(&metrics::STATS.feeds_fetching); - Some(feed::fetch(&client, &feed_cfg, etag.as_deref(), modified.as_deref()).await) + Some(feed::fetch(&client, &feed_cfg.url, feed_cfg.auth(), etag.as_deref(), modified.as_deref()).await) }, tracing::Span::current(), )) @@ -1654,7 +1644,7 @@ async fn scan_one( let (mut fetched, moved) = match prefetched { Some(got) => got?, - None => feed::fetch(&ctx.feed_client, feed_cfg, state.etag.as_deref(), state.last_modified.as_deref()).await?, + None => feed::fetch(&ctx.feed_client, &feed_cfg.url, feed_cfg.auth(), state.etag.as_deref(), state.last_modified.as_deref()).await?, }; if let Some(to) = moved { follow_move(ctx, id, &to).await?; @@ -1669,7 +1659,7 @@ async fn scan_one( if matches!(fetched, feed::Fetched::NotModified) && (stored.entries == 0 || stored.image.is_none()) { tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator"); ctx.db.clear_validators(id).await?; - fetched = feed::fetch(&ctx.feed_client, feed_cfg, None, None).await?.0; + fetched = feed::fetch(&ctx.feed_client, &feed_cfg.url, feed_cfg.auth(), None, None).await?.0; } let (bytes, etag, last_modified) = match fetched { diff --git a/src/metrics.rs b/crates/ipx/src/metrics.rs similarity index 100% rename from src/metrics.rs rename to crates/ipx/src/metrics.rs diff --git a/src/retention.rs b/crates/ipx/src/retention.rs similarity index 100% rename from src/retention.rs rename to crates/ipx/src/retention.rs diff --git a/src/torrent.rs b/crates/ipx/src/torrent.rs similarity index 100% rename from src/torrent.rs rename to crates/ipx/src/torrent.rs diff --git a/src/web.rs b/crates/ipx/src/web.rs similarity index 98% rename from src/web.rs rename to crates/ipx/src/web.rs index ef269ee..65894f5 100644 --- a/src/web.rs +++ b/crates/ipx/src/web.rs @@ -663,7 +663,7 @@ fn script(js: &'static str) -> impl IntoResponse { async fn logo() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "image/svg+xml"), (header::CACHE_CONTROL, "max-age=86400")], - include_str!("../web/logo.svg"), + include_str!("../../../web/logo.svg"), ) } @@ -672,7 +672,7 @@ async fn logo() -> impl IntoResponse { async fn logo_dark() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "image/svg+xml"), (header::CACHE_CONTROL, "max-age=86400")], - include_str!("../web/logo-dark.svg"), + include_str!("../../../web/logo-dark.svg"), ) } @@ -680,7 +680,7 @@ async fn logo_dark() -> impl IntoResponse { async fn icon() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")], - include_bytes!("../web/ipodderx-icon.png").as_slice(), + include_bytes!("../../../web/ipodderx-icon.png").as_slice(), ) } @@ -689,7 +689,7 @@ async fn icon() -> impl IntoResponse { async fn favicon() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")], - include_bytes!("../web/favicon.png").as_slice(), + include_bytes!("../../../web/favicon.png").as_slice(), ) } @@ -697,7 +697,7 @@ async fn favicon() -> impl IntoResponse { async fn favicon_dark() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")], - include_bytes!("../web/favicon-dark.png").as_slice(), + include_bytes!("../../../web/favicon-dark.png").as_slice(), ) } @@ -706,7 +706,7 @@ async fn favicon_dark() -> impl IntoResponse { async fn touch_icon() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")], - include_bytes!("../web/apple-touch-icon.png").as_slice(), + include_bytes!("../../../web/apple-touch-icon.png").as_slice(), ) } @@ -716,7 +716,7 @@ async fn touch_icon() -> impl IntoResponse { async fn inter() -> impl IntoResponse { ( [(header::CONTENT_TYPE, "font/woff2"), (header::CACHE_CONTROL, "max-age=604800")], - include_bytes!("../web/InterVariable.woff2").as_slice(), + include_bytes!("../../../web/InterVariable.woff2").as_slice(), ) } @@ -936,35 +936,15 @@ async fn feed_rows(state: &WebState, user: &crate::db::User, only: Option<&str>) // ---- popular on this server ---- /// A feed that carries a credential is someone's paid or private subscription. Listing it -/// would let anyone signed in subscribe to it and read what they pay for. -/// -/// ponytail: a heuristic. A key hidden in the path of a host not in `PAID_HOSTS` gets -/// through; a per-feed `unlisted` flag is the upgrade if that happens again. +/// would let anyone signed in subscribe to it and read what they pay for. The address half is +/// `feed::url_looks_private`, which the directory shares. fn looks_private(feed: &crate::config::Feed) -> bool { - if feed.username.is_some() || feed.password.is_some() || feed.password_env.is_some() { - return true; - } - let Ok(u) = url::Url::parse(&feed.url) else { return true }; - // Paid-feed services put the subscriber's key in the path as often as in the query, and a - // long path segment alone proves nothing (acast's public show ids look the same). So a - // feed from one of them is private whatever its URL looks like. Supercast is why this - // exists: `feeds.supercast.com/feeds/<key>` reached the directory before it did. - let host = u.host_str().unwrap_or(""); - PAID_HOSTS.iter().any(|h| host == *h || host.ends_with(&format!(".{h}"))) - || !u.username().is_empty() - || u.password().is_some() - || u.query_pairs().any(|(k, _)| { - let k = k.to_ascii_lowercase(); - ["auth", "token", "key", "secret", "pass", "sig", "session", "user", "uid"] - .iter() - .any(|w| k.contains(w)) - }) + feed.username.is_some() + || feed.password.is_some() + || feed.password_env.is_some() + || crate::feed::url_looks_private(&feed.url) } -/// Services whose feeds are always one subscriber's own. -const PAID_HOSTS: &[&str] = - &["patreon.com", "supercast.com", "supportingcast.fm", "glow.fm", "memberful.com"]; - /// Only an id, a title, artwork and a count: never a URL, which is where a key would be. #[derive(Serialize)] struct PopularRow { diff --git a/docs/architecture.md b/docs/architecture.md index 2924a00..dc6969b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,28 +4,32 @@ One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a control server, and the web UI. Everything else is a CLI that either does the work itself or hands it to a running daemon. +The repo is a Cargo workspace (#202): `crates/ipx` is the daemon and CLI, and `crates/feed` +(`ipx-feed`) fetches and parses feeds, kept apart because the directory service (#204) shares it. +The version is the workspace's, in the root `Cargo.toml`. + ## Modules | File | Responsibility | What it replaced in the Python | |---|---|---| -| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` | -| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` | -| `src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` | -| `src/entity.rs` | The tables, as SeaORM entities: the schema | — | -| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` | -| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` | -| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 | -| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` | -| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout | -| `src/auth.rs` | Argon2id hashing, session tokens, header names | — | -| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — | -| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — | +| `crates/ipx/src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` | +| `crates/ipx/src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` | +| `crates/ipx/src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` | +| `crates/ipx/src/entity.rs` | The tables, as SeaORM entities: the schema | — | +| `crates/feed/src/lib.rs` | Conditional GET, RSS/Atom/OPML parsing, `url_looks_private` | `FeedData.__getFeed/__getEntries` | +| `crates/ipx/src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` | +| `crates/ipx/src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 | +| `crates/ipx/src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` | +| `crates/ipx/src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout | +| `crates/ipx/src/auth.rs` | Argon2id hashing, session tokens, header names | — | +| `crates/ipx/src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — | +| `crates/ipx/src/logbuf.rs` | Ring buffer behind the UI's Log view | — | | `web/index.html` | The app's markup | — | | `web/admin.html` | The admin page's markup: server settings, accounts, the log. Sent to admins only | — | | `web/app.css` | The stylesheet both pages share | — | | `web/src/*.ts` | The page's script, one scope split across files, type-checked by `npx tsc` | — | | `web/build.mjs` | swc: strips the types into `app.js`/`login.js`, named in the page by a hash of their contents, and minifies | — | -| `build.rs` | Runs `web/build.mjs` into `OUT_DIR`, where `web.rs` `include_str!`s the result | — | +| `crates/ipx/build.rs` | Runs `web/build.mjs` into `OUT_DIR`, where `web.rs` `include_str!`s the result | — | The page is compiled in, so **editing `web/index.html` or `web/src` needs a rebuild**, and a build needs node and `npm ci` run once. @@ -70,7 +74,7 @@ entry_state user_id, feed_id, guid, read, flagged, position Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from before accounts; two bugs came from queries still reading them, and they were dropped in 0.5. -Schema changes: the tables are the entities in `src/entity.rs`, and `Db::open` creates whatever +Schema changes: the tables are the entities in `crates/ipx/src/entity.rs`, and `Db::open` creates whatever table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It never alters a table that exists, so a new column on one needs its own `ALTER` in `create_missing`, looked for first and run only when it is missing, as `feeds.description`'s is,