Cargo workspace: crates/ipx and a shared crates/feed (#202)
The first step of the split (#207). The daemon moves to crates/ipx unchanged; feed.rs becomes crates/feed (ipx-feed), which the directory service (#204) will share for its metadata refresher and its privacy check. To stand alone, fetch takes the feed's address and its credentials instead of ipx's config::Feed (config::Feed::auth gives them), collect_outlines moves in with the OPML parsing, and the address half of looks_private is feed::url_looks_private. ipx imports the crate as `feed` at its root, so every crate::feed path is unchanged. The version is the workspace's, in the root Cargo.toml. Checked: cargo test on Postgres (118 + 29), tsc, page smoke, native bridge, contrast and Playwright (72) pass; the image built from the new Dockerfile is deployed and healthy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
- Add a feed no longer asks for a folder; a feed's folder is its title (#201).
|
||||
- `docker-compose.yml` is a sample to start from, ipx alone on SQLite with its folders beside the file, instead of a copy of the author's server (#208).
|
||||
- The source is a Cargo workspace: the daemon in `crates/ipx`, feed fetching and parsing in `crates/feed`. `cargo build --release` still makes `target/release/ipx` (#202).
|
||||
|
||||
## [0.12.0] - 2026-10-11
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ running the command and kills the session (exit 144), which has happened more th
|
||||
There are three pages: the app (`web/index.html`), the admin page (`web/admin.html`, sent to
|
||||
admins only) and sign-in (`web/login.html`). The app and admin pages share one stylesheet,
|
||||
`web/app.css`, and their script is TypeScript in `web/src/`; `web/build.mjs` lists which files
|
||||
make up each page's script. `build.rs` runs
|
||||
make up each page's script. `crates/ipx/build.rs` runs
|
||||
`web/build.mjs`, which uses swc to strip the types and minify the script into `app.js` (and
|
||||
`login.js`), and minifies the page, and the results are `include_str!`d into the binary. The page
|
||||
loads its script as `/app.js?v=<hash of its contents>`, and `/app.css` the same way: the page is
|
||||
@@ -205,7 +205,7 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
||||
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
|
||||
different task. `ipx status` checks the control socket and the database; to see the worker
|
||||
getting through its jobs, watch for `scan complete` in the log.
|
||||
* **The database goes through SeaORM, and the entities in `src/entity.rs` are the schema.**
|
||||
* **The database goes through SeaORM, and the entities in `crates/ipx/src/entity.rs` are the schema.**
|
||||
`Db::open` creates any missing table or index from them (`create_missing`), on every `ipx`
|
||||
command, the healthcheck's `ipx status` included, so it must never write when nothing is
|
||||
missing: SeaORM's experimental schema sync dropped and remade an index on every open, the
|
||||
@@ -236,7 +236,7 @@ a feature added and then reworked in the same release is one line saying what it
|
||||
Added; something added and taken out again before the release, or a fix for a bug no release ever
|
||||
had, is not there at all; and "instead of" and "no longer" compare with the last release, not with
|
||||
last week. Issue numbers stay, all of them, on the line that survives. Then rename `[Unreleased]` to
|
||||
`## [X.Y.Z] - YYYY-MM-DD` and open a new empty `[Unreleased]` above it, set `version` in `Cargo.toml` to `X.Y.Z` (dropping `-dev`), tag the commit
|
||||
`## [X.Y.Z] - YYYY-MM-DD` and open a new empty `[Unreleased]` above it, set `version` in the root `Cargo.toml` (`[workspace.package]`) to `X.Y.Z` (dropping `-dev`), tag the commit
|
||||
`vX.Y.Z`, and update the compare links at the bottom of the changelog. Then, in the next commit,
|
||||
set `version` to the next patch with `-dev` (after 0.9.0, `0.9.1-dev`), so a build between releases says so
|
||||
in the logo's tooltip instead of claiming to be the last release. The release that follows can
|
||||
|
||||
18
Cargo.lock
generated
18
Cargo.lock
generated
@@ -1830,6 +1830,7 @@ dependencies = [
|
||||
"chrono",
|
||||
"clap",
|
||||
"futures-util",
|
||||
"ipx-feed",
|
||||
"jsonwebtoken",
|
||||
"librqbit",
|
||||
"opentelemetry",
|
||||
@@ -1854,6 +1855,23 @@ dependencies = [
|
||||
"url",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ipx-feed"
|
||||
version = "0.12.1-dev"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"atom_syndication",
|
||||
"chrono",
|
||||
"opml",
|
||||
"quick-xml 0.42.0",
|
||||
"reqwest",
|
||||
"rss",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"url",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "is_terminal_polyfill"
|
||||
version = "1.70.2"
|
||||
|
||||
11
Cargo.toml
11
Cargo.toml
@@ -1,9 +1,14 @@
|
||||
[package]
|
||||
name = "ipx"
|
||||
[workspace]
|
||||
members = ["crates/ipx", "crates/feed"]
|
||||
resolver = "3"
|
||||
|
||||
# One version for the whole workspace: what a release sets, and what the logo's tooltip shows.
|
||||
[workspace.package]
|
||||
version = "0.12.1-dev"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
[workspace.dependencies]
|
||||
ipx-feed = { path = "crates/feed" }
|
||||
ammonia = "4.1.4"
|
||||
anyhow = "1.0.104"
|
||||
argon2 = "0.6.0"
|
||||
|
||||
15
Dockerfile
15
Dockerfile
@@ -11,17 +11,20 @@ WORKDIR /src
|
||||
COPY package.json package-lock.json ./
|
||||
RUN npm ci --omit=dev
|
||||
|
||||
# Dependencies first, so editing the source does not rebuild librqbit every time.
|
||||
# Dependencies first, so editing the source does not rebuild librqbit every time: each crate of
|
||||
# the workspace as an empty stub, its manifest real.
|
||||
COPY Cargo.toml Cargo.lock ./
|
||||
RUN mkdir src && echo 'fn main(){}' > src/main.rs \
|
||||
COPY crates/ipx/Cargo.toml crates/ipx/
|
||||
COPY crates/feed/Cargo.toml crates/feed/
|
||||
RUN mkdir -p crates/ipx/src crates/feed/src \
|
||||
&& echo 'fn main(){}' > crates/ipx/src/main.rs && touch crates/feed/src/lib.rs \
|
||||
&& cargo build --release --locked \
|
||||
&& rm -rf src
|
||||
&& rm -rf crates/ipx/src crates/feed/src
|
||||
|
||||
COPY build.rs ./
|
||||
COPY src ./src
|
||||
COPY crates ./crates
|
||||
COPY web ./web
|
||||
# cargo skips a rebuild if mtimes look untouched; make sure it does not.
|
||||
RUN touch src/main.rs && cargo build --release --locked
|
||||
RUN touch crates/ipx/src/main.rs crates/feed/src/lib.rs && cargo build --release --locked
|
||||
|
||||
FROM debian:trixie-slim
|
||||
# ffmpeg is DeadAir's (#171): it finds the silences and copies the frames kept. Debian's adds
|
||||
|
||||
19
crates/feed/Cargo.toml
Normal file
19
crates/feed/Cargo.toml
Normal file
@@ -0,0 +1,19 @@
|
||||
[package]
|
||||
name = "ipx-feed"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
atom_syndication.workspace = true
|
||||
chrono.workspace = true
|
||||
opml.workspace = true
|
||||
quick-xml.workspace = true
|
||||
reqwest.workspace = true
|
||||
rss.workspace = true
|
||||
serde_json.workspace = true
|
||||
tracing.workspace = true
|
||||
url.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tokio.workspace = true
|
||||
@@ -1,11 +1,10 @@
|
||||
//! Feed fetching and parsing. Replaces FeedData.__getFeed / __getEntries.
|
||||
//! Feed fetching and parsing, shared by ipx and the directory (#202). Replaces
|
||||
//! FeedData.__getFeed / __getEntries.
|
||||
|
||||
use anyhow::{Context, Result, anyhow};
|
||||
use reqwest::StatusCode;
|
||||
use reqwest::header::{ETAG, IF_MODIFIED_SINCE, IF_NONE_MATCH, LAST_MODIFIED};
|
||||
|
||||
use crate::config::Feed as FeedCfg;
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
pub struct ParsedFeed {
|
||||
pub title: Option<String>,
|
||||
@@ -67,14 +66,16 @@ pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30)
|
||||
/// (301 or 308): a publisher that moved its feed, which the catalogue should follow rather
|
||||
/// than be redirected on every read. A temporary redirect (302, 307) moves nothing.
|
||||
/// `system-proxy` reads http_proxy/https_proxy.
|
||||
#[tracing::instrument(skip_all, fields(url = %cfg.url))]
|
||||
/// `auth` is a username and password, sent only to the feed's own host.
|
||||
#[tracing::instrument(skip_all, fields(url = %feed_url))]
|
||||
pub async fn fetch(
|
||||
client: &reqwest::Client,
|
||||
cfg: &FeedCfg,
|
||||
feed_url: &str,
|
||||
auth: Option<(&str, Option<String>)>,
|
||||
etag: Option<&str>,
|
||||
last_modified: Option<&str>,
|
||||
) -> Result<(Fetched, Option<String>)> {
|
||||
let start = reqwest::Url::parse(&cfg.url).context("the feed's address")?;
|
||||
let start = reqwest::Url::parse(feed_url).context("the feed's address")?;
|
||||
let mut url = start.clone();
|
||||
let mut permanent = true;
|
||||
for _ in 0..10 {
|
||||
@@ -86,10 +87,10 @@ pub async fn fetch(
|
||||
req = req.header(IF_MODIFIED_SINCE, lm);
|
||||
}
|
||||
// The feed's own host only: a redirect elsewhere must not be handed the password.
|
||||
if let Some(user) = &cfg.username
|
||||
if let Some((user, password)) = &auth
|
||||
&& url.host_str() == start.host_str()
|
||||
{
|
||||
req = req.basic_auth(user, cfg.password());
|
||||
req = req.basic_auth(user, password.as_deref());
|
||||
}
|
||||
|
||||
let resp = req.send().await.context("connecting")?;
|
||||
@@ -231,10 +232,21 @@ pub fn parse_opml(bytes: &[u8]) -> Result<Vec<(String, String)>> {
|
||||
let doc = opml::OPML::from_str(&text)
|
||||
.map_err(|e| anyhow!("that does not parse as OPML: {e}"))?;
|
||||
let mut out = vec![];
|
||||
crate::collect_outlines(&doc.body.outlines, &mut out);
|
||||
collect_outlines(&doc.body.outlines, &mut out);
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// OPML nests feeds inside folder outlines, so this walks the whole tree.
|
||||
pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, String)>) {
|
||||
for o in outlines {
|
||||
if let Some(url) = &o.xml_url {
|
||||
let title = o.title.clone().unwrap_or_else(|| o.text.clone());
|
||||
out.push((title, url.clone()));
|
||||
}
|
||||
collect_outlines(&o.outlines, out);
|
||||
}
|
||||
}
|
||||
|
||||
/// The <head><title> of an OPML document.
|
||||
pub fn opml_title(bytes: &[u8]) -> Option<String> {
|
||||
let text = String::from_utf8_lossy(bytes);
|
||||
@@ -1121,6 +1133,34 @@ fn picked(answer: &serde_json::Value) -> Result<String> {
|
||||
.ok_or_else(|| anyhow!("no category in the answer: {answer}"))
|
||||
}
|
||||
|
||||
/// An address that carries a subscriber's own access: a paid-feed host, credentials in the URL,
|
||||
/// or a key in the query. ipx leaves such a feed out of anything others see, and the directory
|
||||
/// refuses one sent to it (#202).
|
||||
///
|
||||
/// ponytail: a heuristic. A key hidden in the path of a host not in `PAID_HOSTS` gets
|
||||
/// through; a per-feed private flag is the upgrade (#205).
|
||||
pub fn url_looks_private(feed_url: &str) -> bool {
|
||||
let Ok(u) = url::Url::parse(feed_url) else { return true };
|
||||
// Paid-feed services put the subscriber's key in the path as often as in the query, and a
|
||||
// long path segment alone proves nothing (acast's public show ids look the same). So a
|
||||
// feed from one of them is private whatever its URL looks like. Supercast is why this
|
||||
// exists: `feeds.supercast.com/feeds/<key>` reached the directory before it did.
|
||||
let host = u.host_str().unwrap_or("");
|
||||
PAID_HOSTS.iter().any(|h| host == *h || host.ends_with(&format!(".{h}")))
|
||||
|| !u.username().is_empty()
|
||||
|| u.password().is_some()
|
||||
|| u.query_pairs().any(|(k, _)| {
|
||||
let k = k.to_ascii_lowercase();
|
||||
["auth", "token", "key", "secret", "pass", "sig", "session", "user", "uid"]
|
||||
.iter()
|
||||
.any(|w| k.contains(w))
|
||||
})
|
||||
}
|
||||
|
||||
/// Services whose feeds are always one subscriber's own.
|
||||
const PAID_HOSTS: &[&str] =
|
||||
&["patreon.com", "supercast.com", "supportingcast.fm", "glow.fm", "memberful.com"];
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
/// A server answering by path: /old moves for good to /new, /tmp for now, /chain for good
|
||||
@@ -1157,9 +1197,9 @@ mod tests {
|
||||
let base = redirecting_server().await;
|
||||
let client = reqwest::Client::builder().redirect(reqwest::redirect::Policy::none()).build().unwrap();
|
||||
let get = |path: &str| {
|
||||
let cfg: crate::config::Feed = serde_json::from_value(serde_json::json!({ "url": format!("{base}{path}") })).unwrap();
|
||||
let url = format!("{base}{path}");
|
||||
let client = client.clone();
|
||||
async move { super::fetch(&client, &cfg, None, None).await }
|
||||
async move { super::fetch(&client, &url, None, None, None).await }
|
||||
};
|
||||
let (got, moved) = get("/old").await.unwrap();
|
||||
assert!(matches!(got, super::Fetched::Body { .. }));
|
||||
@@ -1175,7 +1215,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn parses_rss_with_itunes_extensions() {
|
||||
let bytes = include_bytes!("../tests/data/rss2.xml");
|
||||
let bytes = include_bytes!("../../../tests/data/rss2.xml");
|
||||
let feed = parse(bytes).unwrap();
|
||||
|
||||
assert_eq!(feed.title.as_deref(), Some("Test Cast"));
|
||||
@@ -1381,7 +1421,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn parses_atom_enclosure_links() {
|
||||
let bytes = include_bytes!("../tests/data/atom.xml");
|
||||
let bytes = include_bytes!("../../../tests/data/atom.xml");
|
||||
let feed = parse(bytes).unwrap();
|
||||
|
||||
assert_eq!(feed.title.as_deref(), Some("Atom Cast"));
|
||||
@@ -1479,8 +1519,8 @@ mod tests {
|
||||
assert_eq!(feeds[0], ("Alpha".into(), "https://a.example/rss".into()));
|
||||
|
||||
// A feed must never be mistaken for a subscription list.
|
||||
assert!(!is_opml(include_bytes!("../tests/data/rss2.xml")));
|
||||
assert!(!is_opml(include_bytes!("../tests/data/atom.xml")));
|
||||
assert!(!is_opml(include_bytes!("../../../tests/data/rss2.xml")));
|
||||
assert!(!is_opml(include_bytes!("../../../tests/data/atom.xml")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
37
crates/ipx/Cargo.toml
Normal file
37
crates/ipx/Cargo.toml
Normal file
@@ -0,0 +1,37 @@
|
||||
[package]
|
||||
name = "ipx"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
[dependencies]
|
||||
ipx-feed.workspace = true
|
||||
ammonia.workspace = true
|
||||
anyhow.workspace = true
|
||||
argon2.workspace = true
|
||||
atom_syndication.workspace = true
|
||||
axum.workspace = true
|
||||
chrono.workspace = true
|
||||
clap.workspace = true
|
||||
futures-util.workspace = true
|
||||
jsonwebtoken.workspace = true
|
||||
librqbit.workspace = true
|
||||
opentelemetry.workspace = true
|
||||
opentelemetry-otlp.workspace = true
|
||||
opentelemetry_sdk.workspace = true
|
||||
opml.workspace = true
|
||||
percent-encoding.workspace = true
|
||||
quick-xml.workspace = true
|
||||
reqwest.workspace = true
|
||||
rss.workspace = true
|
||||
sea-orm.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
sha2.workspace = true
|
||||
tokio.workspace = true
|
||||
toml.workspace = true
|
||||
tower.workspace = true
|
||||
tower-http.workspace = true
|
||||
tracing.workspace = true
|
||||
tracing-opentelemetry.workspace = true
|
||||
tracing-subscriber.workspace = true
|
||||
url.workspace = true
|
||||
@@ -1,13 +1,13 @@
|
||||
//! Builds web/index.html and web/login.html from their TypeScript (web/build.mjs) into OUT_DIR,
|
||||
//! where src/web.rs include_str!s them. Needs node and `npm ci` run first.
|
||||
//! where src/web.rs include_str!s them. Needs node and `npm ci` run first, at the repo's root.
|
||||
use std::process::Command;
|
||||
|
||||
fn main() {
|
||||
println!("cargo:rerun-if-changed=web");
|
||||
println!("cargo:rerun-if-changed=package-lock.json");
|
||||
println!("cargo:rerun-if-changed=../../web");
|
||||
println!("cargo:rerun-if-changed=../../package-lock.json");
|
||||
let out = std::env::var("OUT_DIR").unwrap();
|
||||
let status = Command::new("node")
|
||||
.args(["web/build.mjs", &out])
|
||||
.args(["../../web/build.mjs", &out])
|
||||
.status()
|
||||
.expect("building the web pages needs node on PATH (and `npm ci` run once)");
|
||||
assert!(status.success(), "web/build.mjs failed; run `node web/build.mjs` to see why");
|
||||
@@ -137,7 +137,7 @@ mod tests {
|
||||
const AUD: &str = "aud-tag";
|
||||
|
||||
fn jwks() -> JwkSet {
|
||||
serde_json::from_str(include_str!("../tests/data/access-test.jwks.json")).unwrap()
|
||||
serde_json::from_str(include_str!("../../../tests/data/access-test.jwks.json")).unwrap()
|
||||
}
|
||||
|
||||
fn token(key: &[u8], alg: Algorithm, claims: serde_json::Value) -> String {
|
||||
@@ -155,8 +155,8 @@ mod tests {
|
||||
})
|
||||
}
|
||||
|
||||
const SIGNER: &[u8] = include_bytes!("../tests/data/access-test.der");
|
||||
const FORGER: &[u8] = include_bytes!("../tests/data/access-forger.der");
|
||||
const SIGNER: &[u8] = include_bytes!("../../../tests/data/access-test.der");
|
||||
const FORGER: &[u8] = include_bytes!("../../../tests/data/access-forger.der");
|
||||
|
||||
async fn check(keys: &Keys, t: &str) -> Option<String> {
|
||||
keys.verify_with(TEAM, AUD, t, || async { Ok(jwks()) }).await
|
||||
@@ -273,6 +273,11 @@ impl Feed {
|
||||
}
|
||||
self.password.clone()
|
||||
}
|
||||
|
||||
/// What `feed::fetch` sends the feed's host: a username, and its password if there is one.
|
||||
pub fn auth(&self) -> Option<(&str, Option<String>)> {
|
||||
self.username.as_deref().map(|u| (u, self.password()))
|
||||
}
|
||||
}
|
||||
|
||||
impl Config {
|
||||
@@ -6,7 +6,8 @@ mod db;
|
||||
mod deadair;
|
||||
mod entity;
|
||||
mod download;
|
||||
mod feed;
|
||||
// The crate root's import, so `crate::feed` reaches it from every module as before.
|
||||
use ipx_feed as feed;
|
||||
mod ipc;
|
||||
mod logbuf;
|
||||
mod metrics;
|
||||
@@ -908,7 +909,7 @@ async fn add_one(
|
||||
listed: false,
|
||||
};
|
||||
|
||||
let title = match feed::fetch(&ctx.feed_client, &probe, None, None).await.map(|(got, _)| got) {
|
||||
let title = match feed::fetch(&ctx.feed_client, &probe.url, None, None, None).await.map(|(got, _)| got) {
|
||||
// An OPML subscription is named from its own <head><title>, not by trying to
|
||||
// parse it as a feed and falling back to the hostname.
|
||||
Ok(feed::Fetched::Body { bytes, .. }) if feed::is_opml(&bytes) => {
|
||||
@@ -999,7 +1000,7 @@ pub async fn subscribe_opml(
|
||||
user_id: i64,
|
||||
) -> Result<(usize, usize)> {
|
||||
let mut found = vec![];
|
||||
collect_outlines(&doc.body.outlines, &mut found);
|
||||
feed::collect_outlines(&doc.body.outlines, &mut found);
|
||||
|
||||
let mut cfg = ctx.edit_cfg().await;
|
||||
let known = subscriptions(ctx).await?;
|
||||
@@ -1057,17 +1058,6 @@ pub async fn subscribe_opml(
|
||||
Ok((added, had))
|
||||
}
|
||||
|
||||
/// OPML nests feeds inside folder outlines, so this walks the whole tree.
|
||||
pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, String)>) {
|
||||
for o in outlines {
|
||||
if let Some(url) = &o.xml_url {
|
||||
let title = o.title.clone().unwrap_or_else(|| o.text.clone());
|
||||
out.push((title, url.clone()));
|
||||
}
|
||||
collect_outlines(&o.outlines, out);
|
||||
}
|
||||
}
|
||||
|
||||
/// The configuration ipx runs with: config.toml for where things are and who may sign in, the
|
||||
/// database for the feeds and the server settings (issue #18). The first time a database holds
|
||||
/// neither, it takes them from config.toml, which is then cut down to the rest, the original kept
|
||||
@@ -1264,7 +1254,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
|
||||
return None;
|
||||
}
|
||||
let _fetching = metrics::busy(&metrics::STATS.feeds_fetching);
|
||||
Some(feed::fetch(&client, &feed_cfg, etag.as_deref(), modified.as_deref()).await)
|
||||
Some(feed::fetch(&client, &feed_cfg.url, feed_cfg.auth(), etag.as_deref(), modified.as_deref()).await)
|
||||
},
|
||||
tracing::Span::current(),
|
||||
))
|
||||
@@ -1654,7 +1644,7 @@ async fn scan_one(
|
||||
|
||||
let (mut fetched, moved) = match prefetched {
|
||||
Some(got) => got?,
|
||||
None => feed::fetch(&ctx.feed_client, feed_cfg, state.etag.as_deref(), state.last_modified.as_deref()).await?,
|
||||
None => feed::fetch(&ctx.feed_client, &feed_cfg.url, feed_cfg.auth(), state.etag.as_deref(), state.last_modified.as_deref()).await?,
|
||||
};
|
||||
if let Some(to) = moved {
|
||||
follow_move(ctx, id, &to).await?;
|
||||
@@ -1669,7 +1659,7 @@ async fn scan_one(
|
||||
if matches!(fetched, feed::Fetched::NotModified) && (stored.entries == 0 || stored.image.is_none()) {
|
||||
tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator");
|
||||
ctx.db.clear_validators(id).await?;
|
||||
fetched = feed::fetch(&ctx.feed_client, feed_cfg, None, None).await?.0;
|
||||
fetched = feed::fetch(&ctx.feed_client, &feed_cfg.url, feed_cfg.auth(), None, None).await?.0;
|
||||
}
|
||||
|
||||
let (bytes, etag, last_modified) = match fetched {
|
||||
@@ -663,7 +663,7 @@ fn script(js: &'static str) -> impl IntoResponse {
|
||||
async fn logo() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "image/svg+xml"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||
include_str!("../web/logo.svg"),
|
||||
include_str!("../../../web/logo.svg"),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -672,7 +672,7 @@ async fn logo() -> impl IntoResponse {
|
||||
async fn logo_dark() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "image/svg+xml"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||
include_str!("../web/logo-dark.svg"),
|
||||
include_str!("../../../web/logo-dark.svg"),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -680,7 +680,7 @@ async fn logo_dark() -> impl IntoResponse {
|
||||
async fn icon() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||
include_bytes!("../web/ipodderx-icon.png").as_slice(),
|
||||
include_bytes!("../../../web/ipodderx-icon.png").as_slice(),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -689,7 +689,7 @@ async fn icon() -> impl IntoResponse {
|
||||
async fn favicon() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||
include_bytes!("../web/favicon.png").as_slice(),
|
||||
include_bytes!("../../../web/favicon.png").as_slice(),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -697,7 +697,7 @@ async fn favicon() -> impl IntoResponse {
|
||||
async fn favicon_dark() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||
include_bytes!("../web/favicon-dark.png").as_slice(),
|
||||
include_bytes!("../../../web/favicon-dark.png").as_slice(),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -706,7 +706,7 @@ async fn favicon_dark() -> impl IntoResponse {
|
||||
async fn touch_icon() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "image/png"), (header::CACHE_CONTROL, "max-age=86400")],
|
||||
include_bytes!("../web/apple-touch-icon.png").as_slice(),
|
||||
include_bytes!("../../../web/apple-touch-icon.png").as_slice(),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -716,7 +716,7 @@ async fn touch_icon() -> impl IntoResponse {
|
||||
async fn inter() -> impl IntoResponse {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "font/woff2"), (header::CACHE_CONTROL, "max-age=604800")],
|
||||
include_bytes!("../web/InterVariable.woff2").as_slice(),
|
||||
include_bytes!("../../../web/InterVariable.woff2").as_slice(),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -936,35 +936,15 @@ async fn feed_rows(state: &WebState, user: &crate::db::User, only: Option<&str>)
|
||||
// ---- popular on this server ----
|
||||
|
||||
/// A feed that carries a credential is someone's paid or private subscription. Listing it
|
||||
/// would let anyone signed in subscribe to it and read what they pay for.
|
||||
///
|
||||
/// ponytail: a heuristic. A key hidden in the path of a host not in `PAID_HOSTS` gets
|
||||
/// through; a per-feed `unlisted` flag is the upgrade if that happens again.
|
||||
/// would let anyone signed in subscribe to it and read what they pay for. The address half is
|
||||
/// `feed::url_looks_private`, which the directory shares.
|
||||
fn looks_private(feed: &crate::config::Feed) -> bool {
|
||||
if feed.username.is_some() || feed.password.is_some() || feed.password_env.is_some() {
|
||||
return true;
|
||||
}
|
||||
let Ok(u) = url::Url::parse(&feed.url) else { return true };
|
||||
// Paid-feed services put the subscriber's key in the path as often as in the query, and a
|
||||
// long path segment alone proves nothing (acast's public show ids look the same). So a
|
||||
// feed from one of them is private whatever its URL looks like. Supercast is why this
|
||||
// exists: `feeds.supercast.com/feeds/<key>` reached the directory before it did.
|
||||
let host = u.host_str().unwrap_or("");
|
||||
PAID_HOSTS.iter().any(|h| host == *h || host.ends_with(&format!(".{h}")))
|
||||
|| !u.username().is_empty()
|
||||
|| u.password().is_some()
|
||||
|| u.query_pairs().any(|(k, _)| {
|
||||
let k = k.to_ascii_lowercase();
|
||||
["auth", "token", "key", "secret", "pass", "sig", "session", "user", "uid"]
|
||||
.iter()
|
||||
.any(|w| k.contains(w))
|
||||
})
|
||||
feed.username.is_some()
|
||||
|| feed.password.is_some()
|
||||
|| feed.password_env.is_some()
|
||||
|| crate::feed::url_looks_private(&feed.url)
|
||||
}
|
||||
|
||||
/// Services whose feeds are always one subscriber's own.
|
||||
const PAID_HOSTS: &[&str] =
|
||||
&["patreon.com", "supercast.com", "supportingcast.fm", "glow.fm", "memberful.com"];
|
||||
|
||||
/// Only an id, a title, artwork and a count: never a URL, which is where a key would be.
|
||||
#[derive(Serialize)]
|
||||
struct PopularRow {
|
||||
@@ -4,28 +4,32 @@ One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a
|
||||
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
|
||||
it to a running daemon.
|
||||
|
||||
The repo is a Cargo workspace (#202): `crates/ipx` is the daemon and CLI, and `crates/feed`
|
||||
(`ipx-feed`) fetches and parses feeds, kept apart because the directory service (#204) shares it.
|
||||
The version is the workspace's, in the root `Cargo.toml`.
|
||||
|
||||
## Modules
|
||||
|
||||
| File | Responsibility | What it replaced in the Python |
|
||||
|---|---|---|
|
||||
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
||||
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
||||
| `src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
||||
| `src/entity.rs` | The tables, as SeaORM entities: the schema | — |
|
||||
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
|
||||
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
||||
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
||||
| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
|
||||
| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
|
||||
| `src/auth.rs` | Argon2id hashing, session tokens, header names | — |
|
||||
| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
|
||||
| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
|
||||
| `crates/ipx/src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
||||
| `crates/ipx/src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
||||
| `crates/ipx/src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
||||
| `crates/ipx/src/entity.rs` | The tables, as SeaORM entities: the schema | — |
|
||||
| `crates/feed/src/lib.rs` | Conditional GET, RSS/Atom/OPML parsing, `url_looks_private` | `FeedData.__getFeed/__getEntries` |
|
||||
| `crates/ipx/src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
||||
| `crates/ipx/src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
||||
| `crates/ipx/src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
|
||||
| `crates/ipx/src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
|
||||
| `crates/ipx/src/auth.rs` | Argon2id hashing, session tokens, header names | — |
|
||||
| `crates/ipx/src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
|
||||
| `crates/ipx/src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
|
||||
| `web/index.html` | The app's markup | — |
|
||||
| `web/admin.html` | The admin page's markup: server settings, accounts, the log. Sent to admins only | — |
|
||||
| `web/app.css` | The stylesheet both pages share | — |
|
||||
| `web/src/*.ts` | The page's script, one scope split across files, type-checked by `npx tsc` | — |
|
||||
| `web/build.mjs` | swc: strips the types into `app.js`/`login.js`, named in the page by a hash of their contents, and minifies | — |
|
||||
| `build.rs` | Runs `web/build.mjs` into `OUT_DIR`, where `web.rs` `include_str!`s the result | — |
|
||||
| `crates/ipx/build.rs` | Runs `web/build.mjs` into `OUT_DIR`, where `web.rs` `include_str!`s the result | — |
|
||||
|
||||
The page is compiled in, so **editing `web/index.html` or `web/src` needs a rebuild**, and a
|
||||
build needs node and `npm ci` run once.
|
||||
@@ -70,7 +74,7 @@ entry_state user_id, feed_id, guid, read, flagged, position
|
||||
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
||||
before accounts; two bugs came from queries still reading them, and they were dropped in 0.5.
|
||||
|
||||
Schema changes: the tables are the entities in `src/entity.rs`, and `Db::open` creates whatever
|
||||
Schema changes: the tables are the entities in `crates/ipx/src/entity.rs`, and `Db::open` creates whatever
|
||||
table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It
|
||||
never alters a table that exists, so a new column on one needs its own `ALTER` in
|
||||
`create_missing`, looked for first and run only when it is missing, as `feeds.description`'s is,
|
||||
|
||||
Reference in New Issue
Block a user