10 Commits

Author SHA1 Message Date
bc53e0f730 Postgres: pick the database by URL, copy-db, tests on both
- IPX_DATABASE_URL (postgres://...) picks the database; unset, it is the SQLite
  file as before. Passwords are taken out of anything logged.
- `ipx copy-db <state.db>` copies every table into the empty database the URL
  names, in one transaction, and moves the id counters past the copied ids. A
  copy of production went across in 14s with every count and column
  fingerprint identical.
- With IPX_TEST_DATABASE_URL set, each test gets a Postgres schema of its own;
  all 79 pass on both databases. Fixtures write booleans as true/false.
- Sorts say where an item with no value goes (NULLS FIRST going up, LAST going
  down): SQLite counts NULL as smallest, Postgres as largest, so "largest first"
  on Postgres led with every item that has no file. Tested on both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 20:28:50 +00:00
bd8f6ab855 Docs: the database through SeaORM
CLAUDE.md and the architecture notes described the SQL schema and migrate(),
both gone: the entities are the schema, create_missing makes what is missing,
and hand-written SQL has to run on SQLite and Postgres both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 19:11:24 +00:00
611716d8b7 SeaORM: feeds and scanning; rusqlite gone
The last nineteen functions move to SeaORM: recording feeds, items and
enclosures, managed OPML feeds, folding WordPress's repeated files, and handing a
Patreon creator's files to its shows. Two SQLite-only forms go: GLOB becomes a
LIKE with the underscore escaped (broader, harmlessly: the fold still keys on
`_=` and digits), and UPDATE OR IGNORE becomes an UPDATE ... WHERE NOT EXISTS.
The two transactions are SeaORM transactions.

With nothing left on it, rusqlite goes, with the SQL schema and migrate(). The
entities are the schema: create_missing makes whatever tables and indexes a
database lacks, from them, with CREATE ... IF NOT EXISTS. Production's schema
already has every column migrate() added and none it dropped.

Not SeaORM's schema sync, used until now: despite its docs it drops a unique
index the entities do not describe, so it dropped users_name_lower on every open.
Every `ipx` command then took a write lock, and against a daemon busy writing,
`ipx status` -- the healthcheck -- failed 7 times in 15 where the old code
failed none. Now 15 in 15, as before. On Postgres it would not have started.

WAL is set only when a file is not already in it: setting it takes a lock that
cannot wait out a busy daemon.

Checked on copies of production: a forced scan of all 162 feeds against the real
feeds with no database errors; the feed list, filters, sorts, search and the
reaper's candidates against the old code on the same data, earlier in the
branch. The column comments from the SQL schema move to the entities.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 19:10:59 +00:00
a68bfb179b SeaORM: enclosures, downloads and the reaper
Twelve enclosure functions move to SeaORM: recording, the download queue,
marking done or failed, requeueing, and what the reaper may delete. INSERT OR
IGNORE becomes ON CONFLICT DO NOTHING; the reaper's read verdict is true or
false rather than 1 or 0, which Postgres would type as a 32-bit integer and
refuse to read as an i64; `read = 1` and `flagged = 1` test the booleans
themselves. retention::run and its callers (reap, rm, retire_group,
retire_stranded) become async.

The reaper deletes files, so it was checked on a copy of production against the
old SQL on the same file: all 2,195 candidates, identical and in the same order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:38:07 +00:00
6bf1ad6b31 SeaORM: items and read state
The item list, its counts, filters, sorts and search, positions, pins and
mark-all-read move to SeaORM, as SQL written for both databases:

- Parameters are gathered as the SQL is written (Args), so only what a
  statement uses is bound. rusqlite needed every one mentioned, hence the old
  `?1 IS NULL` and `?2 = ''`; Postgres refuses a parameter it cannot type.
- Yes/no columns are tested as booleans (NOT coalesce(s.read, false)) and
  written as true, not 1; SQLite reads true and false as 1 and 0.
- The last tiebreak of the sort is the guid, not SQLite's rowid, which Postgres
  lacks. Only items with the same date change places.
- set_position names entry_state.duration beside excluded.duration.
- The status callback on the control socket returns a future, as the counts
  are now a query.

Checked on a copy of production against the live server: 42 of 48 lists
identical; the other six differ only in how ties fall, or because the test
daemon cleared paths to files this machine does not have. Run on the same file,
every filter's count matches the old SQL exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:32:46 +00:00
d7f8f2df1d SeaORM: subscriptions and pins
Twelve subscription functions move to SeaORM. Lookups use the entity API; the
joins, counts and upserts are SQL written to run on both databases: $n
parameters, ON CONFLICT DO NOTHING in place of INSERT OR IGNORE, and
CASE WHEN on the yes/no column itself rather than comparing it to 1, which
Postgres would refuse for a boolean. INSERT ... SELECT ... ON CONFLICT gets a
WHERE true, which SQLite needs to tell the two apart.

Checked with a daemon on a copy of production: the feed list, read through the
new code, comes back with every feed and its settings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:23:37 +00:00
4927677e66 SeaORM: accounts, sessions and themes
The fourteen user and session functions move from rusqlite to SeaORM and become
async; their callers await them (auth, admin_user, user_cmd, the account
handlers). Checked against a copy of production, where the yes/no columns are
still INTEGER: the admin flag reads back right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:17:52 +00:00
484aaa1849 SeaORM beside rusqlite: entities, schema sync, a second connection
The first step of moving to SeaORM (#18, phase 1). Nothing a user sees changes.

- src/entity.rs: the seven tables as SeaORM entities, matching the SQLite schema.
  Strings are Text, as the columns are; yes/no columns are bool, which is BOOLEAN
  on Postgres and stays INTEGER in the existing SQLite file (sync notes the
  difference and leaves it alone).
- Db holds a SeaORM connection to the same SQLite file beside the rusqlite one;
  functions move to it one at a time, and rusqlite goes with the last of them.
- db::sync creates what a database is missing from the entities (SeaORM's
  schema-sync, experimental, so sea-orm is pinned to ~2.0), plus the two indexes
  an entity cannot express. Checked against a copy of production: it added the
  lower(name) index and changed nothing else.
- Test databases are now built from the entities alone, in a temporary file
  (two connections to one ":memory:" are two databases), so every test also
  checks that the entities describe what the queries need. That caught the one
  difference: finding a user by name relied on COLLATE NOCASE, which Postgres
  lacks; it now compares lower() on both sides.
- rusqlite steps back to 0.39: 0.40's libsqlite3-sys is newer than sqlx accepts,
  and only one may link SQLite. It goes away at the end of this phase.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:11:28 +00:00
c99e17bd80 A pinned item goes to the top of its list
order_sql takes pinned_first, which puts coalesce(s.flagged, 0) DESC ahead of
the chosen sort, so pins lead every list in whatever order is asked for and on
every page of it. Not when sorting by the pin column itself, where the direction
is the point, and not for Currently Listening. Pinning now asks for the list again
so the row moves at once, instead of redrawing it where it stood.

Closes #35.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:51:38 +00:00
e312f11bb1 Remove docs/history.md
It had grown past 1,700 lines, too large to be read or kept up. What it held --
what was wrong before a change and what it cost to find -- goes in commit
message bodies now, beside the change. CLAUDE.md says so; the README and the
changelog no longer point at it. It remains in git history.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 16:47:21 +00:00
15 changed files with 2652 additions and 3303 deletions

View File

@@ -5,11 +5,17 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
The long form, with what was wrong before and how it was found, is in
[docs/history.md](docs/history.md).
## [Unreleased] ## [Unreleased]
### Changed
- The database is reached through SeaORM, on the way to Postgres (issue #18); it is still the
same SQLite file, and nothing you see changes. A database from before 0.7 has to be opened by a
0.7 release first, which brings its tables up to date.
- A pinned item sits at the top of its list, above everything else in whatever order you sort
by, and moves there the moment you pin it. Sorting by the pin column itself still goes both
ways, and Currently Listening keeps its own order.
## [0.7.0] - 2026-09-18 ## [0.7.0] - 2026-09-18
### Added ### Added

View File

@@ -125,7 +125,7 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
subscriber. Two feeds publishing the same URL means only the first one scanned shows it. subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged` * **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
and `position` columns from before accounts; two bugs came from queries still reading them and `position` columns from before accounts; two bugs came from queries still reading them
(retention, and the entry pruner), and `migrate()` now drops them. (retention, and the entry pruner), and they were dropped in 0.5.
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once; * **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are `subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
derived and never written to config. derived and never written to config.
@@ -140,10 +140,16 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a * `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
different task. `ipx status` checks the control socket and the database; to see the worker different task. `ipx status` checks the control socket and the database; to see the worker
getting through its jobs, watch for `scan complete` in the log. getting through its jobs, watch for `scan complete` in the log.
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's * **The database goes through SeaORM, and the entities in `src/entity.rs` are the schema.**
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on `Db::open` creates any missing table or index from them (`create_missing`), on every `ipx`
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait command, the healthcheck's `ipx status` included, so it must never write when nothing is
for `daemon started` in the log. Copy `state.db` aside before deploying one. missing: SeaORM's experimental schema sync dropped and remade an index on every open, the
write lock that took made `ipx status` time out behind a busy daemon, and it was removed for
it. A new column on an existing table needs its own `ALTER`; nothing adds one for you.
* **SQL written by hand in `db.rs` has to run on SQLite and Postgres both** (issue #18): `$1`
parameters, bound only if used; `ON CONFLICT`, not `INSERT OR IGNORE`; yes/no columns tested
as themselves (`NOT coalesce(s.read, false)`) and written as `true`/`false`, never compared to
1; no `rowid`, `GLOB` or `UPDATE OR IGNORE`. `Args` in `db.rs` builds the parameters.
## House style ## House style
@@ -154,9 +160,9 @@ addressed to the person using it.
Every change gets one line under `## [Unreleased]` in [CHANGELOG.md](CHANGELOG.md), in its Every change gets one line under `## [Unreleased]` in [CHANGELOG.md](CHANGELOG.md), in its
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) group: Added, Changed, Deprecated, [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) group: Added, Changed, Deprecated,
Removed, Fixed or Security. Say it the way someone using ipx would notice it. When there is more to Removed, Fixed or Security. Say it the way someone using ipx would notice it. When there is more to
say, such as what was wrong before or what it cost to find out, write it up at the top of say, such as what was wrong before or what it cost to find out, it goes in the commit message's
[docs/history.md](docs/history.md), dated. That record has been more useful than the git log more body, where `git log` and `git blame` find it beside the change. (There was a long-form
than once. `docs/history.md` until 0.7.0; it grew too large to be useful and was removed. It is in git.)
Cutting a release: rename `[Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` and open a new empty Cutting a release: rename `[Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` and open a new empty
`[Unreleased]` above it, bump `version` in `Cargo.toml`, tag the commit `vX.Y.Z`, and update the `[Unreleased]` above it, bump `version` in `Cargo.toml`, tag the commit `vX.Y.Z`, and update the

873
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -18,7 +18,7 @@ percent-encoding = "2.3.2"
quick-xml = { version = "0.42.0", features = ["escape-html"] } quick-xml = { version = "0.42.0", features = ["escape-html"] }
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] } reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] }
rss = "2.1.1" rss = "2.1.1"
rusqlite = { version = "0.40.2", features = ["bundled"] } sea-orm = { version = "2.0.3", default-features = false, features = ["sqlx-sqlite", "sqlx-postgres", "runtime-tokio-rustls", "macros", "with-json", "sqlite-use-returning-for-3_35"] }
serde = { version = "1.0.229", features = ["derive"] } serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151" serde_json = "1.0.151"
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] } tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }

View File

@@ -63,7 +63,6 @@ The UI is plain HTTP, so put TLS in front of it if it is reachable from outside
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik | | [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API | | [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release | | [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed | | [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
## Tests ## Tests

View File

@@ -10,7 +10,8 @@ it to a running daemon.
|---|---|---| |---|---|---|
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` | | `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` | | `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` | | `src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` |
| `src/entity.rs` | The tables, as SeaORM entities: the schema | — |
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` | | `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` | | `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 | | `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
@@ -65,14 +66,14 @@ entry_state user_id, feed_id, guid, read, flagged, position
``` ```
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an before accounts; two bugs came from queries still reading them, and they were dropped in 0.5.
older database.
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table Schema changes: the tables are the entities in `src/entity.rs`, and `Db::open` creates whatever
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before never alters a table that exists, so a new column on one needs its own `ALTER` in
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same `create_missing`, or `sea-orm-migration` once there are several. `Db::memory()` builds its
path as `Db::open`, so a migration cannot pass the tests while missing in production. database the same way, so the tests run on the schema production gets. A database from before
0.7 takes its last columns from the old `migrate()`, so it upgrades through a 0.7 release first.
## Control socket ## Control socket

File diff suppressed because it is too large Load Diff

2467
src/db.rs

File diff suppressed because it is too large Load Diff

244
src/entity.rs Normal file
View File

@@ -0,0 +1,244 @@
//! The database's tables as SeaORM entities: the one description of the schema, from which
//! `Db::open` creates what a database is missing, on SQLite or Postgres alike (see
//! `db::create_missing`). Times are Unix seconds.
pub mod feeds {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "feeds")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub id: String,
#[sea_orm(column_type = "Text")]
pub url: String,
#[sea_orm(column_type = "Text", nullable)]
pub title: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub image: Option<String>,
/// The channel's first <itunes:category>, for the Directory.
#[sea_orm(column_type = "Text", nullable)]
pub category: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub etag: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub last_modified: Option<String>,
pub last_checked: Option<i64>,
pub ttl_mins: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub last_error: Option<String>,
/// When the current run of failures began; NULL while the feed is healthy. Kept through
/// repeated failures so the UI can tell a blip (macmanx: failed once, fine an hour
/// later) from a feed that has been down for a day.
pub error_since: Option<i64>,
/// Came from a subscribed OPML that no longer lists it, but has downloads, so kept.
#[sea_orm(default_value = false)]
pub orphaned: bool,
/// The OPML subscription this feed came from.
#[sea_orm(column_type = "Text", nullable)]
pub group_id: Option<String>,
/// Derived from an OPML and not written to config.toml. Writing 80-odd generated entries
/// into a hand-edited file made it unreadable; the OPML is the source of truth, so they
/// are re-derived instead. Customising one promotes it to config.
#[sea_orm(default_value = false)]
pub managed: bool,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
pub mod entries {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "entries")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub feed_id: String,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub guid: String,
#[sea_orm(column_type = "Text", nullable)]
pub title: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub link: Option<String>,
pub published: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub description: Option<String>,
pub first_seen: i64,
#[sea_orm(column_type = "Text", nullable)]
pub image: Option<String>,
pub duration: Option<i64>,
pub episode: Option<i64>,
pub season: Option<i64>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
pub mod enclosures {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "enclosures")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i64,
#[sea_orm(column_type = "Text")]
pub feed_id: String,
#[sea_orm(column_type = "Text")]
pub guid: String,
/// The dedupe key, and the reason one file serves every subscriber. A reaped file keeps
/// its row with path NULL and state 'reaped', so a purged episode is never fetched again.
#[sea_orm(unique, column_type = "Text")]
pub url: String,
#[sea_orm(column_type = "Text", nullable)]
pub mime: Option<String>,
pub length: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub path: Option<String>,
#[sea_orm(column_type = "Text")]
pub state: String,
#[sea_orm(default_value = 0)]
pub bytes_done: i64,
pub downloaded_at: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub last_error: Option<String>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
pub mod users {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "users")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i64,
/// Unique without regard to case: `db::create_missing` adds the index on lower(name), which
/// works the same on both databases where SQLite's COLLATE NOCASE does not.
#[sea_orm(column_type = "Text")]
pub name: String,
/// NULL for someone who only ever arrives through the proxy: there is no password to
/// check, and leaving it empty is not the same as leaving it unset.
#[sea_orm(column_type = "Text", nullable)]
pub pass_hash: Option<String>,
#[sea_orm(default_value = false)]
pub is_admin: bool,
/// For whoever maintains the server. NULL where it is not known.
pub created: Option<i64>,
pub last_login: Option<i64>,
/// The theme chosen in Settings, and light, dark or auto. NULL until one is chosen.
#[sea_orm(column_type = "Text", nullable)]
pub theme: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub theme_mode: Option<String>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
/// A table of one person's rows, gone when they are.
macro_rules! owned_by_user {
() => {
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {
#[sea_orm(
belongs_to = "super::users::Entity",
from = "Column::UserId",
to = "super::users::Column::Id",
on_delete = "Cascade"
)]
User,
}
impl Related<super::users::Entity> for Entity {
fn to() -> RelationDef {
Relation::User.def()
}
}
impl ActiveModelBehavior for ActiveModel {}
};
}
/// What one person wants from a feed. The feed, its items and its files are shared; this is the
/// part that is not. NULL in a column means: follow the feed's own setting.
pub mod subscriptions {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "subscriptions")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub user_id: i64,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub feed_id: String,
/// JSON array of strings; NULL follows the feed.
#[sea_orm(column_type = "Text", nullable)]
pub keywords: Option<String>,
pub auto_download: Option<bool>,
pub allow_explicit: Option<bool>,
pub max_new_per_check: Option<i64>,
/// Pinned to the top of this person's feed list, a feed inside a folder included.
#[sea_orm(default_value = false)]
pub pinned: bool,
}
owned_by_user!();
}
/// Read, kept and how far in. One row per person per item, created on first touch; an item
/// nobody has touched has no row at all, which is what unread means.
pub mod entry_state {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "entry_state")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub user_id: i64,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub feed_id: String,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub guid: String,
#[sea_orm(default_value = false)]
pub read: bool,
#[sea_orm(default_value = false)]
pub flagged: bool,
#[sea_orm(default_value = 0)]
pub position: i64,
/// The length this person's player measured, beside the position it is measured against.
pub duration: Option<i64>,
}
owned_by_user!();
}
pub mod sessions {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "sessions")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub token: String,
pub user_id: i64,
pub seen: i64,
}
owned_by_user!();
}

View File

@@ -176,7 +176,9 @@ pub async fn daemon_is_live(path: &Path) -> bool {
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon /// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any /// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
/// `ipx fetch` that was watching a scan, since `status` is a terminal event. /// `ipx fetch` that was watching a scan, since `status` is a terminal event.
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>; /// A future, since reading the counts is a database query.
pub type StatusFn =
std::sync::Arc<dyn Fn() -> std::pin::Pin<Box<dyn std::future::Future<Output = Event> + Send>> + Send + Sync>;
/// Accepts connections, feeding commands to `cmds` and events from `events` back out. /// Accepts connections, feeding commands to `cmds` and events from `events` back out.
pub async fn serve( pub async fn serve(
@@ -249,7 +251,7 @@ async fn handle(
// Answered here, not queued behind whatever the worker is on: see StatusFn. // Answered here, not queued behind whatever the worker is on: see StatusFn.
Ok(Command::Status) => { Ok(Command::Status) => {
tracing::info!(target: "ipx::io", "-> {line}"); tracing::info!(target: "ipx::io", "-> {line}");
let ev = status(); let ev = status().await;
if let Ok(json) = serde_json::to_string(&ev) { if let Ok(json) = serde_json::to_string(&ev) {
tracing::info!(target: "ipx::io", "<- {json}"); tracing::info!(target: "ipx::io", "<- {json}");
} }
@@ -366,7 +368,8 @@ mod tests {
// Another client, watching a scan: it must not be handed someone else's answer, which // Another client, watching a scan: it must not be handed someone else's answer, which
// would end its session. // would end its session.
let mut watcher = events.subscribe(); let mut watcher = events.subscribe();
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 }); let status: StatusFn =
std::sync::Arc::new(|| Box::pin(async { Event::Status { feeds: 1, pending: 2, downloaded: 3 } }));
let (client, server) = UnixStream::pair().unwrap(); let (client, server) = UnixStream::pair().unwrap();
tokio::spawn(handle(server, events.subscribe(), cmds, status)); tokio::spawn(handle(server, events.subscribe(), cmds, status));

View File

@@ -1,6 +1,7 @@
mod auth; mod auth;
mod config; mod config;
mod db; mod db;
mod entity;
mod download; mod download;
mod feed; mod feed;
mod ipc; mod ipc;
@@ -36,6 +37,12 @@ struct Cli {
enum Command { enum Command {
/// Show configured feeds and their state /// Show configured feeds and their state
List, List,
/// Copy everything from a SQLite state.db into the database IPX_DATABASE_URL names, which
/// must be empty: the one-off move to Postgres
CopyDb {
/// The SQLite file to copy from
from: PathBuf,
},
/// Scan feeds for new entries /// Scan feeds for new entries
Fetch { Fetch {
/// Only this feed id /// Only this feed id
@@ -179,7 +186,7 @@ async fn main() -> Result<()> {
let config_path = cli.config.clone().unwrap_or_else(config::config_path); let config_path = cli.config.clone().unwrap_or_else(config::config_path);
let cfg = config::Config::load(&config_path)?; let cfg = config::Config::load(&config_path)?;
let db = db::Db::open(&config::data_dir().join("state.db"))?; let db = db::Db::open(&db::location()).await?;
// A daemon owns the state; don't have two processes downloading the same thing. // A daemon owns the state; don't have two processes downloading the same thing.
let wire_cmd = match &cli.command { let wire_cmd = match &cli.command {
@@ -194,7 +201,8 @@ async fn main() -> Result<()> {
| Command::Add { .. } | Command::Add { .. }
| Command::Rm { .. } | Command::Rm { .. }
| Command::Import { .. } | Command::Import { .. }
| Command::Export { .. } => None, | Command::Export { .. }
| Command::CopyDb { .. } => None,
}; };
if let Some(cmd) = &wire_cmd if let Some(cmd) = &wire_cmd
&& !cli.local && !cli.local
@@ -219,22 +227,23 @@ async fn main() -> Result<()> {
}); });
match cli.command { match cli.command {
Command::List => list(&ctx, &config_path), Command::List => list(&ctx, &config_path).await,
Command::Daemon { web } => daemon(ctx, config_path, web, events).await, Command::Daemon { web } => daemon(ctx, config_path, web, events).await,
Command::Add { url, folder, keywords } => { Command::Add { url, folder, keywords } => {
add(&ctx, &config_path, &url, folder, keywords).await add(&ctx, &config_path, &url, folder, keywords).await
} }
Command::Rm { feed } => rm(&ctx, &config_path, &feed), Command::Rm { feed } => rm(&ctx, &config_path, &feed).await,
Command::User { cmd } => user_cmd(&ctx, cmd), Command::User { cmd } => user_cmd(&ctx, cmd).await,
Command::Import { file } => import(&ctx, &config_path, &file).await, Command::Import { file } => import(&ctx, &config_path, &file).await,
Command::Export { file } => export(&ctx, &file), Command::Export { file } => export(&ctx, &file).await,
Command::CopyDb { from } => copy_db(&ctx, &from).await,
_ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await, _ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await,
} }
} }
/// Accounts. Passwords come in on stdin so they never reach a shell history or a `ps` /// Accounts. Passwords come in on stdin so they never reach a shell history or a `ps`
/// listing. /// listing.
fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> { async fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
let read_password = || -> Result<String> { let read_password = || -> Result<String> {
use std::io::Read; use std::io::Read;
let mut buf = String::new(); let mut buf = String::new();
@@ -252,7 +261,7 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
if name.is_empty() { if name.is_empty() {
anyhow::bail!("a name is required"); anyhow::bail!("a name is required");
} }
if ctx.db.user_by_name(&name)?.is_some() { if ctx.db.user_by_name(&name).await?.is_some() {
anyhow::bail!("{name} already exists"); anyhow::bail!("{name} already exists");
} }
let hash = if no_password { let hash = if no_password {
@@ -261,8 +270,8 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
Some(crate::auth::hash_password(&read_password()?)?) Some(crate::auth::hash_password(&read_password()?)?)
}; };
// The first account runs the place; there is nobody else to grant it. // The first account runs the place; there is nobody else to grant it.
let first = ctx.db.users()?.is_empty(); let first = ctx.db.users().await?.is_empty();
ctx.db.create_user(&name, hash.as_deref(), admin || first)?; ctx.db.create_user(&name, hash.as_deref(), admin || first).await?;
println!( println!(
"added {name}{}{}", "added {name}{}{}",
if admin || first { " (admin)" } else { "" }, if admin || first { " (admin)" } else { "" },
@@ -271,7 +280,7 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
Ok(()) Ok(())
} }
UserCmd::List => { UserCmd::List => {
let users = ctx.db.users()?; let users = ctx.db.users().await?;
if users.is_empty() { if users.is_empty() {
println!("no accounts yet: ipx user add <name>"); println!("no accounts yet: ipx user add <name>");
} }
@@ -294,9 +303,9 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
let name = name.trim().to_ascii_lowercase(); let name = name.trim().to_ascii_lowercase();
let user = ctx let user = ctx
.db .db
.user_by_name(&name)? .user_by_name(&name).await?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?; .ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
ctx.db.set_password(user.id, &crate::auth::hash_password(&read_password()?)?)?; ctx.db.set_password(user.id, &crate::auth::hash_password(&read_password()?)?).await?;
println!("password changed for {name}"); println!("password changed for {name}");
Ok(()) Ok(())
} }
@@ -307,12 +316,12 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?; .ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
let user = ctx let user = ctx
.db .db
.user_by_name(&name)? .user_by_name(&name).await?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?; .ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
if ctx.db.user_by_name(&new_name)?.is_some() { if ctx.db.user_by_name(&new_name).await?.is_some() {
anyhow::bail!("{new_name} already exists"); anyhow::bail!("{new_name} already exists");
} }
ctx.db.rename_user(user.id, &new_name)?; ctx.db.rename_user(user.id, &new_name).await?;
println!("renamed {name} to {new_name}"); println!("renamed {name} to {new_name}");
Ok(()) Ok(())
} }
@@ -320,9 +329,9 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
let name = name.trim().to_ascii_lowercase(); let name = name.trim().to_ascii_lowercase();
let user = ctx let user = ctx
.db .db
.user_by_name(&name)? .user_by_name(&name).await?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?; .ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
ctx.db.delete_user(user.id)?; ctx.db.delete_user(user.id).await?;
println!("removed {name}"); println!("removed {name}");
Ok(()) Ok(())
} }
@@ -333,13 +342,13 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
match cmd { match cmd {
Cmd::Fetch { feed, force } => { Cmd::Fetch { feed, force } => {
// Make room before pulling more down, as the original did per download. // Make room before pulling more down, as the original did per download.
reap(ctx, false, false)?; reap(ctx, false, false).await?;
fetch(ctx, feed.as_deref(), force).await fetch(ctx, feed.as_deref(), force).await
} }
Cmd::Reap { dry_run } => reap(ctx, dry_run, true), Cmd::Reap { dry_run } => reap(ctx, dry_run, true).await,
Cmd::Download { enclosure } => download_one(ctx, enclosure).await, Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
Cmd::Status => { Cmd::Status => {
ctx.out.emit(status(ctx)); ctx.out.emit(status(ctx).await);
Ok(()) Ok(())
} }
} }
@@ -347,10 +356,10 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather /// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
/// than through the job queue. /// than through the job queue.
fn status(ctx: &Ctx) -> Event { async fn status(ctx: &Ctx) -> Event {
match ctx.db.counts() { match ctx.db.counts().await {
Ok((pending, downloaded)) => { Ok((pending, downloaded)) => {
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0); let feeds = subscriptions(ctx).await.map(|s| s.len()).unwrap_or(0);
Event::Status { feeds, pending, downloaded } Event::Status { feeds, pending, downloaded }
} }
Err(e) => Event::Error { msg: format!("{e:#}") }, Err(e) => Event::Error { msg: format!("{e:#}") },
@@ -369,30 +378,30 @@ async fn daemon(
} }
// A database with nobody in it cannot be signed into. // A database with nobody in it cannot be signed into.
if ctx.db.users()?.is_empty() { if ctx.db.users().await?.is_empty() {
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?; ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true).await?;
tracing::warn!( tracing::warn!(
"no accounts yet: created 'admin' with the default password '{DEFAULT_PASSWORD}'. \ "no accounts yet: created 'admin' with the default password '{DEFAULT_PASSWORD}'. \
Change it with `echo -n <password> | ipx user passwd admin`" Change it with `echo -n <password> | ipx user passwd admin`"
); );
} }
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) { if let Some(admin) = ctx.db.users().await?.into_iter().find(|u| u.is_admin) {
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect(); let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
match ctx.db.adopt_catalogue(admin.id, &catalogue) { match ctx.db.adopt_catalogue(admin.id, &catalogue).await {
Ok(0) => {} Ok(0) => {}
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"), Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"), Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
} }
} }
match ctx.db.requeue_interrupted() { match ctx.db.requeue_interrupted().await {
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"), Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
Ok(_) => {} Ok(_) => {}
Err(e) => tracing::warn!(error = ?e, "could not requeue interrupted downloads"), Err(e) => tracing::warn!(error = ?e, "could not requeue interrupted downloads"),
} }
match retire_stranded(&ctx) { match retire_stranded(&ctx).await {
Ok(0) => {} Ok(0) => {}
Ok(n) => tracing::info!(feeds = n, "retired feeds whose OPML is no longer in config"), Ok(n) => tracing::info!(feeds = n, "retired feeds whose OPML is no longer in config"),
Err(e) => tracing::warn!(error = ?e, "could not retire feeds whose OPML is no longer in config"), Err(e) => tracing::warn!(error = ?e, "could not retire feeds whose OPML is no longer in config"),
@@ -400,7 +409,7 @@ async fn daemon(
// Before the parser knew WordPress's numbered player URLs, a file it listed twice was // Before the parser knew WordPress's numbered player URLs, a file it listed twice was
// downloaded twice. The repeats fold into the first, and their spare copies are deleted. // downloaded twice. The repeats fold into the first, and their spare copies are deleted.
match ctx.db.merge_repeated_enclosures(feed::same_file_key) { match ctx.db.merge_repeated_enclosures(feed::same_file_key).await {
Ok((0, _)) => {} Ok((0, _)) => {}
Ok((n, spare)) => { Ok((n, spare)) => {
for path in &spare { for path in &spare {
@@ -419,7 +428,10 @@ async fn daemon(
// status is answered by the socket itself; everything else waits its turn in the queue. // status is answered by the socket itself; everything else waits its turn in the queue.
let answer: ipc::StatusFn = { let answer: ipc::StatusFn = {
let ctx = ctx.clone(); let ctx = ctx.clone();
Arc::new(move || status(&ctx)) Arc::new(move || {
let ctx = ctx.clone();
Box::pin(async move { status(&ctx).await })
})
}; };
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer)); let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
@@ -427,7 +439,7 @@ async fn daemon(
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60)); let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
tracing::info!( tracing::info!(
feeds = subscriptions(&ctx).map(|s| s.len()).unwrap_or(0), feeds = subscriptions(&ctx).await.map(|s| s.len()).unwrap_or(0),
"daemon started" "daemon started"
); );
@@ -571,7 +583,7 @@ async fn add(
let mut cfg = (*ctx.cfg()).clone(); let mut cfg = (*ctx.cfg()).clone();
let url = &feed::expand_input(url); let url = &feed::expand_input(url);
// Includes feeds derived from an OPML, or the same show could be added twice. // Includes feeds derived from an OPML, or the same show could be added twice.
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) { if let Some(existing) = subscriptions(ctx).await?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
anyhow::bail!("already subscribed as {:?}", existing.id); anyhow::bail!("already subscribed as {:?}", existing.id);
} }
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?; let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
@@ -623,13 +635,13 @@ pub async fn add_one(
// Slugs must be unique across derived feeds too, or a new feed can collide with one // Slugs must be unique across derived feeds too, or a new feed can collide with one
// an OPML already introduced. // an OPML already introduced.
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)? let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx).await?
.into_iter() .into_iter()
.map(|s| (s.id, s.cfg)) .map(|s| (s.id, s.cfg))
.collect(); .collect();
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding // A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
// it gets its history back, and a different feed does not inherit someone else's. // it gets its history back, and a different feed does not inherit someone else's.
for (id, other) in ctx.db.feed_urls()? { for (id, other) in ctx.db.feed_urls().await? {
if !feed::same_feed(&other, url) { if !feed::same_feed(&other, url) {
taken.entry(id).or_insert_with(|| probe.clone()); taken.entry(id).or_insert_with(|| probe.clone());
} }
@@ -647,19 +659,19 @@ fn url_stem(url: &str) -> String {
.unwrap_or_else(|| url.to_owned()) .unwrap_or_else(|| url.to_owned())
} }
fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> { async fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
let mut cfg = (*ctx.cfg()).clone(); let mut cfg = (*ctx.cfg()).clone();
if cfg.feeds.remove(feed).is_none() { if cfg.feeds.remove(feed).is_none() {
// Derived from an OPML: drop it here, though the subscription will list it again // Derived from an OPML: drop it here, though the subscription will list it again
// on the next read unless the OPML itself goes. // on the next read unless the OPML itself goes.
ctx.db.drop_managed(feed)?; ctx.db.drop_managed(feed).await?;
println!("removed {feed}; it came from an OPML subscription and may return on the next read"); println!("removed {feed}; it came from an OPML subscription and may return on the next read");
return Ok(()); return Ok(());
} }
cfg.save(config_path)?; cfg.save(config_path)?;
// State and files stay: re-adding the feed should not re-download its back catalogue. // State and files stay: re-adding the feed should not re-download its back catalogue.
println!("removed {feed}; downloads and history kept"); println!("removed {feed}; downloads and history kept");
retire_group(ctx, feed)?; retire_group(ctx, feed).await?;
Ok(()) Ok(())
} }
@@ -669,13 +681,13 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
// The CLI speaks for the operator, as the shared web token does. // The CLI speaks for the operator, as the shared web token does.
let admin = ctx let admin = ctx
.db .db
.users()? .users().await?
.into_iter() .into_iter()
.find(|u| u.is_admin) .find(|u| u.is_admin)
.ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?; .ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?;
let doc = opml::OPML::from_str(&text) let doc = opml::OPML::from_str(&text)
.map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?; .map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?;
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id)?; let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id).await?;
println!("subscribed {} to {added} feed(s); {had} already there", admin.name); println!("subscribed {} to {added} feed(s); {had} already there", admin.name);
Ok(()) Ok(())
} }
@@ -690,7 +702,7 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
/// ///
/// The caller parses the document, so each refuses a file that is not OPML in its own terms, /// The caller parses the document, so each refuses a file that is not OPML in its own terms,
/// before anything is touched: a 400 from the web, a message from the CLI. /// before anything is touched: a 400 from the web, a message from the CLI.
pub fn subscribe_opml( pub async fn subscribe_opml(
ctx: &Ctx, ctx: &Ctx,
config_path: &std::path::Path, config_path: &std::path::Path,
doc: &opml::OPML, doc: &opml::OPML,
@@ -699,7 +711,7 @@ pub fn subscribe_opml(
let mut found = vec![]; let mut found = vec![];
collect_outlines(&doc.body.outlines, &mut found); collect_outlines(&doc.body.outlines, &mut found);
let known = subscriptions(ctx)?; let known = subscriptions(ctx).await?;
let mut cfg = (*ctx.cfg()).clone(); let mut cfg = (*ctx.cfg()).clone();
let mut ids = vec![]; let mut ids = vec![];
let mut grew = false; let mut grew = false;
@@ -745,10 +757,10 @@ pub fn subscribe_opml(
let (mut added, mut had) = (0, 0); let (mut added, mut had) = (0, 0);
for id in ids { for id in ids {
if ctx.db.subscription(user_id, &id)?.is_some() { if ctx.db.subscription(user_id, &id).await?.is_some() {
had += 1; had += 1;
} else { } else {
ctx.db.subscribe(user_id, &id)?; ctx.db.subscribe(user_id, &id).await?;
added += 1; added += 1;
} }
} }
@@ -766,7 +778,16 @@ pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, Strin
} }
} }
fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> { async fn copy_db(ctx: &Ctx, from: &std::path::Path) -> Result<()> {
anyhow::ensure!(from.exists(), "{} does not exist", from.display());
let source = db::Db::open(&from.display().to_string()).await?;
for (table, n) in ctx.db.copy_from(&source).await? {
println!("{table:14} {n}");
}
Ok(())
}
async fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
let mut doc = opml::OPML::default(); let mut doc = opml::OPML::default();
doc.head = Some(opml::Head { doc.head = Some(opml::Head {
title: Some("ipx subscriptions".into()), title: Some("ipx subscriptions".into()),
@@ -775,7 +796,7 @@ fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
for (id, feed) in &ctx.cfg().feeds { for (id, feed) in &ctx.cfg().feeds {
let title = ctx let title = ctx
.db .db
.feed_summary(id) .feed_summary(id).await
.ok() .ok()
.and_then(|s| s.title) .and_then(|s| s.title)
.unwrap_or_else(|| id.clone()); .unwrap_or_else(|| id.clone());
@@ -787,14 +808,14 @@ fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
Ok(()) Ok(())
} }
fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> { async fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
if cfg.feeds.is_empty() { if cfg.feeds.is_empty() {
println!("No feeds configured in {}", config_path.display()); println!("No feeds configured in {}", config_path.display());
return Ok(()); return Ok(());
} }
for (id, feed) in &cfg.feeds { for (id, feed) in &cfg.feeds {
let s = ctx.db.feed_summary(id)?; let s = ctx.db.feed_summary(id).await?;
println!("{id} {}", s.title.as_deref().unwrap_or("-")); println!("{id} {}", s.title.as_deref().unwrap_or("-"));
println!(" url {}", feed.url); println!(" url {}", feed.url);
println!(" last checked {}", ago(s.last_checked)); println!(" last checked {}", ago(s.last_checked));
@@ -809,8 +830,8 @@ fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
/// `standalone` false means this is the sweep that runs before a scan: it reports what it /// `standalone` false means this is the sweep that runs before a scan: it reports what it
/// deleted, but must not emit the terminal ReapDone, or a client waiting on its `fetch` /// deleted, but must not emit the terminal ReapDone, or a client waiting on its `fetch`
/// would stop reading before the scan had even started. /// would stop reading before the scan had even started.
fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> { async fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
let r = retention::run(&ctx.cfg(), &ctx.db, dry_run)?; let r = retention::run(&ctx.cfg(), &ctx.db, dry_run).await?;
for c in r.aged_out.iter().chain(r.over_quota.iter()) { for c in r.aged_out.iter().chain(r.over_quota.iter()) {
ctx.out.emit(Event::Reaped { ctx.out.emit(Event::Reaped {
path: c.path.clone(), path: c.path.clone(),
@@ -828,7 +849,7 @@ fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> { async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
let subs = subscriptions(ctx)?; let subs = subscriptions(ctx).await?;
if let Some(id) = only if let Some(id) = only
&& !subs.iter().any(|s| s.id == id) && !subs.iter().any(|s| s.id == id)
{ {
@@ -839,7 +860,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
let mut fresh: Vec<String> = vec![]; let mut fresh: Vec<String> = vec![];
for sub in subs.iter().filter(|s| only.is_none_or(|o| o == s.id)) { for sub in subs.iter().filter(|s| only.is_none_or(|o| o == s.id)) {
let (id, feed_cfg) = (&sub.id, &sub.cfg); let (id, feed_cfg) = (&sub.id, &sub.cfg);
let state = ctx.db.http_state(id)?; let state = ctx.db.http_state(id).await?;
if !force && let Some(last) = state.last_checked { if !force && let Some(last) = state.last_checked {
let due = last + due_after(&cfg, feed_cfg, state.ttl_mins) as i64; let due = last + due_after(&cfg, feed_cfg, state.ttl_mins) as i64;
@@ -886,20 +907,20 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
// One bad feed must not end the scan. // One bad feed must not end the scan.
let msg = format!("{e:#}"); let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() }); ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg)?; ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
} }
} }
} }
// Feeds a subscribed OPML just introduced: scan them now, in this run. // Feeds a subscribed OPML just introduced: scan them now, in this run.
if !fresh.is_empty() { if !fresh.is_empty() {
let subs = subscriptions(ctx)?; let subs = subscriptions(ctx).await?;
for id in &fresh { for id in &fresh {
let Some(feed_cfg) = subs.iter().find(|s| &s.id == id).map(|s| &s.cfg) else { let Some(feed_cfg) = subs.iter().find(|s| &s.id == id).map(|s| &s.cfg) else {
continue; continue;
}; };
scanned += 1; scanned += 1;
ctx.out.emit(Event::FeedStart { feed: id.clone() }); ctx.out.emit(Event::FeedStart { feed: id.clone() });
let state = ctx.db.http_state(id)?; let state = ctx.db.http_state(id).await?;
match scan_one(ctx, id, feed_cfg, &state).await { match scan_one(ctx, id, feed_cfg, &state).await {
Ok(Outcome::Feed(s)) => ctx.out.emit(Event::FeedDone { Ok(Outcome::Feed(s)) => ctx.out.emit(Event::FeedDone {
feed: id.clone(), feed: id.clone(),
@@ -912,7 +933,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
Err(e) => { Err(e) => {
let msg = format!("{e:#}"); let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() }); ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg)?; ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
} }
} }
} }
@@ -934,7 +955,7 @@ pub struct Sub {
/// ///
/// A derived feed borrows its parent's settings wholesale. That is why it needs no config /// A derived feed borrows its parent's settings wholesale. That is why it needs no config
/// entry -- there is nothing to store but its URL and where it came from. /// entry -- there is nothing to store but its URL and where it came from.
pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> { pub async fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
let mut out: Vec<Sub> = cfg let mut out: Vec<Sub> = cfg
.feeds .feeds
@@ -942,7 +963,7 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
.map(|(id, f)| Sub { id: id.clone(), cfg: f.clone(), managed: false }) .map(|(id, f)| Sub { id: id.clone(), cfg: f.clone(), managed: false })
.collect(); .collect();
for m in ctx.db.managed_feeds()? { for m in ctx.db.managed_feeds().await? {
if cfg.feeds.contains_key(&m.id) { if cfg.feeds.contains_key(&m.id) {
continue; // promoted to config at some point; that entry wins continue; // promoted to config at some point; that entry wins
} }
@@ -953,10 +974,10 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
// skip them here regardless so a row that slips through is never scanned. // skip them here regardless so a row that slips through is never scanned.
continue; continue;
} }
let base = parent let base = match parent.and_then(|p| p.folder.clone()) {
.and_then(|p| p.folder.clone()) Some(folder) => folder,
.or_else(|| ctx.db.feed_summary(&m.group_id).ok().and_then(|s| s.title)) None => ctx.db.feed_summary(&m.group_id).await.ok().and_then(|s| s.title).unwrap_or_else(|| m.group_id.clone()),
.unwrap_or_else(|| m.group_id.clone()); };
let title = m.title.clone().unwrap_or_else(|| m.id.clone()); let title = m.title.clone().unwrap_or_else(|| m.id.clone());
out.push(Sub { out.push(Sub {
id: m.id.clone(), id: m.id.clone(),
@@ -988,17 +1009,17 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
/// itself is removed, since `subscriptions()` would otherwise keep scanning them under a /// itself is removed, since `subscriptions()` would otherwise keep scanning them under a
/// fallback policy meant for a feed with no parent at all. A feed promoted to config is not /// fallback policy meant for a feed with no parent at all. A feed promoted to config is not
/// derived any more, so it is only unmanaged. /// derived any more, so it is only unmanaged.
pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> { pub async fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
for m in ctx.db.managed_feeds()?.into_iter().filter(|m| m.group_id == parent_id) { for m in ctx.db.managed_feeds().await?.into_iter().filter(|m| m.group_id == parent_id) {
if cfg.feeds.contains_key(&m.id) { if cfg.feeds.contains_key(&m.id) {
// Scanned from its config entry and still read. Dropped as derived, its stored // Scanned from its config entry and still read. Dropped as derived, its stored
// entries would go with it: davewiner's 11 were promoted without being unmanaged. // entries would go with it: davewiner's 11 were promoted without being unmanaged.
ctx.db.unmanage(&m.id)?; ctx.db.unmanage(&m.id).await?;
} else if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 { } else if ctx.db.downloaded_count(&m.id).await.unwrap_or(1) > 0 {
ctx.db.set_orphaned(&m.id, true)?; ctx.db.set_orphaned(&m.id, true).await?;
} else { } else {
ctx.db.drop_managed(&m.id)?; ctx.db.drop_managed(&m.id).await?;
} }
} }
Ok(()) Ok(())
@@ -1008,18 +1029,18 @@ pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
/// dropped or unmanaged. An OPML removed before `retire_group` existed left its feeds behind: /// dropped or unmanaged. An OPML removed before `retire_group` existed left its feeds behind:
/// davewiner's 922 were skipped by every scan and never cleared, and their stale errors were /// davewiner's 922 were skipped by every scan and never cleared, and their stale errors were
/// most of the ones stored. /// most of the ones stored.
fn retire_stranded(ctx: &Ctx) -> Result<usize> { async fn retire_stranded(ctx: &Ctx) -> Result<usize> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
let before = ctx.db.managed_feeds()?; let before = ctx.db.managed_feeds().await?;
let stranded: std::collections::BTreeSet<&str> = before let stranded: std::collections::BTreeSet<&str> = before
.iter() .iter()
.map(|m| m.group_id.as_str()) .map(|m| m.group_id.as_str())
.filter(|g| !cfg.feeds.contains_key(*g)) .filter(|g| !cfg.feeds.contains_key(*g))
.collect(); .collect();
for group in stranded { for group in stranded {
retire_group(ctx, group)?; retire_group(ctx, group).await?;
} }
Ok(before.len() - ctx.db.managed_feeds()?.len()) Ok(before.len() - ctx.db.managed_feeds().await?.len())
} }
/// Seconds to wait before re-checking a feed. /// Seconds to wait before re-checking a feed.
@@ -1064,19 +1085,19 @@ async fn scan_one(
if feed::is_patreon_creator(&feed_cfg.url) { if feed::is_patreon_creator(&feed_cfg.url) {
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await { match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
Ok((name, shows)) if shows.len() > 1 => { Ok((name, shows)) if shows.len() > 1 => {
ctx.db.touch_feed(id, &feed_cfg.url)?; ctx.db.touch_feed(id, &feed_cfg.url).await?;
if let Some(name) = name { if let Some(name) = name {
ctx.db.set_title(id, &name)?; ctx.db.set_title(id, &name).await?;
} }
// Read as one feed before it was split, it listed every show's items in one // Read as one feed before it was split, it listed every show's items in one
// heap. The items go; its files and read state move to each show as the show // heap. The items go; its files and read state move to each show as the show
// lists them (`Db::adopt`), so no show comes up empty for want of a URL. // lists them (`Db::adopt`), so no show comes up empty for want of a URL.
ctx.db.clear_entries(id)?; ctx.db.clear_entries(id).await?;
return sync_group(ctx, id, feed_cfg, &shows).await; return sync_group(ctx, id, feed_cfg, &shows).await;
} }
Ok(_) => {} // One show: the creator's feed is that show. Ok(_) => {} // One show: the creator's feed is that show.
// Already split: keep the shows it has rather than read the creator as one heap. // Already split: keep the shows it has rather than read the creator as one heap.
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e), Err(e) if ctx.db.managed_feeds().await?.iter().any(|m| m.group_id == id) => return Err(e),
Err(e) => tracing::warn!( Err(e) => tracing::warn!(
feed = id, feed = id,
error = %format!("{e:#}"), error = %format!("{e:#}"),
@@ -1097,29 +1118,29 @@ async fn scan_one(
// from backup, a manual edit, a cleanup that removed entries. Believe the database over // from backup, a manual edit, a cleanup that removed entries. Believe the database over
// the validator: drop it and ask again, or the feed stays empty until the publisher // the validator: drop it and ask again, or the feed stays empty until the publisher
// happens to change something. // happens to change something.
if matches!(fetched, feed::Fetched::NotModified) && ctx.db.feed_summary(id)?.entries == 0 { if matches!(fetched, feed::Fetched::NotModified) && ctx.db.feed_summary(id).await?.entries == 0 {
tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator"); tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator");
ctx.db.clear_validators(id)?; ctx.db.clear_validators(id).await?;
fetched = feed::fetch(&ctx.client, feed_cfg, None, None).await?; fetched = feed::fetch(&ctx.client, feed_cfg, None, None).await?;
} }
let (bytes, etag, last_modified) = match fetched { let (bytes, etag, last_modified) = match fetched {
feed::Fetched::NotModified => { feed::Fetched::NotModified => {
ctx.db.touch_feed(id, &feed_cfg.url)?; ctx.db.touch_feed(id, &feed_cfg.url).await?;
return Ok(Outcome::NotModified); return Ok(Outcome::NotModified);
} }
feed::Fetched::Body { bytes, etag, last_modified } => (bytes, etag, last_modified), feed::Fetched::Body { bytes, etag, last_modified } => (bytes, etag, last_modified),
}; };
if bytes.iter().all(u8::is_ascii_whitespace) { if bytes.iter().all(u8::is_ascii_whitespace) {
ctx.db.touch_feed(id, &feed_cfg.url)?; ctx.db.touch_feed(id, &feed_cfg.url).await?;
return Ok(Outcome::Empty); return Ok(Outcome::Empty);
} }
// A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml" // A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml"
// URL; sniffing the body also catches one served from a URL without that extension. // URL; sniffing the body also catches one served from a URL without that extension.
if feed::is_opml(&bytes) { if feed::is_opml(&bytes) {
ctx.db.touch_feed(id, &feed_cfg.url)?; ctx.db.touch_feed(id, &feed_cfg.url).await?;
return sync_opml(ctx, id, feed_cfg, &bytes).await; return sync_opml(ctx, id, feed_cfg, &bytes).await;
} }
@@ -1133,29 +1154,29 @@ async fn scan_one(
parsed.ttl_mins, parsed.ttl_mins,
parsed.image.as_deref(), parsed.image.as_deref(),
parsed.category.as_deref(), parsed.category.as_deref(),
)?; ).await?;
let policy = policy_for(ctx, id, feed_cfg)?; let policy = policy_for(ctx, id, feed_cfg).await?;
if let Some(parent) = &feed_cfg.group { if let Some(parent) = &feed_cfg.group {
let listed: Vec<(&str, &str)> = parsed let listed: Vec<(&str, &str)> = parsed
.entries .entries
.iter() .iter()
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str()))) .flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
.collect(); .collect();
ctx.db.adopt(parent, id, &listed)?; ctx.db.adopt(parent, id, &listed).await?;
} }
// Verdicts are recorded in `state`, so the download queue below is just "everything still // Verdicts are recorded in `state`, so the download queue below is just "everything still
// pending". A filter's verdict is looked at again on every scan, though: made once, at // pending". A filter's verdict is looked at again on every scan, though: made once, at
// discovery, it outlived the setting behind it, and allowing explicit items afterwards // discovery, it outlived the setting behind it, and allowing explicit items afterwards
// changed nothing however often the feed was scanned. // changed nothing however often the feed was scanned.
let skipped = ctx.db.skipped_by_filter(id)?; let skipped = ctx.db.skipped_by_filter(id).await?;
let mut scan = Scan::default(); let mut scan = Scan::default();
for entry in &parsed.entries { for entry in &parsed.entries {
if ctx.db.record_entry(id, entry)? { if ctx.db.record_entry(id, entry).await? {
scan.new_entries += 1; scan.new_entries += 1;
} }
for enc in &entry.enclosures { for enc in &entry.enclosures {
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? { let was = if ctx.db.record_enclosure(id, &entry.guid, enc).await? {
None None
} else if let Some(reason) = skipped.get(&enc.url) { } else if let Some(reason) = skipped.get(&enc.url) {
Some(reason.as_str()) Some(reason.as_str())
@@ -1165,8 +1186,8 @@ async fn scan_one(
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc); let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
if now != was { if now != was {
match now { match now {
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?, Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason)).await?,
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?, None => ctx.db.mark_enclosure(&enc.url, "pending", None).await?,
} }
} }
} }
@@ -1178,10 +1199,10 @@ async fn scan_one(
let folder = download::folder_for(&cfg, id, feed_cfg, parsed.title.as_deref()); let folder = download::folder_for(&cfg, id, feed_cfg, parsed.title.as_deref());
let dest_dir = cfg.general.download_dir.join(&folder); let dest_dir = cfg.general.download_dir.join(&folder);
for item in ctx.db.pending(id, budget)? { for item in ctx.db.pending(id, budget).await? {
if download::looks_like_torrent(&item.url, item.mime.as_deref()) { if download::looks_like_torrent(&item.url, item.mime.as_deref()) {
if !ctx.cfg().torrent.enabled { if !ctx.cfg().torrent.enabled {
ctx.db.mark_enclosure(&item.url, "skipped", Some("torrents disabled"))?; ctx.db.mark_enclosure(&item.url, "skipped", Some("torrents disabled")).await?;
ctx.out.emit(Event::TorrentDeferred { ctx.out.emit(Event::TorrentDeferred {
feed: id.to_string(), feed: id.to_string(),
url: item.url.clone(), url: item.url.clone(),
@@ -1191,14 +1212,14 @@ async fn scan_one(
} }
if ctx.detach_torrents { if ctx.detach_torrents {
// 'downloading' keeps the next scan from queueing it a second time. // 'downloading' keeps the next scan from queueing it a second time.
ctx.db.mark_enclosure(&item.url, "downloading", None)?; ctx.db.mark_enclosure(&item.url, "downloading", None).await?;
spawn_torrent(ctx, id.to_string(), item.id, item.url.clone(), dest_dir.clone()); spawn_torrent(ctx, id.to_string(), item.id, item.url.clone(), dest_dir.clone());
scan.torrents += 1; scan.torrents += 1;
continue; continue;
} }
match torrent_one(ctx, id, item.id, &item.url, &dest_dir).await { match torrent_one(ctx, id, item.id, &item.url, &dest_dir).await {
Ok((path, bytes)) => { Ok((path, bytes)) => {
ctx.db.mark_downloaded(&item.url, &path, bytes)?; ctx.db.mark_downloaded(&item.url, &path, bytes).await?;
ctx.out.emit(Event::DownloadDone { ctx.out.emit(Event::DownloadDone {
feed: id.to_string(), feed: id.to_string(),
enclosure: item.id, enclosure: item.id,
@@ -1216,7 +1237,7 @@ async fn scan_one(
url: item.url.clone(), url: item.url.clone(),
msg: msg.clone(), msg: msg.clone(),
}); });
ctx.db.mark_enclosure(&item.url, "error", Some(&msg))?; ctx.db.mark_enclosure(&item.url, "error", Some(&msg)).await?;
scan.failed += 1; scan.failed += 1;
} }
} }
@@ -1241,7 +1262,7 @@ async fn scan_one(
url: item.url.clone(), url: item.url.clone(),
msg: msg.clone(), msg: msg.clone(),
}); });
ctx.db.mark_enclosure(&item.url, "error", Some(&msg))?; ctx.db.mark_enclosure(&item.url, "error", Some(&msg)).await?;
scan.failed += 1; scan.failed += 1;
} }
} }
@@ -1259,7 +1280,7 @@ async fn sync_opml(
) -> Result<Outcome> { ) -> Result<Outcome> {
let listed = feed::parse_opml(bytes)?; let listed = feed::parse_opml(bytes)?;
if let Some(title) = feed::opml_title(bytes) { if let Some(title) = feed::opml_title(bytes) {
ctx.db.set_title(parent_id, &title)?; ctx.db.set_title(parent_id, &title).await?;
} }
sync_group(ctx, parent_id, parent, &listed).await sync_group(ctx, parent_id, parent, &listed).await
} }
@@ -1278,13 +1299,13 @@ async fn sync_group(
listed: &[(String, String)], listed: &[(String, String)],
) -> Result<Outcome> { ) -> Result<Outcome> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
let existing = ctx.db.managed_feeds()?; let existing = ctx.db.managed_feeds().await?;
let mut added = vec![]; let mut added = vec![];
for (title, url) in listed { for (title, url) in listed {
// Already known, whether derived or promoted into the config. // Already known, whether derived or promoted into the config.
if let Some(m) = existing.iter().find(|m| &m.url == url) { if let Some(m) = existing.iter().find(|m| &m.url == url) {
ctx.db.upsert_managed(&m.id, url, title, parent_id)?; ctx.db.upsert_managed(&m.id, url, title, parent_id).await?;
continue; continue;
} }
// A Patreon show you added by hand may be spelled differently from the one listed. // A Patreon show you added by hand may be spelled differently from the one listed.
@@ -1292,7 +1313,7 @@ async fn sync_group(
continue; continue;
} }
// A removed feed keeps its rows, so its id is only free again for the same feed. // A removed feed keeps its rows, so its id is only free again for the same feed.
let known = ctx.db.feed_urls()?; let known = ctx.db.feed_urls().await?;
let taken: std::collections::BTreeMap<String, config::Feed> = cfg let taken: std::collections::BTreeMap<String, config::Feed> = cfg
.feeds .feeds
.keys() .keys()
@@ -1302,7 +1323,7 @@ async fn sync_group(
.map(|id| (id.clone(), parent.clone())) .map(|id| (id.clone(), parent.clone()))
.collect(); .collect();
let id = config::unique_slug(title, &taken); let id = config::unique_slug(title, &taken);
ctx.db.upsert_managed(&id, url, title, parent_id)?; ctx.db.upsert_managed(&id, url, title, parent_id).await?;
added.push(id); added.push(id);
} }
@@ -1310,15 +1331,15 @@ async fn sync_group(
// subscription means. Their own feeds are untouched. // subscription means. Their own feeds are untouched.
for id in ctx for id in ctx
.db .db
.managed_feeds()? .managed_feeds().await?
.iter() .iter()
.filter(|m| m.group_id == parent_id) .filter(|m| m.group_id == parent_id)
.map(|m| m.id.clone()) .map(|m| m.id.clone())
.chain(std::iter::once(parent_id.to_string())) .chain(std::iter::once(parent_id.to_string()))
{ {
for user in ctx.db.users()? { for user in ctx.db.users().await? {
if ctx.db.subscription(user.id, parent_id)?.is_some() { if ctx.db.subscription(user.id, parent_id).await?.is_some() {
ctx.db.subscribe(user.id, &id)?; ctx.db.subscribe(user.id, &id).await?;
} }
} }
} }
@@ -1330,13 +1351,13 @@ async fn sync_group(
if listed.iter().any(|(_, u)| u == &m.url) { if listed.iter().any(|(_, u)| u == &m.url) {
continue; continue;
} }
if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 { if ctx.db.downloaded_count(&m.id).await.unwrap_or(1) > 0 {
// Never orphan a downloaded file: keep the feed and say why in the UI. // Never orphan a downloaded file: keep the feed and say why in the UI.
ctx.db.set_orphaned(&m.id, true)?; ctx.db.set_orphaned(&m.id, true).await?;
kept += 1; kept += 1;
tracing::info!(feed = %m.id, "dropped from the OPML but has downloads; keeping it"); tracing::info!(feed = %m.id, "dropped from the OPML but has downloads; keeping it");
} else { } else {
ctx.db.drop_managed(&m.id)?; ctx.db.drop_managed(&m.id).await?;
removed += 1; removed += 1;
tracing::info!(feed = %m.id, "dropped from the OPML with nothing downloaded; removed"); tracing::info!(feed = %m.id, "dropped from the OPML with nothing downloaded; removed");
} }
@@ -1403,9 +1424,9 @@ pub struct Policy {
pub budget: usize, pub budget: usize,
} }
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> { async fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
let global = ctx.cfg().general.max_new_per_check; let global = ctx.cfg().general.max_new_per_check;
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global)) Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref()).await?, feed_cfg, global))
} }
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy { fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
@@ -1484,14 +1505,14 @@ async fn fetch_one(
// as if it were an episode. // as if it were an episode.
let _ = tokio::fs::remove_file(&got.tmp).await; let _ = tokio::fs::remove_file(&got.tmp).await;
if !ctx.cfg().torrent.enabled { if !ctx.cfg().torrent.enabled {
ctx.db.mark_enclosure(url, "skipped", Some("torrents disabled"))?; ctx.db.mark_enclosure(url, "skipped", Some("torrents disabled")).await?;
anyhow::bail!("body is a torrent and torrents are disabled"); anyhow::bail!("body is a torrent and torrents are disabled");
} }
return torrent_one(ctx, feed_id, enclosure, url, dest_dir).await; return torrent_one(ctx, feed_id, enclosure, url, dest_dir).await;
} }
let path = download::place(&got, dest_dir).await?; let path = download::place(&got, dest_dir).await?;
ctx.db.mark_downloaded(url, &path, got.bytes)?; ctx.db.mark_downloaded(url, &path, got.bytes).await?;
Ok((path, got.bytes)) Ok((path, got.bytes))
} }
@@ -1509,7 +1530,7 @@ fn spawn_torrent(ctx: &Arc<Ctx>, feed_id: String, enclosure: i64, url: String, d
let db = &ctx.db; let db = &ctx.db;
match outcome { match outcome {
Ok((path, bytes)) => { Ok((path, bytes)) => {
if let Err(e) = db.mark_downloaded(&url, &path, bytes) { if let Err(e) = db.mark_downloaded(&url, &path, bytes).await {
tracing::warn!(error = ?e, "could not record the finished torrent"); tracing::warn!(error = ?e, "could not record the finished torrent");
} }
ctx.out.emit(Event::DownloadDone { ctx.out.emit(Event::DownloadDone {
@@ -1522,7 +1543,7 @@ fn spawn_torrent(ctx: &Arc<Ctx>, feed_id: String, enclosure: i64, url: String, d
} }
Err(e) => { Err(e) => {
let msg = format!("{e:#}"); let msg = format!("{e:#}");
let _ = db.mark_enclosure(&url, "error", Some(&msg)); let _ = db.mark_enclosure(&url, "error", Some(&msg)).await;
ctx.out.emit(Event::DownloadError { feed: feed_id, enclosure, url, msg }); ctx.out.emit(Event::DownloadError { feed: feed_id, enclosure, url, msg });
} }
} }
@@ -1535,14 +1556,14 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
let enc = ctx let enc = ctx
.db .db
.enclosure(id)? .enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?; .ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
if enc.path.is_some() { if enc.path.is_some() {
return Ok(()); // Already here. return Ok(()); // Already here.
} }
// Must look through the derived feeds too: anything inside an OPML subscription has // Must look through the derived feeds too: anything inside an OPML subscription has
// no config entry, so a config-only lookup called every one of them "unsubscribed". // no config entry, so a config-only lookup called every one of them "unsubscribed".
let subs = subscriptions(ctx)?; let subs = subscriptions(ctx).await?;
let feed_cfg = subs let feed_cfg = subs
.iter() .iter()
.find(|s| s.id == enc.feed_id) .find(|s| s.id == enc.feed_id)
@@ -1550,14 +1571,14 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
.ok_or_else(|| anyhow::anyhow!("enclosure {id} belongs to unsubscribed feed {:?}", enc.feed_id))?; .ok_or_else(|| anyhow::anyhow!("enclosure {id} belongs to unsubscribed feed {:?}", enc.feed_id))?;
let feed_cfg = &feed_cfg; let feed_cfg = &feed_cfg;
let title = ctx.db.feed_summary(&enc.feed_id)?.title; let title = ctx.db.feed_summary(&enc.feed_id).await?.title;
let folder = download::folder_for(&cfg, &enc.feed_id, feed_cfg, title.as_deref()); let folder = download::folder_for(&cfg, &enc.feed_id, feed_cfg, title.as_deref());
let dest_dir = cfg.general.download_dir.join(&folder); let dest_dir = cfg.general.download_dir.join(&folder);
ctx.out.emit(Event::FeedStart { feed: enc.feed_id.clone() }); ctx.out.emit(Event::FeedStart { feed: enc.feed_id.clone() });
let is_torrent = download::looks_like_torrent(&enc.url, enc.mime.as_deref()); let is_torrent = download::looks_like_torrent(&enc.url, enc.mime.as_deref());
if is_torrent && cfg.torrent.enabled && ctx.detach_torrents { if is_torrent && cfg.torrent.enabled && ctx.detach_torrents {
ctx.db.mark_enclosure(&enc.url, "downloading", None)?; ctx.db.mark_enclosure(&enc.url, "downloading", None).await?;
spawn_torrent(ctx, enc.feed_id.clone(), enc.id, enc.url.clone(), dest_dir); spawn_torrent(ctx, enc.feed_id.clone(), enc.id, enc.url.clone(), dest_dir);
return Ok(()); return Ok(());
} }
@@ -1573,7 +1594,7 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
match result { match result {
Ok((path, bytes)) => { Ok((path, bytes)) => {
ctx.db.mark_downloaded(&enc.url, &path, bytes)?; ctx.db.mark_downloaded(&enc.url, &path, bytes).await?;
ctx.out.emit(Event::DownloadDone { ctx.out.emit(Event::DownloadDone {
feed: enc.feed_id.clone(), feed: enc.feed_id.clone(),
enclosure: enc.id, enclosure: enc.id,
@@ -1584,7 +1605,7 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
} }
Err(e) => { Err(e) => {
let msg = format!("{e:#}"); let msg = format!("{e:#}");
ctx.db.mark_enclosure(&enc.url, "error", Some(&msg))?; ctx.db.mark_enclosure(&enc.url, "error", Some(&msg)).await?;
ctx.out.emit(Event::DownloadError { ctx.out.emit(Event::DownloadError {
feed: enc.feed_id.clone(), feed: enc.feed_id.clone(),
enclosure: enc.id, enclosure: enc.id,
@@ -1690,8 +1711,8 @@ mod tests {
} }
} }
#[test] #[tokio::test]
fn a_shared_feed_is_fetched_for_whoever_wants_the_most() { async fn a_shared_feed_is_fetched_for_whoever_wants_the_most() {
// Nobody subscribed: the feed's own settings stand, as in a single-user install. // Nobody subscribed: the feed's own settings stand, as in a single-user install.
let p = merge_policy(&[], &feed(), 3); let p = merge_policy(&[], &feed(), 3);
assert!(p.auto_download); assert!(p.auto_download);
@@ -1721,10 +1742,10 @@ mod tests {
assert!(p.auto_download); assert!(p.auto_download);
} }
fn test_ctx(cfg: config::Config) -> Ctx { async fn test_ctx(cfg: config::Config) -> Ctx {
Ctx { Ctx {
cfg: std::sync::RwLock::new(Arc::new(cfg)), cfg: std::sync::RwLock::new(Arc::new(cfg)),
db: db::Db::memory().unwrap(), db: db::Db::memory().await.unwrap(),
client: reqwest::Client::new(), client: reqwest::Client::new(),
out: Emitter::terminal(), out: Emitter::terminal(),
torrents: tokio::sync::OnceCell::new(), torrents: tokio::sync::OnceCell::new(),
@@ -1734,52 +1755,52 @@ mod tests {
} }
} }
#[test] #[tokio::test]
fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() { async fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() {
// davewiner: the OPML subscription left config.toml, but its 922 derived rows // davewiner: the OPML subscription left config.toml, but its 922 derived rows
// stayed in the database and kept being scanned under the no-parent fallback. // stayed in the database and kept being scanned under the no-parent fallback.
let ctx = test_ctx(config::Config::default()); let ctx = test_ctx(config::Config::default()).await;
ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").unwrap(); ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").await.unwrap();
assert!( assert!(
subscriptions(&ctx).unwrap().iter().all(|s| s.id != "child"), subscriptions(&ctx).await.unwrap().iter().all(|s| s.id != "child"),
"a derived feed whose parent is gone from config must not be scanned" "a derived feed whose parent is gone from config must not be scanned"
); );
} }
#[test] #[tokio::test]
fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() { async fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() {
let ctx = test_ctx(config::Config::default()); let ctx = test_ctx(config::Config::default()).await;
ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").unwrap(); ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").await.unwrap();
ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").unwrap(); ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").await.unwrap();
let enc = feed::Enclosure { url: "http://x/ep.mp3".into(), mime: None, length: None }; let enc = feed::Enclosure { url: "http://x/ep.mp3".into(), mime: None, length: None };
ctx.db.record_enclosure("has-file", "g1", &enc).unwrap(); ctx.db.record_enclosure("has-file", "g1", &enc).await.unwrap();
ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).unwrap(); ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).await.unwrap();
retire_group(&ctx, "parent").unwrap(); retire_group(&ctx, "parent").await.unwrap();
let managed = ctx.db.managed_feeds().unwrap(); let managed = ctx.db.managed_feeds().await.unwrap();
assert!(!managed.iter().any(|m| m.id == "empty"), "nothing downloaded, so it is forgotten"); assert!(!managed.iter().any(|m| m.id == "empty"), "nothing downloaded, so it is forgotten");
assert!(managed.iter().any(|m| m.id == "has-file"), "has a file on disk, so it is kept"); assert!(managed.iter().any(|m| m.id == "has-file"), "has a file on disk, so it is kept");
assert!(ctx.db.feed_summary("has-file").unwrap().orphaned, "and flagged as orphaned"); assert!(ctx.db.feed_summary("has-file").await.unwrap().orphaned, "and flagged as orphaned");
} }
#[test] #[tokio::test]
fn a_stranded_group_is_retired_but_a_promoted_feed_keeps_its_entries() { async fn a_stranded_group_is_retired_but_a_promoted_feed_keeps_its_entries() {
// davewiner: the OPML left config before retire_group existed, and 11 of its feeds // davewiner: the OPML left config before retire_group existed, and 11 of its feeds
// promoted to config since still said managed = 1. // promoted to config since still said managed = 1.
let mut cfg = config::Config::default(); let mut cfg = config::Config::default();
cfg.feeds.insert("promoted".into(), feed()); cfg.feeds.insert("promoted".into(), feed());
cfg.feeds.insert("live-opml".into(), feed()); cfg.feeds.insert("live-opml".into(), feed());
let ctx = test_ctx(cfg); let ctx = test_ctx(cfg).await;
ctx.db.upsert_managed("promoted", "http://x/p.xml", "Promoted", "gone-opml").unwrap(); ctx.db.upsert_managed("promoted", "http://x/p.xml", "Promoted", "gone-opml").await.unwrap();
ctx.db.upsert_managed("empty", "http://x/e.xml", "Empty", "gone-opml").unwrap(); ctx.db.upsert_managed("empty", "http://x/e.xml", "Empty", "gone-opml").await.unwrap();
ctx.db.upsert_managed("listed", "http://x/l.xml", "Listed", "live-opml").unwrap(); ctx.db.upsert_managed("listed", "http://x/l.xml", "Listed", "live-opml").await.unwrap();
ctx.db.record_entry("promoted", &feed::Entry { guid: "g1".into(), ..Default::default() }).unwrap(); ctx.db.record_entry("promoted", &feed::Entry { guid: "g1".into(), ..Default::default() }).await.unwrap();
assert_eq!(retire_stranded(&ctx).unwrap(), 2, "empty dropped, promoted unmanaged"); assert_eq!(retire_stranded(&ctx).await.unwrap(), 2, "empty dropped, promoted unmanaged");
let managed: Vec<String> = ctx.db.managed_feeds().unwrap().into_iter().map(|m| m.id).collect(); let managed: Vec<String> = ctx.db.managed_feeds().await.unwrap().into_iter().map(|m| m.id).collect();
assert_eq!(managed, ["listed"], "a group still in config is left alone"); assert_eq!(managed, ["listed"], "a group still in config is left alone");
assert_eq!(ctx.db.feed_summary("promoted").unwrap().entries, 1, "its entries survive"); assert_eq!(ctx.db.feed_summary("promoted").await.unwrap().entries, 1, "its entries survive");
} }
} }

View File

@@ -45,28 +45,28 @@ pub fn aged(candidates: &[Candidate], cutoff: i64) -> Vec<Candidate> {
.collect() .collect()
} }
pub fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> { pub async fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
let mut report = Report::default(); let mut report = Report::default();
// Someone may have deleted a file by hand; the row must stop claiming it exists. // Someone may have deleted a file by hand; the row must stop claiming it exists.
for (id, path) in db.missing_files()? { for (id, path) in db.missing_files().await? {
if !dry_run { if !dry_run {
db.mark_reaped(id)?; db.mark_reaped(id).await?;
} }
tracing::debug!(path, "file gone, row reaped"); tracing::debug!(path, "file gone, row reaped");
report.reconciled += 1; report.reconciled += 1;
} }
let candidates = db.reap_candidates()?; let candidates = db.reap_candidates().await?;
if cfg.general.max_age_days > 0 { if cfg.general.max_age_days > 0 {
let cutoff = now() - (cfg.general.max_age_days * 86_400) as i64; let cutoff = now() - (cfg.general.max_age_days * 86_400) as i64;
report.aged_out = aged(&candidates, cutoff); report.aged_out = aged(&candidates, cutoff);
for c in &report.aged_out { for c in &report.aged_out {
report.bytes_freed += remove(db, c, dry_run)?; report.bytes_freed += remove(db, c, dry_run).await?;
} }
if !dry_run { if !dry_run {
report.entries_pruned = db.prune_entries(cutoff)?; report.entries_pruned = db.prune_entries(cutoff).await?;
} }
} }
@@ -81,14 +81,14 @@ pub fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
let total: u64 = remaining.iter().map(|c| c.bytes.max(0) as u64).sum(); let total: u64 = remaining.iter().map(|c| c.bytes.max(0) as u64).sum();
report.over_quota = pick(&remaining, total, limit); report.over_quota = pick(&remaining, total, limit);
for c in &report.over_quota { for c in &report.over_quota {
report.bytes_freed += remove(db, c, dry_run)?; report.bytes_freed += remove(db, c, dry_run).await?;
} }
} }
Ok(report) Ok(report)
} }
fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> { async fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
if dry_run { if dry_run {
return Ok(c.bytes.max(0) as u64); return Ok(c.bytes.max(0) as u64);
} }
@@ -100,7 +100,7 @@ fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
tracing::warn!(path = c.path, error = %e, "could not delete"); tracing::warn!(path = c.path, error = %e, "could not delete");
return Ok(0); return Ok(0);
} }
db.mark_reaped(c.id)?; db.mark_reaped(c.id).await?;
Ok(size) Ok(size)
} }
@@ -149,12 +149,12 @@ mod tests {
// age_key 0 means "never recorded" -- not the same as "infinitely old". // age_key 0 means "never recorded" -- not the same as "infinitely old".
} }
#[test] #[tokio::test]
fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() { async fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
// One file serves both subscribers, so it takes both of them to release it. // One file serves both subscribers, so it takes both of them to release it.
let db = Db::memory().unwrap(); let db = Db::memory().await.unwrap();
db.exec_for_test( db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0); "INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true),(2,'sam',false);
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f'); INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
INSERT INTO entries (feed_id, guid, first_seen) VALUES INSERT INTO entries (feed_id, guid, first_seen) VALUES
('f', 'keep', 0), ('f', 'keep', 0),
@@ -163,20 +163,20 @@ mod tests {
('f', 'read', 0); ('f', 'read', 0);
-- Starred by one of the two, so it stays whatever the other thinks. -- Starred by one of the two, so it stays whatever the other thinks.
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
(1, 'f', 'keep', 1, 1), (1, 'f', 'keep', true, true),
(2, 'f', 'keep', 1, 0), (2, 'f', 'keep', true, false),
(1, 'f', 'half', 1, 0), (1, 'f', 'half', true, false),
(1, 'f', 'read', 1, 0), (1, 'f', 'read', true, false),
(2, 'f', 'read', 1, 0); (2, 'f', 'read', true, false);
INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES
(1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10), (1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10),
(2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20), (2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20),
(3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30), (3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30),
(4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);", (4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);",
) ).await
.unwrap(); .unwrap();
let got: Vec<i64> = db.reap_candidates().unwrap().iter().map(|c| c.id).collect(); let got: Vec<i64> = db.reap_candidates().await.unwrap().iter().map(|c| c.id).collect();
assert_eq!( assert_eq!(
got, got,
vec![4, 2, 3], vec![4, 2, 3],
@@ -185,12 +185,12 @@ mod tests {
); );
} }
#[test] #[tokio::test]
fn prune_keeps_entries_that_still_have_a_file() { async fn prune_keeps_entries_that_still_have_a_file() {
let db = Db::memory().unwrap(); let db = Db::memory().await.unwrap();
db.exec_for_test( db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1); "INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true);
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1); INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',true);
INSERT INTO entries (feed_id, guid, first_seen) VALUES INSERT INTO entries (feed_id, guid, first_seen) VALUES
('f', 'has-file', 100), ('f', 'has-file', 100),
('f', 'no-file', 100), ('f', 'no-file', 100),
@@ -198,9 +198,9 @@ mod tests {
('f', 'recent', 900); ('f', 'recent', 900);
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
(1, 'f', 'has-file', 'u1', '/tmp/x', 'done');", (1, 'f', 'has-file', 'u1', '/tmp/x', 'done');",
) ).await
.unwrap(); .unwrap();
assert_eq!(db.prune_entries(500).unwrap(), 1, "only the old, fileless, unflagged one"); assert_eq!(db.prune_entries(500).await.unwrap(), 1, "only the old, fileless, unflagged one");
} }
} }

View File

@@ -107,16 +107,16 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
let mut user = None; let mut user = None;
if let Some(name) = vouched { if let Some(name) = vouched {
user = match state.ctx.db.user_by_name(&name) { user = match state.ctx.db.user_by_name(&name).await {
Ok(Some(u)) => Some(u), Ok(Some(u)) => Some(u),
Ok(None) if cfg.web.auto_create_users => { Ok(None) if cfg.web.auto_create_users => {
tracing::info!(user = %name, "creating an account for a name the proxy vouched for"); tracing::info!(user = %name, "creating an account for a name the proxy vouched for");
state // The first account made is the admin.
.ctx let first = state.ctx.db.users().await.map(|u| u.is_empty()).unwrap_or(false);
.db match state.ctx.db.create_user(&name, None, first).await {
.create_user(&name, None, state.ctx.db.users().map(|u| u.is_empty()).unwrap_or(false)) Ok(id) => state.ctx.db.user_by_id(id).await.ok().flatten(),
.ok() Err(_) => None,
.and_then(|id| state.ctx.db.user_by_id(id).ok().flatten()) }
} }
Ok(None) => { Ok(None) => {
tracing::warn!(user = %name, "proxy vouched for an unknown name and auto_create_users is off"); tracing::warn!(user = %name, "proxy vouched for an unknown name and auto_create_users is off");
@@ -130,7 +130,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
// Every request comes vouched for; signed_in keeps one an hour. Failing to note the time // Every request comes vouched for; signed_in keeps one an hour. Failing to note the time
// must not turn anyone away, so its error goes unanswered. // must not turn anyone away, so its error goes unanswered.
if let Some(u) = &user { if let Some(u) = &user {
let _ = state.ctx.db.signed_in(u.id); let _ = state.ctx.db.signed_in(u.id).await;
} }
} }
let by_proxy = user.is_some(); let by_proxy = user.is_some();
@@ -141,7 +141,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
user = state user = state
.ctx .ctx
.db .db
.session_user(&sid, cfg.web.session_days.max(1) * 86_400) .session_user(&sid, cfg.web.session_days.max(1) * 86_400).await
.unwrap_or(None); .unwrap_or(None);
} }
} }
@@ -155,11 +155,11 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
if user.is_none() && !token.is_empty() { if user.is_none() && !token.is_empty() {
let supplied = from_query.clone().or_else(|| cookie(&req, COOKIE)); let supplied = from_query.clone().or_else(|| cookie(&req, COOKIE));
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) { if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
user = admin_user(&state); user = admin_user(&state).await;
if from_query.is_some() { if from_query.is_some() {
// The token link is a sign-in; the cookie it leaves behind is not one each time. // The token link is a sign-in; the cookie it leaves behind is not one each time.
if let Some(u) = &user { if let Some(u) = &user {
let _ = state.ctx.db.signed_in(u.id); let _ = state.ctx.db.signed_in(u.id).await;
} }
set_cookie = Some(format!( set_cookie = Some(format!(
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000" "{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
@@ -247,8 +247,8 @@ fn cookie(req: &Request, name: &str) -> Option<String> {
} }
/// The account the shared token stands for: the first admin, or the first user at all. /// The account the shared token stands for: the first admin, or the first user at all.
fn admin_user(state: &WebState) -> Option<crate::db::User> { async fn admin_user(state: &WebState) -> Option<crate::db::User> {
let users = state.ctx.db.users().ok()?; let users = state.ctx.db.users().await.ok()?;
users users
.iter() .iter()
.find(|u| u.is_admin) .find(|u| u.is_admin)
@@ -267,7 +267,7 @@ async fn login(
Json(body): Json<Credentials>, Json(body): Json<Credentials>,
) -> Result<Response, ApiError> { ) -> Result<Response, ApiError> {
let name = body.name.trim().to_ascii_lowercase(); let name = body.name.trim().to_ascii_lowercase();
let user = state.ctx.db.user_by_name(&name)?; let user = state.ctx.db.user_by_name(&name).await?;
// The same answer either way: whether a name exists is not something to leak. // The same answer either way: whether a name exists is not something to leak.
let ok = user let ok = user
.as_ref() .as_ref()
@@ -280,8 +280,8 @@ async fn login(
let user = user.expect("verified above"); let user = user.expect("verified above");
let token = crate::auth::new_session_token(); let token = crate::auth::new_session_token();
state.ctx.db.create_session(user.id, &token)?; state.ctx.db.create_session(user.id, &token).await?;
state.ctx.db.signed_in(user.id)?; state.ctx.db.signed_in(user.id).await?;
tracing::info!(user = %user.name, "signed in"); tracing::info!(user = %user.name, "signed in");
let days = state.ctx.cfg().web.session_days.max(1); let days = state.ctx.cfg().web.session_days.max(1);
@@ -298,7 +298,7 @@ async fn login(
async fn logout(State(state): State<WebState>, req: Request) -> Response { async fn logout(State(state): State<WebState>, req: Request) -> Response {
if let Some(sid) = cookie(&req, SESSION_COOKIE) { if let Some(sid) = cookie(&req, SESSION_COOKIE) {
let _ = state.ctx.db.delete_session(&sid); let _ = state.ctx.db.delete_session(&sid).await;
} }
let mut resp = StatusCode::NO_CONTENT.into_response(); let mut resp = StatusCode::NO_CONTENT.into_response();
for c in [ for c in [
@@ -320,7 +320,7 @@ async fn me(
) -> Json<serde_json::Value> { ) -> Json<serde_json::Value> {
let url = state.ctx.cfg().web.sign_out_url.clone(); let url = state.ctx.cfg().web.sign_out_url.clone();
let sign_out = (by_proxy && !url.is_empty()).then_some(url); let sign_out = (by_proxy && !url.is_empty()).then_some(url);
let (theme, mode) = state.ctx.db.theme(user.id).unwrap_or_default(); let (theme, mode) = state.ctx.db.theme(user.id).await.unwrap_or_default();
Json(serde_json::json!({ Json(serde_json::json!({
"name": user.name, "admin": user.is_admin, "sign_out": sign_out, "theme": theme, "mode": mode, "name": user.name, "admin": user.is_admin, "sign_out": sign_out, "theme": theme, "mode": mode,
})) }))
@@ -343,7 +343,7 @@ async fn patch_me(
if !theme_ok(&body.theme, &body.mode) { if !theme_ok(&body.theme, &body.mode) {
return Err(ApiError::bad_request("not a theme")); return Err(ApiError::bad_request("not a theme"));
} }
state.ctx.db.set_theme(user.id, &body.theme, &body.mode)?; state.ctx.db.set_theme(user.id, &body.theme, &body.mode).await?;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -378,7 +378,7 @@ async fn list_users(
let users: Vec<_> = state let users: Vec<_> = state
.ctx .ctx
.db .db
.users()? .users().await?
.iter() .iter()
.map(|u| { .map(|u| {
serde_json::json!({ serde_json::json!({
@@ -409,7 +409,7 @@ async fn add_user(
let name = crate::auth::name_from_header(&body.name).ok_or_else(|| { let name = crate::auth::name_from_header(&body.name).ok_or_else(|| {
ApiError::bad_request("a name is required, without commas, semicolons or line breaks") ApiError::bad_request("a name is required, without commas, semicolons or line breaks")
})?; })?;
if state.ctx.db.user_by_name(&name)?.is_some() { if state.ctx.db.user_by_name(&name).await?.is_some() {
return Err(ApiError::bad_request(format!("{name} already exists"))); return Err(ApiError::bad_request(format!("{name} already exists")));
} }
// No password is someone the proxy signs in, as with `ipx user add --no-password`. // No password is someone the proxy signs in, as with `ipx user add --no-password`.
@@ -418,7 +418,7 @@ async fn add_user(
} else { } else {
Some(crate::auth::hash_password(&body.password).map_err(|e| ApiError::bad_request(format!("{e:#}")))?) Some(crate::auth::hash_password(&body.password).map_err(|e| ApiError::bad_request(format!("{e:#}")))?)
}; };
state.ctx.db.create_user(&name, hash.as_deref(), body.admin)?; state.ctx.db.create_user(&name, hash.as_deref(), body.admin).await?;
tracing::info!(by = %user.name, user = %name, admin = body.admin, "account added"); tracing::info!(by = %user.name, user = %name, admin = body.admin, "account added");
Ok(StatusCode::CREATED) Ok(StatusCode::CREATED)
} }
@@ -435,7 +435,7 @@ async fn patch_user(
Json(body): Json<UserPatch>, Json(body): Json<UserPatch>,
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
require_admin(&user)?; require_admin(&user)?;
let users = state.ctx.db.users()?; let users = state.ctx.db.users().await?;
let target = users let target = users
.iter() .iter()
.find(|u| u.id == id) .find(|u| u.id == id)
@@ -446,7 +446,7 @@ async fn patch_user(
target.name target.name
))); )));
} }
state.ctx.db.set_admin(id, body.admin)?; state.ctx.db.set_admin(id, body.admin).await?;
tracing::info!(by = %user.name, user = %target.name, admin = body.admin, "admin changed"); tracing::info!(by = %user.name, user = %target.name, admin = body.admin, "admin changed");
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -457,7 +457,7 @@ async fn remove_user(
Path(id): Path<i64>, Path(id): Path<i64>,
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
require_admin(&user)?; require_admin(&user)?;
let users = state.ctx.db.users()?; let users = state.ctx.db.users().await?;
let target = users let target = users
.iter() .iter()
.find(|u| u.id == id) .find(|u| u.id == id)
@@ -468,7 +468,7 @@ async fn remove_user(
target.name target.name
))); )));
} }
state.ctx.db.delete_user(id)?; state.ctx.db.delete_user(id).await?;
tracing::info!(by = %user.name, user = %target.name, "account removed"); tracing::info!(by = %user.name, user = %target.name, "account removed");
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -510,7 +510,7 @@ async fn admin_page(State(state): State<WebState>, user: crate::db::User) -> Res
if !user.is_admin { if !user.is_admin {
return Redirect::to("/").into_response(); return Redirect::to("/").into_response();
} }
let theme = state.ctx.db.theme(user.id).unwrap_or_default(); let theme = state.ctx.db.theme(user.id).await.unwrap_or_default();
let page = with_theme(include_str!(concat!(env!("OUT_DIR"), "/admin.html")), theme); let page = with_theme(include_str!(concat!(env!("OUT_DIR"), "/admin.html")), theme);
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page)).into_response() ([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page)).into_response()
} }
@@ -584,7 +584,7 @@ const ADMIN_LINK: &str = "<a id=admin ";
/// The page, with the log button left out for anyone but an admin. Hiding it from the page's /// The page, with the log button left out for anyone but an admin. Hiding it from the page's
/// script instead showed it for a moment on every load, until /api/me answered. /// script instead showed it for a moment on every load, until /api/me answered.
async fn index(State(state): State<WebState>, user: crate::db::User) -> impl IntoResponse { async fn index(State(state): State<WebState>, user: crate::db::User) -> impl IntoResponse {
let theme = state.ctx.db.theme(user.id).unwrap_or_default(); let theme = state.ctx.db.theme(user.id).await.unwrap_or_default();
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page_for(user.is_admin, theme))) ([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page_for(user.is_admin, theme)))
} }
@@ -672,16 +672,16 @@ async fn feeds(
let cfg = state.ctx.cfg(); let cfg = state.ctx.cfg();
// Config entries plus the feeds derived from OPML subscriptions -- the catalogue. // Config entries plus the feeds derived from OPML subscriptions -- the catalogue.
// What comes back is only the part of it this person subscribes to. // What comes back is only the part of it this person subscribes to.
let subs = crate::subscriptions(&state.ctx)?; let subs = crate::subscriptions(&state.ctx).await?;
let mine: std::collections::HashMap<String, crate::db::Sub> = state let mine: std::collections::HashMap<String, crate::db::Sub> = state
.ctx .ctx
.db .db
.subscriptions_for(user.id)? .subscriptions_for(user.id).await?
.into_iter() .into_iter()
.map(|s| (s.feed_id.clone(), s)) .map(|s| (s.feed_id.clone(), s))
.collect(); .collect();
let counts = state.ctx.db.subscriber_counts()?; let counts = state.ctx.db.subscriber_counts().await?;
let pinned = state.ctx.db.pinned_feeds(user.id)?; let pinned = state.ctx.db.pinned_feeds(user.id).await?;
let mut out = Vec::with_capacity(mine.len()); let mut out = Vec::with_capacity(mine.len());
for sub in &subs { for sub in &subs {
let (id, feed) = (&sub.id, &sub.cfg); let (id, feed) = (&sub.id, &sub.cfg);
@@ -689,8 +689,8 @@ async fn feeds(
// the same fallback the scanner uses (`Db::subscribers`). // the same fallback the scanner uses (`Db::subscribers`).
let up = feed.group.as_deref().and_then(|g| mine.get(g)); let up = feed.group.as_deref().and_then(|g| mine.get(g));
let Some(mine) = mine.get(id) else { continue }; let Some(mine) = mine.get(id) else { continue };
let s = state.ctx.db.feed_summary(id)?; let s = state.ctx.db.feed_summary(id).await?;
let st = state.ctx.db.http_state(id)?; let st = state.ctx.db.http_state(id).await?;
out.push(FeedRow { out.push(FeedRow {
id: id.clone(), id: id.clone(),
url: feed.url.clone(), url: feed.url.clone(),
@@ -740,7 +740,7 @@ async fn feeds(
last_error: s.last_error, last_error: s.last_error,
entries: s.entries, entries: s.entries,
downloaded: s.downloaded, downloaded: s.downloaded,
unread: state.ctx.db.unread_count(user.id, id)?, unread: state.ctx.db.unread_count(user.id, id).await?,
subscribers: counts.get(id).copied().unwrap_or(0), subscribers: counts.get(id).copied().unwrap_or(0),
pinned: pinned.contains(id), pinned: pinned.contains(id),
}); });
@@ -800,13 +800,13 @@ struct PopularRow {
/// first. Popular is the top of it, the directory is all of it, and it is all that /// first. Popular is the top of it, the directory is all of it, and it is all that
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the /// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
/// feeds inside it and never itself: both lists are for finding a show. /// feeds inside it and never itself: both lists are for finding a show.
fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> { async fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
let db = &state.ctx.db; let db = &state.ctx.db;
let mine: std::collections::HashSet<String> = let mine: std::collections::HashSet<String> =
db.subscriptions_for(user_id)?.into_iter().map(|s| s.feed_id).collect(); db.subscriptions_for(user_id).await?.into_iter().map(|s| s.feed_id).collect();
let counts = db.subscriber_counts()?; let counts = db.subscriber_counts().await?;
let media = db.media_feeds()?; let media = db.media_feeds().await?;
let catalogue = crate::subscriptions(&state.ctx)?; let catalogue = crate::subscriptions(&state.ctx).await?;
let by_id: std::collections::HashMap<&str, &crate::config::Feed> = let by_id: std::collections::HashMap<&str, &crate::config::Feed> =
catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect(); catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect();
let is_folder: std::collections::HashSet<&str> = let is_folder: std::collections::HashSet<&str> =
@@ -823,7 +823,7 @@ fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
{ {
continue; continue;
} }
let sum = db.feed_summary(&s.id)?; let sum = db.feed_summary(&s.id).await?;
let subscribed = mine.contains(&s.id); let subscribed = mine.contains(&s.id);
out.push(PopularRow { out.push(PopularRow {
id: s.id.clone(), id: s.id.clone(),
@@ -844,7 +844,7 @@ async fn get_popular(
State(state): State<WebState>, State(state): State<WebState>,
user: crate::db::User, user: crate::db::User,
) -> Result<Json<Vec<PopularRow>>, ApiError> { ) -> Result<Json<Vec<PopularRow>>, ApiError> {
let mut rows = popular(&state, user.id)?; let mut rows = popular(&state, user.id).await?;
rows.truncate(10); rows.truncate(10);
Ok(Json(rows)) Ok(Json(rows))
} }
@@ -854,7 +854,7 @@ async fn get_directory(
State(state): State<WebState>, State(state): State<WebState>,
user: crate::db::User, user: crate::db::User,
) -> Result<Json<Vec<PopularRow>>, ApiError> { ) -> Result<Json<Vec<PopularRow>>, ApiError> {
let mut rows = popular(&state, user.id)?; let mut rows = popular(&state, user.id).await?;
rows.sort_by_key(sort_name); rows.sort_by_key(sort_name);
Ok(Json(rows)) Ok(Json(rows))
} }
@@ -870,10 +870,10 @@ async fn subscribe_popular(
user: crate::db::User, user: crate::db::User,
Path(id): Path<String>, Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> { ) -> Result<Json<serde_json::Value>, ApiError> {
if !popular(&state, user.id)?.iter().any(|p| p.id == id) { if !popular(&state, user.id).await?.iter().any(|p| p.id == id) {
return Err(ApiError::bad_request(format!("{id:?} is not in the directory"))); return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
} }
state.ctx.db.subscribe(user.id, &id)?; state.ctx.db.subscribe(user.id, &id).await?;
Ok(Json(serde_json::json!({ "id": id }))) Ok(Json(serde_json::json!({ "id": id })))
} }
@@ -1124,7 +1124,7 @@ async fn entries(
user: crate::db::User, user: crate::db::User,
Query(page): Query<Page>, Query(page): Query<Page>,
) -> Result<Json<EntryPage>, ApiError> { ) -> Result<Json<EntryPage>, ApiError> {
entry_page(&state, user.id, Some(&id), &page) entry_page(&state, user.id, Some(&id), &page).await
} }
/// Every subscribed feed's items together, newest first: All Subscriptions. /// Every subscribed feed's items together, newest first: All Subscriptions.
@@ -1133,11 +1133,11 @@ async fn all_entries(
user: crate::db::User, user: crate::db::User,
Query(page): Query<Page>, Query(page): Query<Page>,
) -> Result<Json<EntryPage>, ApiError> { ) -> Result<Json<EntryPage>, ApiError> {
entry_page(&state, user.id, None, &page) entry_page(&state, user.id, None, &page).await
} }
/// One feed's page of items, or every subscribed feed's when `feed` is None. /// One feed's page of items, or every subscribed feed's when `feed` is None.
fn entry_page( async fn entry_page(
state: &WebState, state: &WebState,
user_id: i64, user_id: i64,
feed: Option<&str>, feed: Option<&str>,
@@ -1146,19 +1146,21 @@ fn entry_page(
let filter = crate::db::Filter::parse(page.filter.as_deref().unwrap_or("all")); let filter = crate::db::Filter::parse(page.filter.as_deref().unwrap_or("all"));
let search = page.q.as_deref().map(str::trim).filter(|q| !q.is_empty()); let search = page.q.as_deref().map(str::trim).filter(|q| !q.is_empty());
let db = &state.ctx.db; let db = &state.ctx.db;
// Currently Listening keeps its own order, pinned or not: it is what you are part-way through.
let order = crate::db::order_sql( let order = crate::db::order_sql(
page.sort.as_deref().unwrap_or("published"), page.sort.as_deref().unwrap_or("published"),
page.dir.as_deref().unwrap_or("desc"), page.dir.as_deref().unwrap_or("desc"),
filter != crate::db::Filter::InProgress,
); );
let mut rows = let mut rows =
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order)?; db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order).await?;
let mut sanitizer = feed_sanitizer(); let mut sanitizer = feed_sanitizer();
for row in &mut rows { for row in &mut rows {
if let Some(d) = &row.description { if let Some(d) = &row.description {
row.description = Some(clean_description(&mut sanitizer, d, row.link.as_deref())); row.description = Some(clean_description(&mut sanitizer, d, row.link.as_deref()));
} }
} }
let total = db.count_in(user_id, feed, filter, search)?; let total = db.count_in(user_id, feed, filter, search).await?;
Ok(Json(EntryPage { total, entries: rows })) Ok(Json(EntryPage { total, entries: rows }))
} }
@@ -1200,10 +1202,10 @@ struct NewFeed {
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it /// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
/// goes on your subscription, and before the first scan, which would otherwise skip every /// goes on your subscription, and before the first scan, which would otherwise skip every
/// explicit item. /// explicit item.
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> { async fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
if allow { if allow {
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() }; let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
state.ctx.db.set_subscription(user_id, &sub)?; state.ctx.db.set_subscription(user_id, &sub).await?;
} }
Ok(()) Ok(())
} }
@@ -1217,14 +1219,14 @@ async fn add_feed(
let url = crate::feed::expand_input(&body.url); let url = crate::feed::expand_input(&body.url);
// Someone else may already have it. Then adding costs nothing: no second fetch, no // Someone else may already have it. Then adding costs nothing: no second fetch, no
// second copy on disk, just another name against the same feed. // second copy on disk, just another name against the same feed.
if let Some(existing) = crate::subscriptions(&state.ctx)? if let Some(existing) = crate::subscriptions(&state.ctx).await?
.into_iter() .into_iter()
.find(|s| crate::feed::same_feed(&s.cfg.url, &url)) .find(|s| crate::feed::same_feed(&s.cfg.url, &url))
{ {
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some(); let already = state.ctx.db.subscription(user.id, &existing.id).await?.is_some();
state.ctx.db.subscribe(user.id, &existing.id)?; state.ctx.db.subscribe(user.id, &existing.id).await?;
if !already { if !already {
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?; explicit_on_add(&state, user.id, &existing.id, body.allow_explicit).await?;
} }
scan_soon(&state, Some(existing.id.clone())).await; scan_soon(&state, Some(existing.id.clone())).await;
return Ok(Json( return Ok(Json(
@@ -1234,8 +1236,8 @@ async fn add_feed(
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?; let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
cfg.save(&state.config_path)?; cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?; state.ctx.reload_cfg(&state.config_path)?;
state.ctx.db.subscribe(user.id, &id)?; state.ctx.db.subscribe(user.id, &id).await?;
explicit_on_add(&state, user.id, &id, body.allow_explicit)?; explicit_on_add(&state, user.id, &id, body.allow_explicit).await?;
scan_soon(&state, Some(id.clone())).await; scan_soon(&state, Some(id.clone())).await;
Ok(Json(serde_json::json!({ "id": id, "existing": false }))) Ok(Json(serde_json::json!({ "id": id, "existing": false })))
} }
@@ -1288,17 +1290,17 @@ async fn patch_feed(
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
// Pinning is yours alone too, and means nothing for a feed you do not subscribe to. // Pinning is yours alone too, and means nothing for a feed you do not subscribe to.
if let Some(on) = body.pinned if let Some(on) = body.pinned
&& !state.ctx.db.set_pinned(user.id, &id, on)? && !state.ctx.db.set_pinned(user.id, &id, on).await?
{ {
return Err(ApiError::not_found("you do not subscribe to that feed")); return Err(ApiError::not_found("you do not subscribe to that feed"));
} }
// What one person wants -- which items, whether to fetch them, how many at a time -- // What one person wants -- which items, whether to fetch them, how many at a time --
// is theirs. It goes on their subscription and nobody else sees the change. // is theirs. It goes on their subscription and nobody else sees the change.
if state.ctx.db.subscription(user.id, &id)?.is_some() { if state.ctx.db.subscription(user.id, &id).await?.is_some() {
let mut mine = state let mut mine = state
.ctx .ctx
.db .db
.subscription(user.id, &id)? .subscription(user.id, &id).await?
.unwrap_or_else(|| crate::db::Sub { feed_id: id.clone(), ..Default::default() }); .unwrap_or_else(|| crate::db::Sub { feed_id: id.clone(), ..Default::default() });
let mut touched = false; let mut touched = false;
if let Some(v) = body.keywords.clone() { if let Some(v) = body.keywords.clone() {
@@ -1318,7 +1320,7 @@ async fn patch_feed(
touched = true; touched = true;
} }
if touched { if touched {
state.ctx.db.set_subscription(user.id, &mine)?; state.ctx.db.set_subscription(user.id, &mine).await?;
} }
} }
@@ -1339,13 +1341,13 @@ async fn patch_feed(
// Derived feeds have no config entry. Editing one is the moment it earns a real // Derived feeds have no config entry. Editing one is the moment it earns a real
// entry: promote it, so the config holds your decisions and nothing else. // entry: promote it, so the config holds your decisions and nothing else.
if !cfg.feeds.contains_key(&id) { if !cfg.feeds.contains_key(&id) {
let subs = crate::subscriptions(&state.ctx)?; let subs = crate::subscriptions(&state.ctx).await?;
let found = subs let found = subs
.iter() .iter()
.find(|s| s.id == id) .find(|s| s.id == id)
.ok_or_else(|| ApiError::bad_request(format!("no feed with id {id:?}")))?; .ok_or_else(|| ApiError::bad_request(format!("no feed with id {id:?}")))?;
cfg.feeds.insert(id.clone(), found.cfg.clone()); cfg.feeds.insert(id.clone(), found.cfg.clone());
state.ctx.db.unmanage(&id)?; state.ctx.db.unmanage(&id).await?;
} }
let checked = match &body.url { let checked = match &body.url {
@@ -1386,7 +1388,7 @@ async fn patch_feed(
if url_changed { if url_changed {
// Refreshing a rotated auth token is the common case; entries and download history // Refreshing a rotated auth token is the common case; entries and download history
// are keyed by feed id, so they survive the change. // are keyed by feed id, so they survive the change.
state.ctx.db.clear_validators(&id)?; state.ctx.db.clear_validators(&id).await?;
} }
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -1398,14 +1400,14 @@ async fn remove_feed(
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
// Unsubscribing is personal: it takes the feed off your list and leaves everyone // Unsubscribing is personal: it takes the feed off your list and leaves everyone
// else's alone. // else's alone.
state.ctx.db.unsubscribe(user.id, &id)?; state.ctx.db.unsubscribe(user.id, &id).await?;
for child in crate::subscriptions(&state.ctx)? for child in crate::subscriptions(&state.ctx).await?
.iter() .iter()
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str())) .filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
{ {
state.ctx.db.unsubscribe(user.id, &child.id)?; state.ctx.db.unsubscribe(user.id, &child.id).await?;
} }
if state.ctx.db.subscriber_counts()?.contains_key(&id) { if state.ctx.db.subscriber_counts().await?.contains_key(&id) {
return Ok(StatusCode::NO_CONTENT); return Ok(StatusCode::NO_CONTENT);
} }
@@ -1415,12 +1417,12 @@ async fn remove_feed(
if cfg.feeds.remove(&id).is_none() { if cfg.feeds.remove(&id).is_none() {
// A derived feed: forget it here, though the OPML will list it again on the next // A derived feed: forget it here, though the OPML will list it again on the next
// read unless you unsubscribe from the OPML itself. // read unless you unsubscribe from the OPML itself.
state.ctx.db.drop_managed(&id)?; state.ctx.db.drop_managed(&id).await?;
return Ok(StatusCode::NO_CONTENT); return Ok(StatusCode::NO_CONTENT);
} }
cfg.save(&state.config_path)?; cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?; state.ctx.reload_cfg(&state.config_path)?;
crate::retire_group(&state.ctx, &id)?; crate::retire_group(&state.ctx, &id).await?;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -1438,10 +1440,10 @@ async fn set_flags(
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
use crate::db::EntryFlag; use crate::db::EntryFlag;
if let Some(v) = body.read { if let Some(v) = body.read {
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Read, v)?; state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Read, v).await?;
} }
if let Some(v) = body.flagged { if let Some(v) = body.flagged {
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Flagged, v)?; state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Flagged, v).await?;
} }
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -1456,12 +1458,12 @@ async fn download_now(
let enc = state let enc = state
.ctx .ctx
.db .db
.enclosure(id)? .enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?; .ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
if enc.path.is_some() { if enc.path.is_some() {
return Ok(StatusCode::NO_CONTENT); // Already here. return Ok(StatusCode::NO_CONTENT); // Already here.
} }
state.ctx.db.requeue(id)?; state.ctx.db.requeue(id).await?;
state state
.cmds .cmds
.send(Command::Download { enclosure: id }) .send(Command::Download { enclosure: id })
@@ -1485,13 +1487,13 @@ async fn delete_file(
let enc = state let enc = state
.ctx .ctx
.db .db
.enclosure(id)? .enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?; .ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
// There is one copy of the file: deleting it deletes everyone's. Say so before doing // There is one copy of the file: deleting it deletes everyone's. Say so before doing
// it, once, and let them decide. // it, once, and let them decide.
if !q.force { if !q.force {
let (starred, unread) = state.ctx.db.others_wanting(id, user.id)?; let (starred, unread) = state.ctx.db.others_wanting(id, user.id).await?;
let people = |n: i64| if n == 1 { "person".to_string() } else { format!("{n} people") }; let people = |n: i64| if n == 1 { "person".to_string() } else { format!("{n} people") };
let complaint = match (starred, unread) { let complaint = match (starred, unread) {
(0, 0) => None, (0, 0) => None,
@@ -1517,7 +1519,7 @@ async fn delete_file(
return Err(e.into()); return Err(e.into());
} }
// The row survives as 'reaped', which is what stops the next scan re-downloading it. // The row survives as 'reaped', which is what stops the next scan re-downloading it.
state.ctx.db.mark_reaped(id)?; state.ctx.db.mark_reaped(id).await?;
state.events.send(Event::Reaped { state.events.send(Event::Reaped {
path: enc.path.unwrap_or_default(), path: enc.path.unwrap_or_default(),
bytes: enc.length.unwrap_or(0).max(0) as u64, bytes: enc.length.unwrap_or(0).max(0) as u64,
@@ -1571,7 +1573,7 @@ async fn media(
Path(id): Path<i64>, Path(id): Path<i64>,
req: Request, req: Request,
) -> Response { ) -> Response {
let Ok(Some(enc)) = state.ctx.db.enclosure(id) else { let Ok(Some(enc)) = state.ctx.db.enclosure(id).await else {
return (StatusCode::NOT_FOUND, "no such enclosure").into_response(); return (StatusCode::NOT_FOUND, "no such enclosure").into_response();
}; };
let Some(path) = enc.path else { let Some(path) = enc.path else {
@@ -1596,7 +1598,7 @@ async fn set_position(
user: crate::db::User, user: crate::db::User,
Json(body): Json<Position>, Json(body): Json<Position>,
) -> Result<StatusCode, ApiError> { ) -> Result<StatusCode, ApiError> {
state.ctx.db.set_position(user.id, &feed_id, &guid, body.secs, body.duration)?; state.ctx.db.set_position(user.id, &feed_id, &guid, body.secs, body.duration).await?;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -1608,12 +1610,12 @@ async fn read_all(
// A subscription's own row has no entries, so marking it read means everything under it. // A subscription's own row has no entries, so marking it read means everything under it.
let mut ids = vec![id.clone()]; let mut ids = vec![id.clone()];
ids.extend( ids.extend(
crate::subscriptions(&state.ctx)? crate::subscriptions(&state.ctx).await?
.into_iter() .into_iter()
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str())) .filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
.map(|s| s.id), .map(|s| s.id),
); );
let n = state.ctx.db.mark_all_read(user.id, &ids)?; let n = state.ctx.db.mark_all_read(user.id, &ids).await?;
Ok(Json(serde_json::json!({ "marked": n }))) Ok(Json(serde_json::json!({ "marked": n })))
} }
@@ -1624,8 +1626,8 @@ async fn read_all_mine(
user: crate::db::User, user: crate::db::User,
) -> Result<Json<serde_json::Value>, ApiError> { ) -> Result<Json<serde_json::Value>, ApiError> {
let ids: Vec<String> = let ids: Vec<String> =
state.ctx.db.subscriptions_for(user.id)?.into_iter().map(|s| s.feed_id).collect(); state.ctx.db.subscriptions_for(user.id).await?.into_iter().map(|s| s.feed_id).collect();
let n = state.ctx.db.mark_all_read(user.id, &ids)?; let n = state.ctx.db.mark_all_read(user.id, &ids).await?;
Ok(Json(serde_json::json!({ "marked": n }))) Ok(Json(serde_json::json!({ "marked": n })))
} }
@@ -1646,9 +1648,9 @@ async fn download_latest(
Path(id): Path<String>, Path(id): Path<String>,
Json(body): Json<HowMany>, Json(body): Json<HowMany>,
) -> Result<Json<serde_json::Value>, ApiError> { ) -> Result<Json<serde_json::Value>, ApiError> {
let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100))?; let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100)).await?;
for enc in &ids { for enc in &ids {
state.ctx.db.requeue(*enc)?; state.ctx.db.requeue(*enc).await?;
state state
.cmds .cmds
.send(Command::Download { enclosure: *enc }) .send(Command::Download { enclosure: *enc })
@@ -1666,7 +1668,7 @@ async fn export_opml(
// Yours, not the whole catalogue: other people's feeds, and any private URLs in them, are // Yours, not the whole catalogue: other people's feeds, and any private URLs in them, are
// not yours to download. This used to export config.toml to whoever asked. // not yours to download. This used to export config.toml to whoever asked.
let mine: std::collections::HashSet<String> = let mine: std::collections::HashSet<String> =
state.ctx.db.subscriptions_for(user.id)?.into_iter().map(|s| s.feed_id).collect(); state.ctx.db.subscriptions_for(user.id).await?.into_iter().map(|s| s.feed_id).collect();
let mut doc = opml::OPML { let mut doc = opml::OPML {
head: Some(opml::Head { head: Some(opml::Head {
title: Some("ipx subscriptions".into()), title: Some("ipx subscriptions".into()),
@@ -1674,7 +1676,7 @@ async fn export_opml(
}), }),
..Default::default() ..Default::default()
}; };
for s in crate::subscriptions(&state.ctx)? { for s in crate::subscriptions(&state.ctx).await? {
// A feed from an OPML subscription comes back with the OPML itself. // A feed from an OPML subscription comes back with the OPML itself.
if s.managed || !mine.contains(&s.id) { if s.managed || !mine.contains(&s.id) {
continue; continue;
@@ -1682,7 +1684,7 @@ async fn export_opml(
let title = state let title = state
.ctx .ctx
.db .db
.feed_summary(&s.id) .feed_summary(&s.id).await
.ok() .ok()
.and_then(|sum| sum.title) .and_then(|sum| sum.title)
.unwrap_or_else(|| s.id.clone()); .unwrap_or_else(|| s.id.clone());
@@ -1716,7 +1718,7 @@ async fn import_opml(
// file arrives as text, is read here, and is gone when the request ends. // file arrives as text, is read here, and is gone when the request ends.
let doc = opml::OPML::from_str(&body.xml) let doc = opml::OPML::from_str(&body.xml)
.map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?; .map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?;
let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id)?; let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id).await?;
if added > 0 { if added > 0 {
scan_soon(&state, None).await; scan_soon(&state, None).await;
} }

View File

@@ -1299,3 +1299,20 @@ test('a pinned feed, even one from inside a folder, goes to the top of the list'
await expect(page.locator('.feed.pinned')).toHaveCount(0); await expect(page.locator('.feed.pinned')).toHaveCount(0);
await expect(page.locator('.feed.child', { hasText: name })).toHaveCount(1); await expect(page.locator('.feed.child', { hasText: name })).toHaveCount(1);
}); });
test('a pinned item goes to the top of its list, and back when unpinned', async ({ page }) => {
await page.locator('.feed', { hasText: 'Test Show' }).click();
await page.locator('.tabs button', { hasText: 'All' }).first().click();
await expect(page.locator('.ep').nth(1)).toBeVisible({ timeout: 20_000 });
const guids = () => page.locator('.ep').evaluateAll(rows => rows.map(r => r.dataset.guid));
const before = await guids();
const last = before[before.length - 1];
const row = page.locator(`.ep[data-guid="${last}"]`);
await row.locator('[data-a="flag"]').click();
await expect.poll(async () => (await guids())[0]).toBe(last);
// It is the server's order, so it holds on a reload.
await page.reload();
await expect.poll(async () => (await guids())[0]).toBe(last);
await page.locator(`.ep[data-guid="${last}"] [data-a="flag"]`).click();
await expect.poll(guids).toEqual(before);
});

View File

@@ -314,7 +314,9 @@ async function epAction(a: string, e, el, encId?: number){
const path=`/api/entries/${encodeURIComponent(e.feed_id)}/${encodeURIComponent(e.guid)}`; const path=`/api/entries/${encodeURIComponent(e.feed_id)}/${encodeURIComponent(e.guid)}`;
try{ try{
if(a==='play') play(e, encId!=null ? enc : undefined); if(a==='play') play(e, encId!=null ? enc : undefined);
if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); } // A pinned item sits at the top of its list (the server sorts it there), so the list is
// asked for again rather than the row redrawn where it stands.
if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); loadEntries(); }
if(a==='read'){ await setRead(e,!e.read); redraw(); } if(a==='read'){ await setRead(e,!e.read); redraw(); }
if(a==='get'){ if(a==='get'){
if(!enc) return; if(!enc) return;