10 Commits

Author SHA1 Message Date
bc53e0f730 Postgres: pick the database by URL, copy-db, tests on both
- IPX_DATABASE_URL (postgres://...) picks the database; unset, it is the SQLite
  file as before. Passwords are taken out of anything logged.
- `ipx copy-db <state.db>` copies every table into the empty database the URL
  names, in one transaction, and moves the id counters past the copied ids. A
  copy of production went across in 14s with every count and column
  fingerprint identical.
- With IPX_TEST_DATABASE_URL set, each test gets a Postgres schema of its own;
  all 79 pass on both databases. Fixtures write booleans as true/false.
- Sorts say where an item with no value goes (NULLS FIRST going up, LAST going
  down): SQLite counts NULL as smallest, Postgres as largest, so "largest first"
  on Postgres led with every item that has no file. Tested on both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 20:28:50 +00:00
bd8f6ab855 Docs: the database through SeaORM
CLAUDE.md and the architecture notes described the SQL schema and migrate(),
both gone: the entities are the schema, create_missing makes what is missing,
and hand-written SQL has to run on SQLite and Postgres both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 19:11:24 +00:00
611716d8b7 SeaORM: feeds and scanning; rusqlite gone
The last nineteen functions move to SeaORM: recording feeds, items and
enclosures, managed OPML feeds, folding WordPress's repeated files, and handing a
Patreon creator's files to its shows. Two SQLite-only forms go: GLOB becomes a
LIKE with the underscore escaped (broader, harmlessly: the fold still keys on
`_=` and digits), and UPDATE OR IGNORE becomes an UPDATE ... WHERE NOT EXISTS.
The two transactions are SeaORM transactions.

With nothing left on it, rusqlite goes, with the SQL schema and migrate(). The
entities are the schema: create_missing makes whatever tables and indexes a
database lacks, from them, with CREATE ... IF NOT EXISTS. Production's schema
already has every column migrate() added and none it dropped.

Not SeaORM's schema sync, used until now: despite its docs it drops a unique
index the entities do not describe, so it dropped users_name_lower on every open.
Every `ipx` command then took a write lock, and against a daemon busy writing,
`ipx status` -- the healthcheck -- failed 7 times in 15 where the old code
failed none. Now 15 in 15, as before. On Postgres it would not have started.

WAL is set only when a file is not already in it: setting it takes a lock that
cannot wait out a busy daemon.

Checked on copies of production: a forced scan of all 162 feeds against the real
feeds with no database errors; the feed list, filters, sorts, search and the
reaper's candidates against the old code on the same data, earlier in the
branch. The column comments from the SQL schema move to the entities.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 19:10:59 +00:00
a68bfb179b SeaORM: enclosures, downloads and the reaper
Twelve enclosure functions move to SeaORM: recording, the download queue,
marking done or failed, requeueing, and what the reaper may delete. INSERT OR
IGNORE becomes ON CONFLICT DO NOTHING; the reaper's read verdict is true or
false rather than 1 or 0, which Postgres would type as a 32-bit integer and
refuse to read as an i64; `read = 1` and `flagged = 1` test the booleans
themselves. retention::run and its callers (reap, rm, retire_group,
retire_stranded) become async.

The reaper deletes files, so it was checked on a copy of production against the
old SQL on the same file: all 2,195 candidates, identical and in the same order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:38:07 +00:00
6bf1ad6b31 SeaORM: items and read state
The item list, its counts, filters, sorts and search, positions, pins and
mark-all-read move to SeaORM, as SQL written for both databases:

- Parameters are gathered as the SQL is written (Args), so only what a
  statement uses is bound. rusqlite needed every one mentioned, hence the old
  `?1 IS NULL` and `?2 = ''`; Postgres refuses a parameter it cannot type.
- Yes/no columns are tested as booleans (NOT coalesce(s.read, false)) and
  written as true, not 1; SQLite reads true and false as 1 and 0.
- The last tiebreak of the sort is the guid, not SQLite's rowid, which Postgres
  lacks. Only items with the same date change places.
- set_position names entry_state.duration beside excluded.duration.
- The status callback on the control socket returns a future, as the counts
  are now a query.

Checked on a copy of production against the live server: 42 of 48 lists
identical; the other six differ only in how ties fall, or because the test
daemon cleared paths to files this machine does not have. Run on the same file,
every filter's count matches the old SQL exactly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:32:46 +00:00
d7f8f2df1d SeaORM: subscriptions and pins
Twelve subscription functions move to SeaORM. Lookups use the entity API; the
joins, counts and upserts are SQL written to run on both databases: $n
parameters, ON CONFLICT DO NOTHING in place of INSERT OR IGNORE, and
CASE WHEN on the yes/no column itself rather than comparing it to 1, which
Postgres would refuse for a boolean. INSERT ... SELECT ... ON CONFLICT gets a
WHERE true, which SQLite needs to tell the two apart.

Checked with a daemon on a copy of production: the feed list, read through the
new code, comes back with every feed and its settings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:23:37 +00:00
4927677e66 SeaORM: accounts, sessions and themes
The fourteen user and session functions move from rusqlite to SeaORM and become
async; their callers await them (auth, admin_user, user_cmd, the account
handlers). Checked against a copy of production, where the yes/no columns are
still INTEGER: the admin flag reads back right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:17:52 +00:00
484aaa1849 SeaORM beside rusqlite: entities, schema sync, a second connection
The first step of moving to SeaORM (#18, phase 1). Nothing a user sees changes.

- src/entity.rs: the seven tables as SeaORM entities, matching the SQLite schema.
  Strings are Text, as the columns are; yes/no columns are bool, which is BOOLEAN
  on Postgres and stays INTEGER in the existing SQLite file (sync notes the
  difference and leaves it alone).
- Db holds a SeaORM connection to the same SQLite file beside the rusqlite one;
  functions move to it one at a time, and rusqlite goes with the last of them.
- db::sync creates what a database is missing from the entities (SeaORM's
  schema-sync, experimental, so sea-orm is pinned to ~2.0), plus the two indexes
  an entity cannot express. Checked against a copy of production: it added the
  lower(name) index and changed nothing else.
- Test databases are now built from the entities alone, in a temporary file
  (two connections to one ":memory:" are two databases), so every test also
  checks that the entities describe what the queries need. That caught the one
  difference: finding a user by name relied on COLLATE NOCASE, which Postgres
  lacks; it now compares lower() on both sides.
- rusqlite steps back to 0.39: 0.40's libsqlite3-sys is newer than sqlx accepts,
  and only one may link SQLite. It goes away at the end of this phase.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 18:11:28 +00:00
c99e17bd80 A pinned item goes to the top of its list
order_sql takes pinned_first, which puts coalesce(s.flagged, 0) DESC ahead of
the chosen sort, so pins lead every list in whatever order is asked for and on
every page of it. Not when sorting by the pin column itself, where the direction
is the point, and not for Currently Listening. Pinning now asks for the list again
so the row moves at once, instead of redrawing it where it stood.

Closes #35.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 17:51:38 +00:00
e312f11bb1 Remove docs/history.md
It had grown past 1,700 lines, too large to be read or kept up. What it held --
what was wrong before a change and what it cost to find -- goes in commit
message bodies now, beside the change. CLAUDE.md says so; the README and the
changelog no longer point at it. It remains in git history.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 16:47:21 +00:00
15 changed files with 2652 additions and 3303 deletions

View File

@@ -5,11 +5,17 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
The long form, with what was wrong before and how it was found, is in
[docs/history.md](docs/history.md).
## [Unreleased]
### Changed
- The database is reached through SeaORM, on the way to Postgres (issue #18); it is still the
same SQLite file, and nothing you see changes. A database from before 0.7 has to be opened by a
0.7 release first, which brings its tables up to date.
- A pinned item sits at the top of its list, above everything else in whatever order you sort
by, and moves there the moment you pin it. Sorting by the pin column itself still goes both
ways, and Currently Listening keeps its own order.
## [0.7.0] - 2026-09-18
### Added

View File

@@ -125,7 +125,7 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
and `position` columns from before accounts; two bugs came from queries still reading them
(retention, and the entry pruner), and `migrate()` now drops them.
(retention, and the entry pruner), and they were dropped in 0.5.
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
derived and never written to config.
@@ -140,10 +140,16 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
different task. `ipx status` checks the control socket and the database; to see the worker
getting through its jobs, watch for `scan complete` in the log.
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
for `daemon started` in the log. Copy `state.db` aside before deploying one.
* **The database goes through SeaORM, and the entities in `src/entity.rs` are the schema.**
`Db::open` creates any missing table or index from them (`create_missing`), on every `ipx`
command, the healthcheck's `ipx status` included, so it must never write when nothing is
missing: SeaORM's experimental schema sync dropped and remade an index on every open, the
write lock that took made `ipx status` time out behind a busy daemon, and it was removed for
it. A new column on an existing table needs its own `ALTER`; nothing adds one for you.
* **SQL written by hand in `db.rs` has to run on SQLite and Postgres both** (issue #18): `$1`
parameters, bound only if used; `ON CONFLICT`, not `INSERT OR IGNORE`; yes/no columns tested
as themselves (`NOT coalesce(s.read, false)`) and written as `true`/`false`, never compared to
1; no `rowid`, `GLOB` or `UPDATE OR IGNORE`. `Args` in `db.rs` builds the parameters.
## House style
@@ -154,9 +160,9 @@ addressed to the person using it.
Every change gets one line under `## [Unreleased]` in [CHANGELOG.md](CHANGELOG.md), in its
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) group: Added, Changed, Deprecated,
Removed, Fixed or Security. Say it the way someone using ipx would notice it. When there is more to
say, such as what was wrong before or what it cost to find out, write it up at the top of
[docs/history.md](docs/history.md), dated. That record has been more useful than the git log more
than once.
say, such as what was wrong before or what it cost to find out, it goes in the commit message's
body, where `git log` and `git blame` find it beside the change. (There was a long-form
`docs/history.md` until 0.7.0; it grew too large to be useful and was removed. It is in git.)
Cutting a release: rename `[Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` and open a new empty
`[Unreleased]` above it, bump `version` in `Cargo.toml`, tag the commit `vX.Y.Z`, and update the

873
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -18,7 +18,7 @@ percent-encoding = "2.3.2"
quick-xml = { version = "0.42.0", features = ["escape-html"] }
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] }
rss = "2.1.1"
rusqlite = { version = "0.40.2", features = ["bundled"] }
sea-orm = { version = "2.0.3", default-features = false, features = ["sqlx-sqlite", "sqlx-postgres", "runtime-tokio-rustls", "macros", "with-json", "sqlite-use-returning-for-3_35"] }
serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151"
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }

View File

@@ -63,7 +63,6 @@ The UI is plain HTTP, so put TLS in front of it if it is reachable from outside
| [docs/sso.md](docs/sso.md) | Signing in through Cloudflare Zero Trust or Authentik |
| [docs/architecture.md](docs/architecture.md) | How it works: modules, schema, control socket, HTTP API |
| [CHANGELOG.md](CHANGELOG.md) | What changed, by release |
| [docs/history.md](docs/history.md) | How it was built, with what was wrong and why |
| [CLAUDE.md](CLAUDE.md) | Notes for working on the code, including how production is deployed |
## Tests

View File

@@ -10,7 +10,8 @@ it to a running daemon.
|---|---|---|
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
| `src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` |
| `src/entity.rs` | The tables, as SeaORM entities: the schema | — |
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
@@ -65,14 +66,14 @@ entry_state user_id, feed_id, guid, read, flagged, position
```
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
older database.
before accounts; two bugs came from queries still reading them, and they were dropped in 0.5.
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
path as `Db::open`, so a migration cannot pass the tests while missing in production.
Schema changes: the tables are the entities in `src/entity.rs`, and `Db::open` creates whatever
table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It
never alters a table that exists, so a new column on one needs its own `ALTER` in
`create_missing`, or `sea-orm-migration` once there are several. `Db::memory()` builds its
database the same way, so the tests run on the schema production gets. A database from before
0.7 takes its last columns from the old `migrate()`, so it upgrades through a 0.7 release first.
## Control socket

File diff suppressed because it is too large Load Diff

2391
src/db.rs

File diff suppressed because it is too large Load Diff

244
src/entity.rs Normal file
View File

@@ -0,0 +1,244 @@
//! The database's tables as SeaORM entities: the one description of the schema, from which
//! `Db::open` creates what a database is missing, on SQLite or Postgres alike (see
//! `db::create_missing`). Times are Unix seconds.
pub mod feeds {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "feeds")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub id: String,
#[sea_orm(column_type = "Text")]
pub url: String,
#[sea_orm(column_type = "Text", nullable)]
pub title: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub image: Option<String>,
/// The channel's first <itunes:category>, for the Directory.
#[sea_orm(column_type = "Text", nullable)]
pub category: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub etag: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub last_modified: Option<String>,
pub last_checked: Option<i64>,
pub ttl_mins: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub last_error: Option<String>,
/// When the current run of failures began; NULL while the feed is healthy. Kept through
/// repeated failures so the UI can tell a blip (macmanx: failed once, fine an hour
/// later) from a feed that has been down for a day.
pub error_since: Option<i64>,
/// Came from a subscribed OPML that no longer lists it, but has downloads, so kept.
#[sea_orm(default_value = false)]
pub orphaned: bool,
/// The OPML subscription this feed came from.
#[sea_orm(column_type = "Text", nullable)]
pub group_id: Option<String>,
/// Derived from an OPML and not written to config.toml. Writing 80-odd generated entries
/// into a hand-edited file made it unreadable; the OPML is the source of truth, so they
/// are re-derived instead. Customising one promotes it to config.
#[sea_orm(default_value = false)]
pub managed: bool,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
pub mod entries {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "entries")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub feed_id: String,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub guid: String,
#[sea_orm(column_type = "Text", nullable)]
pub title: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub link: Option<String>,
pub published: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub description: Option<String>,
pub first_seen: i64,
#[sea_orm(column_type = "Text", nullable)]
pub image: Option<String>,
pub duration: Option<i64>,
pub episode: Option<i64>,
pub season: Option<i64>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
pub mod enclosures {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "enclosures")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i64,
#[sea_orm(column_type = "Text")]
pub feed_id: String,
#[sea_orm(column_type = "Text")]
pub guid: String,
/// The dedupe key, and the reason one file serves every subscriber. A reaped file keeps
/// its row with path NULL and state 'reaped', so a purged episode is never fetched again.
#[sea_orm(unique, column_type = "Text")]
pub url: String,
#[sea_orm(column_type = "Text", nullable)]
pub mime: Option<String>,
pub length: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub path: Option<String>,
#[sea_orm(column_type = "Text")]
pub state: String,
#[sea_orm(default_value = 0)]
pub bytes_done: i64,
pub downloaded_at: Option<i64>,
#[sea_orm(column_type = "Text", nullable)]
pub last_error: Option<String>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
pub mod users {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "users")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i64,
/// Unique without regard to case: `db::create_missing` adds the index on lower(name), which
/// works the same on both databases where SQLite's COLLATE NOCASE does not.
#[sea_orm(column_type = "Text")]
pub name: String,
/// NULL for someone who only ever arrives through the proxy: there is no password to
/// check, and leaving it empty is not the same as leaving it unset.
#[sea_orm(column_type = "Text", nullable)]
pub pass_hash: Option<String>,
#[sea_orm(default_value = false)]
pub is_admin: bool,
/// For whoever maintains the server. NULL where it is not known.
pub created: Option<i64>,
pub last_login: Option<i64>,
/// The theme chosen in Settings, and light, dark or auto. NULL until one is chosen.
#[sea_orm(column_type = "Text", nullable)]
pub theme: Option<String>,
#[sea_orm(column_type = "Text", nullable)]
pub theme_mode: Option<String>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
/// A table of one person's rows, gone when they are.
macro_rules! owned_by_user {
() => {
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {
#[sea_orm(
belongs_to = "super::users::Entity",
from = "Column::UserId",
to = "super::users::Column::Id",
on_delete = "Cascade"
)]
User,
}
impl Related<super::users::Entity> for Entity {
fn to() -> RelationDef {
Relation::User.def()
}
}
impl ActiveModelBehavior for ActiveModel {}
};
}
/// What one person wants from a feed. The feed, its items and its files are shared; this is the
/// part that is not. NULL in a column means: follow the feed's own setting.
pub mod subscriptions {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "subscriptions")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub user_id: i64,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub feed_id: String,
/// JSON array of strings; NULL follows the feed.
#[sea_orm(column_type = "Text", nullable)]
pub keywords: Option<String>,
pub auto_download: Option<bool>,
pub allow_explicit: Option<bool>,
pub max_new_per_check: Option<i64>,
/// Pinned to the top of this person's feed list, a feed inside a folder included.
#[sea_orm(default_value = false)]
pub pinned: bool,
}
owned_by_user!();
}
/// Read, kept and how far in. One row per person per item, created on first touch; an item
/// nobody has touched has no row at all, which is what unread means.
pub mod entry_state {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "entry_state")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub user_id: i64,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub feed_id: String,
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub guid: String,
#[sea_orm(default_value = false)]
pub read: bool,
#[sea_orm(default_value = false)]
pub flagged: bool,
#[sea_orm(default_value = 0)]
pub position: i64,
/// The length this person's player measured, beside the position it is measured against.
pub duration: Option<i64>,
}
owned_by_user!();
}
pub mod sessions {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "sessions")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub token: String,
pub user_id: i64,
pub seen: i64,
}
owned_by_user!();
}

View File

@@ -176,7 +176,9 @@ pub async fn daemon_is_live(path: &Path) -> bool {
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>;
/// A future, since reading the counts is a database query.
pub type StatusFn =
std::sync::Arc<dyn Fn() -> std::pin::Pin<Box<dyn std::future::Future<Output = Event> + Send>> + Send + Sync>;
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
pub async fn serve(
@@ -249,7 +251,7 @@ async fn handle(
// Answered here, not queued behind whatever the worker is on: see StatusFn.
Ok(Command::Status) => {
tracing::info!(target: "ipx::io", "-> {line}");
let ev = status();
let ev = status().await;
if let Ok(json) = serde_json::to_string(&ev) {
tracing::info!(target: "ipx::io", "<- {json}");
}
@@ -366,7 +368,8 @@ mod tests {
// Another client, watching a scan: it must not be handed someone else's answer, which
// would end its session.
let mut watcher = events.subscribe();
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 });
let status: StatusFn =
std::sync::Arc::new(|| Box::pin(async { Event::Status { feeds: 1, pending: 2, downloaded: 3 } }));
let (client, server) = UnixStream::pair().unwrap();
tokio::spawn(handle(server, events.subscribe(), cmds, status));

View File

@@ -1,6 +1,7 @@
mod auth;
mod config;
mod db;
mod entity;
mod download;
mod feed;
mod ipc;
@@ -36,6 +37,12 @@ struct Cli {
enum Command {
/// Show configured feeds and their state
List,
/// Copy everything from a SQLite state.db into the database IPX_DATABASE_URL names, which
/// must be empty: the one-off move to Postgres
CopyDb {
/// The SQLite file to copy from
from: PathBuf,
},
/// Scan feeds for new entries
Fetch {
/// Only this feed id
@@ -179,7 +186,7 @@ async fn main() -> Result<()> {
let config_path = cli.config.clone().unwrap_or_else(config::config_path);
let cfg = config::Config::load(&config_path)?;
let db = db::Db::open(&config::data_dir().join("state.db"))?;
let db = db::Db::open(&db::location()).await?;
// A daemon owns the state; don't have two processes downloading the same thing.
let wire_cmd = match &cli.command {
@@ -194,7 +201,8 @@ async fn main() -> Result<()> {
| Command::Add { .. }
| Command::Rm { .. }
| Command::Import { .. }
| Command::Export { .. } => None,
| Command::Export { .. }
| Command::CopyDb { .. } => None,
};
if let Some(cmd) = &wire_cmd
&& !cli.local
@@ -219,22 +227,23 @@ async fn main() -> Result<()> {
});
match cli.command {
Command::List => list(&ctx, &config_path),
Command::List => list(&ctx, &config_path).await,
Command::Daemon { web } => daemon(ctx, config_path, web, events).await,
Command::Add { url, folder, keywords } => {
add(&ctx, &config_path, &url, folder, keywords).await
}
Command::Rm { feed } => rm(&ctx, &config_path, &feed),
Command::User { cmd } => user_cmd(&ctx, cmd),
Command::Rm { feed } => rm(&ctx, &config_path, &feed).await,
Command::User { cmd } => user_cmd(&ctx, cmd).await,
Command::Import { file } => import(&ctx, &config_path, &file).await,
Command::Export { file } => export(&ctx, &file),
Command::Export { file } => export(&ctx, &file).await,
Command::CopyDb { from } => copy_db(&ctx, &from).await,
_ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await,
}
}
/// Accounts. Passwords come in on stdin so they never reach a shell history or a `ps`
/// listing.
fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
async fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
let read_password = || -> Result<String> {
use std::io::Read;
let mut buf = String::new();
@@ -252,7 +261,7 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
if name.is_empty() {
anyhow::bail!("a name is required");
}
if ctx.db.user_by_name(&name)?.is_some() {
if ctx.db.user_by_name(&name).await?.is_some() {
anyhow::bail!("{name} already exists");
}
let hash = if no_password {
@@ -261,8 +270,8 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
Some(crate::auth::hash_password(&read_password()?)?)
};
// The first account runs the place; there is nobody else to grant it.
let first = ctx.db.users()?.is_empty();
ctx.db.create_user(&name, hash.as_deref(), admin || first)?;
let first = ctx.db.users().await?.is_empty();
ctx.db.create_user(&name, hash.as_deref(), admin || first).await?;
println!(
"added {name}{}{}",
if admin || first { " (admin)" } else { "" },
@@ -271,7 +280,7 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
Ok(())
}
UserCmd::List => {
let users = ctx.db.users()?;
let users = ctx.db.users().await?;
if users.is_empty() {
println!("no accounts yet: ipx user add <name>");
}
@@ -294,9 +303,9 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
let name = name.trim().to_ascii_lowercase();
let user = ctx
.db
.user_by_name(&name)?
.user_by_name(&name).await?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
ctx.db.set_password(user.id, &crate::auth::hash_password(&read_password()?)?)?;
ctx.db.set_password(user.id, &crate::auth::hash_password(&read_password()?)?).await?;
println!("password changed for {name}");
Ok(())
}
@@ -307,12 +316,12 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
let user = ctx
.db
.user_by_name(&name)?
.user_by_name(&name).await?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
if ctx.db.user_by_name(&new_name)?.is_some() {
if ctx.db.user_by_name(&new_name).await?.is_some() {
anyhow::bail!("{new_name} already exists");
}
ctx.db.rename_user(user.id, &new_name)?;
ctx.db.rename_user(user.id, &new_name).await?;
println!("renamed {name} to {new_name}");
Ok(())
}
@@ -320,9 +329,9 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
let name = name.trim().to_ascii_lowercase();
let user = ctx
.db
.user_by_name(&name)?
.user_by_name(&name).await?
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
ctx.db.delete_user(user.id)?;
ctx.db.delete_user(user.id).await?;
println!("removed {name}");
Ok(())
}
@@ -333,13 +342,13 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
match cmd {
Cmd::Fetch { feed, force } => {
// Make room before pulling more down, as the original did per download.
reap(ctx, false, false)?;
reap(ctx, false, false).await?;
fetch(ctx, feed.as_deref(), force).await
}
Cmd::Reap { dry_run } => reap(ctx, dry_run, true),
Cmd::Reap { dry_run } => reap(ctx, dry_run, true).await,
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
Cmd::Status => {
ctx.out.emit(status(ctx));
ctx.out.emit(status(ctx).await);
Ok(())
}
}
@@ -347,10 +356,10 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
/// than through the job queue.
fn status(ctx: &Ctx) -> Event {
match ctx.db.counts() {
async fn status(ctx: &Ctx) -> Event {
match ctx.db.counts().await {
Ok((pending, downloaded)) => {
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
let feeds = subscriptions(ctx).await.map(|s| s.len()).unwrap_or(0);
Event::Status { feeds, pending, downloaded }
}
Err(e) => Event::Error { msg: format!("{e:#}") },
@@ -369,30 +378,30 @@ async fn daemon(
}
// A database with nobody in it cannot be signed into.
if ctx.db.users()?.is_empty() {
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
if ctx.db.users().await?.is_empty() {
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true).await?;
tracing::warn!(
"no accounts yet: created 'admin' with the default password '{DEFAULT_PASSWORD}'. \
Change it with `echo -n <password> | ipx user passwd admin`"
);
}
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
if let Some(admin) = ctx.db.users().await?.into_iter().find(|u| u.is_admin) {
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
match ctx.db.adopt_catalogue(admin.id, &catalogue) {
match ctx.db.adopt_catalogue(admin.id, &catalogue).await {
Ok(0) => {}
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
}
}
match ctx.db.requeue_interrupted() {
match ctx.db.requeue_interrupted().await {
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
Ok(_) => {}
Err(e) => tracing::warn!(error = ?e, "could not requeue interrupted downloads"),
}
match retire_stranded(&ctx) {
match retire_stranded(&ctx).await {
Ok(0) => {}
Ok(n) => tracing::info!(feeds = n, "retired feeds whose OPML is no longer in config"),
Err(e) => tracing::warn!(error = ?e, "could not retire feeds whose OPML is no longer in config"),
@@ -400,7 +409,7 @@ async fn daemon(
// Before the parser knew WordPress's numbered player URLs, a file it listed twice was
// downloaded twice. The repeats fold into the first, and their spare copies are deleted.
match ctx.db.merge_repeated_enclosures(feed::same_file_key) {
match ctx.db.merge_repeated_enclosures(feed::same_file_key).await {
Ok((0, _)) => {}
Ok((n, spare)) => {
for path in &spare {
@@ -419,7 +428,10 @@ async fn daemon(
// status is answered by the socket itself; everything else waits its turn in the queue.
let answer: ipc::StatusFn = {
let ctx = ctx.clone();
Arc::new(move || status(&ctx))
Arc::new(move || {
let ctx = ctx.clone();
Box::pin(async move { status(&ctx).await })
})
};
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
@@ -427,7 +439,7 @@ async fn daemon(
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
tracing::info!(
feeds = subscriptions(&ctx).map(|s| s.len()).unwrap_or(0),
feeds = subscriptions(&ctx).await.map(|s| s.len()).unwrap_or(0),
"daemon started"
);
@@ -571,7 +583,7 @@ async fn add(
let mut cfg = (*ctx.cfg()).clone();
let url = &feed::expand_input(url);
// Includes feeds derived from an OPML, or the same show could be added twice.
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
if let Some(existing) = subscriptions(ctx).await?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
anyhow::bail!("already subscribed as {:?}", existing.id);
}
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
@@ -623,13 +635,13 @@ pub async fn add_one(
// Slugs must be unique across derived feeds too, or a new feed can collide with one
// an OPML already introduced.
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx).await?
.into_iter()
.map(|s| (s.id, s.cfg))
.collect();
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
// it gets its history back, and a different feed does not inherit someone else's.
for (id, other) in ctx.db.feed_urls()? {
for (id, other) in ctx.db.feed_urls().await? {
if !feed::same_feed(&other, url) {
taken.entry(id).or_insert_with(|| probe.clone());
}
@@ -647,19 +659,19 @@ fn url_stem(url: &str) -> String {
.unwrap_or_else(|| url.to_owned())
}
fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
async fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
let mut cfg = (*ctx.cfg()).clone();
if cfg.feeds.remove(feed).is_none() {
// Derived from an OPML: drop it here, though the subscription will list it again
// on the next read unless the OPML itself goes.
ctx.db.drop_managed(feed)?;
ctx.db.drop_managed(feed).await?;
println!("removed {feed}; it came from an OPML subscription and may return on the next read");
return Ok(());
}
cfg.save(config_path)?;
// State and files stay: re-adding the feed should not re-download its back catalogue.
println!("removed {feed}; downloads and history kept");
retire_group(ctx, feed)?;
retire_group(ctx, feed).await?;
Ok(())
}
@@ -669,13 +681,13 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
// The CLI speaks for the operator, as the shared web token does.
let admin = ctx
.db
.users()?
.users().await?
.into_iter()
.find(|u| u.is_admin)
.ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?;
let doc = opml::OPML::from_str(&text)
.map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?;
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id)?;
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id).await?;
println!("subscribed {} to {added} feed(s); {had} already there", admin.name);
Ok(())
}
@@ -690,7 +702,7 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
///
/// The caller parses the document, so each refuses a file that is not OPML in its own terms,
/// before anything is touched: a 400 from the web, a message from the CLI.
pub fn subscribe_opml(
pub async fn subscribe_opml(
ctx: &Ctx,
config_path: &std::path::Path,
doc: &opml::OPML,
@@ -699,7 +711,7 @@ pub fn subscribe_opml(
let mut found = vec![];
collect_outlines(&doc.body.outlines, &mut found);
let known = subscriptions(ctx)?;
let known = subscriptions(ctx).await?;
let mut cfg = (*ctx.cfg()).clone();
let mut ids = vec![];
let mut grew = false;
@@ -745,10 +757,10 @@ pub fn subscribe_opml(
let (mut added, mut had) = (0, 0);
for id in ids {
if ctx.db.subscription(user_id, &id)?.is_some() {
if ctx.db.subscription(user_id, &id).await?.is_some() {
had += 1;
} else {
ctx.db.subscribe(user_id, &id)?;
ctx.db.subscribe(user_id, &id).await?;
added += 1;
}
}
@@ -766,7 +778,16 @@ pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, Strin
}
}
fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
async fn copy_db(ctx: &Ctx, from: &std::path::Path) -> Result<()> {
anyhow::ensure!(from.exists(), "{} does not exist", from.display());
let source = db::Db::open(&from.display().to_string()).await?;
for (table, n) in ctx.db.copy_from(&source).await? {
println!("{table:14} {n}");
}
Ok(())
}
async fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
let mut doc = opml::OPML::default();
doc.head = Some(opml::Head {
title: Some("ipx subscriptions".into()),
@@ -775,7 +796,7 @@ fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
for (id, feed) in &ctx.cfg().feeds {
let title = ctx
.db
.feed_summary(id)
.feed_summary(id).await
.ok()
.and_then(|s| s.title)
.unwrap_or_else(|| id.clone());
@@ -787,14 +808,14 @@ fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
Ok(())
}
fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
async fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
let cfg = ctx.cfg();
if cfg.feeds.is_empty() {
println!("No feeds configured in {}", config_path.display());
return Ok(());
}
for (id, feed) in &cfg.feeds {
let s = ctx.db.feed_summary(id)?;
let s = ctx.db.feed_summary(id).await?;
println!("{id} {}", s.title.as_deref().unwrap_or("-"));
println!(" url {}", feed.url);
println!(" last checked {}", ago(s.last_checked));
@@ -809,8 +830,8 @@ fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
/// `standalone` false means this is the sweep that runs before a scan: it reports what it
/// deleted, but must not emit the terminal ReapDone, or a client waiting on its `fetch`
/// would stop reading before the scan had even started.
fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
let r = retention::run(&ctx.cfg(), &ctx.db, dry_run)?;
async fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
let r = retention::run(&ctx.cfg(), &ctx.db, dry_run).await?;
for c in r.aged_out.iter().chain(r.over_quota.iter()) {
ctx.out.emit(Event::Reaped {
path: c.path.clone(),
@@ -828,7 +849,7 @@ fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
let cfg = ctx.cfg();
let subs = subscriptions(ctx)?;
let subs = subscriptions(ctx).await?;
if let Some(id) = only
&& !subs.iter().any(|s| s.id == id)
{
@@ -839,7 +860,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
let mut fresh: Vec<String> = vec![];
for sub in subs.iter().filter(|s| only.is_none_or(|o| o == s.id)) {
let (id, feed_cfg) = (&sub.id, &sub.cfg);
let state = ctx.db.http_state(id)?;
let state = ctx.db.http_state(id).await?;
if !force && let Some(last) = state.last_checked {
let due = last + due_after(&cfg, feed_cfg, state.ttl_mins) as i64;
@@ -886,20 +907,20 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
// One bad feed must not end the scan.
let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg)?;
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
}
}
}
// Feeds a subscribed OPML just introduced: scan them now, in this run.
if !fresh.is_empty() {
let subs = subscriptions(ctx)?;
let subs = subscriptions(ctx).await?;
for id in &fresh {
let Some(feed_cfg) = subs.iter().find(|s| &s.id == id).map(|s| &s.cfg) else {
continue;
};
scanned += 1;
ctx.out.emit(Event::FeedStart { feed: id.clone() });
let state = ctx.db.http_state(id)?;
let state = ctx.db.http_state(id).await?;
match scan_one(ctx, id, feed_cfg, &state).await {
Ok(Outcome::Feed(s)) => ctx.out.emit(Event::FeedDone {
feed: id.clone(),
@@ -912,7 +933,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
Err(e) => {
let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg)?;
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
}
}
}
@@ -934,7 +955,7 @@ pub struct Sub {
///
/// A derived feed borrows its parent's settings wholesale. That is why it needs no config
/// entry -- there is nothing to store but its URL and where it came from.
pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
pub async fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
let cfg = ctx.cfg();
let mut out: Vec<Sub> = cfg
.feeds
@@ -942,7 +963,7 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
.map(|(id, f)| Sub { id: id.clone(), cfg: f.clone(), managed: false })
.collect();
for m in ctx.db.managed_feeds()? {
for m in ctx.db.managed_feeds().await? {
if cfg.feeds.contains_key(&m.id) {
continue; // promoted to config at some point; that entry wins
}
@@ -953,10 +974,10 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
// skip them here regardless so a row that slips through is never scanned.
continue;
}
let base = parent
.and_then(|p| p.folder.clone())
.or_else(|| ctx.db.feed_summary(&m.group_id).ok().and_then(|s| s.title))
.unwrap_or_else(|| m.group_id.clone());
let base = match parent.and_then(|p| p.folder.clone()) {
Some(folder) => folder,
None => ctx.db.feed_summary(&m.group_id).await.ok().and_then(|s| s.title).unwrap_or_else(|| m.group_id.clone()),
};
let title = m.title.clone().unwrap_or_else(|| m.id.clone());
out.push(Sub {
id: m.id.clone(),
@@ -988,17 +1009,17 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
/// itself is removed, since `subscriptions()` would otherwise keep scanning them under a
/// fallback policy meant for a feed with no parent at all. A feed promoted to config is not
/// derived any more, so it is only unmanaged.
pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
pub async fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
let cfg = ctx.cfg();
for m in ctx.db.managed_feeds()?.into_iter().filter(|m| m.group_id == parent_id) {
for m in ctx.db.managed_feeds().await?.into_iter().filter(|m| m.group_id == parent_id) {
if cfg.feeds.contains_key(&m.id) {
// Scanned from its config entry and still read. Dropped as derived, its stored
// entries would go with it: davewiner's 11 were promoted without being unmanaged.
ctx.db.unmanage(&m.id)?;
} else if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 {
ctx.db.set_orphaned(&m.id, true)?;
ctx.db.unmanage(&m.id).await?;
} else if ctx.db.downloaded_count(&m.id).await.unwrap_or(1) > 0 {
ctx.db.set_orphaned(&m.id, true).await?;
} else {
ctx.db.drop_managed(&m.id)?;
ctx.db.drop_managed(&m.id).await?;
}
}
Ok(())
@@ -1008,18 +1029,18 @@ pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
/// dropped or unmanaged. An OPML removed before `retire_group` existed left its feeds behind:
/// davewiner's 922 were skipped by every scan and never cleared, and their stale errors were
/// most of the ones stored.
fn retire_stranded(ctx: &Ctx) -> Result<usize> {
async fn retire_stranded(ctx: &Ctx) -> Result<usize> {
let cfg = ctx.cfg();
let before = ctx.db.managed_feeds()?;
let before = ctx.db.managed_feeds().await?;
let stranded: std::collections::BTreeSet<&str> = before
.iter()
.map(|m| m.group_id.as_str())
.filter(|g| !cfg.feeds.contains_key(*g))
.collect();
for group in stranded {
retire_group(ctx, group)?;
retire_group(ctx, group).await?;
}
Ok(before.len() - ctx.db.managed_feeds()?.len())
Ok(before.len() - ctx.db.managed_feeds().await?.len())
}
/// Seconds to wait before re-checking a feed.
@@ -1064,19 +1085,19 @@ async fn scan_one(
if feed::is_patreon_creator(&feed_cfg.url) {
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
Ok((name, shows)) if shows.len() > 1 => {
ctx.db.touch_feed(id, &feed_cfg.url)?;
ctx.db.touch_feed(id, &feed_cfg.url).await?;
if let Some(name) = name {
ctx.db.set_title(id, &name)?;
ctx.db.set_title(id, &name).await?;
}
// Read as one feed before it was split, it listed every show's items in one
// heap. The items go; its files and read state move to each show as the show
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
ctx.db.clear_entries(id)?;
ctx.db.clear_entries(id).await?;
return sync_group(ctx, id, feed_cfg, &shows).await;
}
Ok(_) => {} // One show: the creator's feed is that show.
// Already split: keep the shows it has rather than read the creator as one heap.
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
Err(e) if ctx.db.managed_feeds().await?.iter().any(|m| m.group_id == id) => return Err(e),
Err(e) => tracing::warn!(
feed = id,
error = %format!("{e:#}"),
@@ -1097,29 +1118,29 @@ async fn scan_one(
// from backup, a manual edit, a cleanup that removed entries. Believe the database over
// the validator: drop it and ask again, or the feed stays empty until the publisher
// happens to change something.
if matches!(fetched, feed::Fetched::NotModified) && ctx.db.feed_summary(id)?.entries == 0 {
if matches!(fetched, feed::Fetched::NotModified) && ctx.db.feed_summary(id).await?.entries == 0 {
tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator");
ctx.db.clear_validators(id)?;
ctx.db.clear_validators(id).await?;
fetched = feed::fetch(&ctx.client, feed_cfg, None, None).await?;
}
let (bytes, etag, last_modified) = match fetched {
feed::Fetched::NotModified => {
ctx.db.touch_feed(id, &feed_cfg.url)?;
ctx.db.touch_feed(id, &feed_cfg.url).await?;
return Ok(Outcome::NotModified);
}
feed::Fetched::Body { bytes, etag, last_modified } => (bytes, etag, last_modified),
};
if bytes.iter().all(u8::is_ascii_whitespace) {
ctx.db.touch_feed(id, &feed_cfg.url)?;
ctx.db.touch_feed(id, &feed_cfg.url).await?;
return Ok(Outcome::Empty);
}
// A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml"
// URL; sniffing the body also catches one served from a URL without that extension.
if feed::is_opml(&bytes) {
ctx.db.touch_feed(id, &feed_cfg.url)?;
ctx.db.touch_feed(id, &feed_cfg.url).await?;
return sync_opml(ctx, id, feed_cfg, &bytes).await;
}
@@ -1133,29 +1154,29 @@ async fn scan_one(
parsed.ttl_mins,
parsed.image.as_deref(),
parsed.category.as_deref(),
)?;
).await?;
let policy = policy_for(ctx, id, feed_cfg)?;
let policy = policy_for(ctx, id, feed_cfg).await?;
if let Some(parent) = &feed_cfg.group {
let listed: Vec<(&str, &str)> = parsed
.entries
.iter()
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
.collect();
ctx.db.adopt(parent, id, &listed)?;
ctx.db.adopt(parent, id, &listed).await?;
}
// Verdicts are recorded in `state`, so the download queue below is just "everything still
// pending". A filter's verdict is looked at again on every scan, though: made once, at
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
// changed nothing however often the feed was scanned.
let skipped = ctx.db.skipped_by_filter(id)?;
let skipped = ctx.db.skipped_by_filter(id).await?;
let mut scan = Scan::default();
for entry in &parsed.entries {
if ctx.db.record_entry(id, entry)? {
if ctx.db.record_entry(id, entry).await? {
scan.new_entries += 1;
}
for enc in &entry.enclosures {
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
let was = if ctx.db.record_enclosure(id, &entry.guid, enc).await? {
None
} else if let Some(reason) = skipped.get(&enc.url) {
Some(reason.as_str())
@@ -1165,8 +1186,8 @@ async fn scan_one(
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
if now != was {
match now {
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason)).await?,
None => ctx.db.mark_enclosure(&enc.url, "pending", None).await?,
}
}
}
@@ -1178,10 +1199,10 @@ async fn scan_one(
let folder = download::folder_for(&cfg, id, feed_cfg, parsed.title.as_deref());
let dest_dir = cfg.general.download_dir.join(&folder);
for item in ctx.db.pending(id, budget)? {
for item in ctx.db.pending(id, budget).await? {
if download::looks_like_torrent(&item.url, item.mime.as_deref()) {
if !ctx.cfg().torrent.enabled {
ctx.db.mark_enclosure(&item.url, "skipped", Some("torrents disabled"))?;
ctx.db.mark_enclosure(&item.url, "skipped", Some("torrents disabled")).await?;
ctx.out.emit(Event::TorrentDeferred {
feed: id.to_string(),
url: item.url.clone(),
@@ -1191,14 +1212,14 @@ async fn scan_one(
}
if ctx.detach_torrents {
// 'downloading' keeps the next scan from queueing it a second time.
ctx.db.mark_enclosure(&item.url, "downloading", None)?;
ctx.db.mark_enclosure(&item.url, "downloading", None).await?;
spawn_torrent(ctx, id.to_string(), item.id, item.url.clone(), dest_dir.clone());
scan.torrents += 1;
continue;
}
match torrent_one(ctx, id, item.id, &item.url, &dest_dir).await {
Ok((path, bytes)) => {
ctx.db.mark_downloaded(&item.url, &path, bytes)?;
ctx.db.mark_downloaded(&item.url, &path, bytes).await?;
ctx.out.emit(Event::DownloadDone {
feed: id.to_string(),
enclosure: item.id,
@@ -1216,7 +1237,7 @@ async fn scan_one(
url: item.url.clone(),
msg: msg.clone(),
});
ctx.db.mark_enclosure(&item.url, "error", Some(&msg))?;
ctx.db.mark_enclosure(&item.url, "error", Some(&msg)).await?;
scan.failed += 1;
}
}
@@ -1241,7 +1262,7 @@ async fn scan_one(
url: item.url.clone(),
msg: msg.clone(),
});
ctx.db.mark_enclosure(&item.url, "error", Some(&msg))?;
ctx.db.mark_enclosure(&item.url, "error", Some(&msg)).await?;
scan.failed += 1;
}
}
@@ -1259,7 +1280,7 @@ async fn sync_opml(
) -> Result<Outcome> {
let listed = feed::parse_opml(bytes)?;
if let Some(title) = feed::opml_title(bytes) {
ctx.db.set_title(parent_id, &title)?;
ctx.db.set_title(parent_id, &title).await?;
}
sync_group(ctx, parent_id, parent, &listed).await
}
@@ -1278,13 +1299,13 @@ async fn sync_group(
listed: &[(String, String)],
) -> Result<Outcome> {
let cfg = ctx.cfg();
let existing = ctx.db.managed_feeds()?;
let existing = ctx.db.managed_feeds().await?;
let mut added = vec![];
for (title, url) in listed {
// Already known, whether derived or promoted into the config.
if let Some(m) = existing.iter().find(|m| &m.url == url) {
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
ctx.db.upsert_managed(&m.id, url, title, parent_id).await?;
continue;
}
// A Patreon show you added by hand may be spelled differently from the one listed.
@@ -1292,7 +1313,7 @@ async fn sync_group(
continue;
}
// A removed feed keeps its rows, so its id is only free again for the same feed.
let known = ctx.db.feed_urls()?;
let known = ctx.db.feed_urls().await?;
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
.feeds
.keys()
@@ -1302,7 +1323,7 @@ async fn sync_group(
.map(|id| (id.clone(), parent.clone()))
.collect();
let id = config::unique_slug(title, &taken);
ctx.db.upsert_managed(&id, url, title, parent_id)?;
ctx.db.upsert_managed(&id, url, title, parent_id).await?;
added.push(id);
}
@@ -1310,15 +1331,15 @@ async fn sync_group(
// subscription means. Their own feeds are untouched.
for id in ctx
.db
.managed_feeds()?
.managed_feeds().await?
.iter()
.filter(|m| m.group_id == parent_id)
.map(|m| m.id.clone())
.chain(std::iter::once(parent_id.to_string()))
{
for user in ctx.db.users()? {
if ctx.db.subscription(user.id, parent_id)?.is_some() {
ctx.db.subscribe(user.id, &id)?;
for user in ctx.db.users().await? {
if ctx.db.subscription(user.id, parent_id).await?.is_some() {
ctx.db.subscribe(user.id, &id).await?;
}
}
}
@@ -1330,13 +1351,13 @@ async fn sync_group(
if listed.iter().any(|(_, u)| u == &m.url) {
continue;
}
if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 {
if ctx.db.downloaded_count(&m.id).await.unwrap_or(1) > 0 {
// Never orphan a downloaded file: keep the feed and say why in the UI.
ctx.db.set_orphaned(&m.id, true)?;
ctx.db.set_orphaned(&m.id, true).await?;
kept += 1;
tracing::info!(feed = %m.id, "dropped from the OPML but has downloads; keeping it");
} else {
ctx.db.drop_managed(&m.id)?;
ctx.db.drop_managed(&m.id).await?;
removed += 1;
tracing::info!(feed = %m.id, "dropped from the OPML with nothing downloaded; removed");
}
@@ -1403,9 +1424,9 @@ pub struct Policy {
pub budget: usize,
}
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
async fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
let global = ctx.cfg().general.max_new_per_check;
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref()).await?, feed_cfg, global))
}
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
@@ -1484,14 +1505,14 @@ async fn fetch_one(
// as if it were an episode.
let _ = tokio::fs::remove_file(&got.tmp).await;
if !ctx.cfg().torrent.enabled {
ctx.db.mark_enclosure(url, "skipped", Some("torrents disabled"))?;
ctx.db.mark_enclosure(url, "skipped", Some("torrents disabled")).await?;
anyhow::bail!("body is a torrent and torrents are disabled");
}
return torrent_one(ctx, feed_id, enclosure, url, dest_dir).await;
}
let path = download::place(&got, dest_dir).await?;
ctx.db.mark_downloaded(url, &path, got.bytes)?;
ctx.db.mark_downloaded(url, &path, got.bytes).await?;
Ok((path, got.bytes))
}
@@ -1509,7 +1530,7 @@ fn spawn_torrent(ctx: &Arc<Ctx>, feed_id: String, enclosure: i64, url: String, d
let db = &ctx.db;
match outcome {
Ok((path, bytes)) => {
if let Err(e) = db.mark_downloaded(&url, &path, bytes) {
if let Err(e) = db.mark_downloaded(&url, &path, bytes).await {
tracing::warn!(error = ?e, "could not record the finished torrent");
}
ctx.out.emit(Event::DownloadDone {
@@ -1522,7 +1543,7 @@ fn spawn_torrent(ctx: &Arc<Ctx>, feed_id: String, enclosure: i64, url: String, d
}
Err(e) => {
let msg = format!("{e:#}");
let _ = db.mark_enclosure(&url, "error", Some(&msg));
let _ = db.mark_enclosure(&url, "error", Some(&msg)).await;
ctx.out.emit(Event::DownloadError { feed: feed_id, enclosure, url, msg });
}
}
@@ -1535,14 +1556,14 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
let cfg = ctx.cfg();
let enc = ctx
.db
.enclosure(id)?
.enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
if enc.path.is_some() {
return Ok(()); // Already here.
}
// Must look through the derived feeds too: anything inside an OPML subscription has
// no config entry, so a config-only lookup called every one of them "unsubscribed".
let subs = subscriptions(ctx)?;
let subs = subscriptions(ctx).await?;
let feed_cfg = subs
.iter()
.find(|s| s.id == enc.feed_id)
@@ -1550,14 +1571,14 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
.ok_or_else(|| anyhow::anyhow!("enclosure {id} belongs to unsubscribed feed {:?}", enc.feed_id))?;
let feed_cfg = &feed_cfg;
let title = ctx.db.feed_summary(&enc.feed_id)?.title;
let title = ctx.db.feed_summary(&enc.feed_id).await?.title;
let folder = download::folder_for(&cfg, &enc.feed_id, feed_cfg, title.as_deref());
let dest_dir = cfg.general.download_dir.join(&folder);
ctx.out.emit(Event::FeedStart { feed: enc.feed_id.clone() });
let is_torrent = download::looks_like_torrent(&enc.url, enc.mime.as_deref());
if is_torrent && cfg.torrent.enabled && ctx.detach_torrents {
ctx.db.mark_enclosure(&enc.url, "downloading", None)?;
ctx.db.mark_enclosure(&enc.url, "downloading", None).await?;
spawn_torrent(ctx, enc.feed_id.clone(), enc.id, enc.url.clone(), dest_dir);
return Ok(());
}
@@ -1573,7 +1594,7 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
match result {
Ok((path, bytes)) => {
ctx.db.mark_downloaded(&enc.url, &path, bytes)?;
ctx.db.mark_downloaded(&enc.url, &path, bytes).await?;
ctx.out.emit(Event::DownloadDone {
feed: enc.feed_id.clone(),
enclosure: enc.id,
@@ -1584,7 +1605,7 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
}
Err(e) => {
let msg = format!("{e:#}");
ctx.db.mark_enclosure(&enc.url, "error", Some(&msg))?;
ctx.db.mark_enclosure(&enc.url, "error", Some(&msg)).await?;
ctx.out.emit(Event::DownloadError {
feed: enc.feed_id.clone(),
enclosure: enc.id,
@@ -1690,8 +1711,8 @@ mod tests {
}
}
#[test]
fn a_shared_feed_is_fetched_for_whoever_wants_the_most() {
#[tokio::test]
async fn a_shared_feed_is_fetched_for_whoever_wants_the_most() {
// Nobody subscribed: the feed's own settings stand, as in a single-user install.
let p = merge_policy(&[], &feed(), 3);
assert!(p.auto_download);
@@ -1721,10 +1742,10 @@ mod tests {
assert!(p.auto_download);
}
fn test_ctx(cfg: config::Config) -> Ctx {
async fn test_ctx(cfg: config::Config) -> Ctx {
Ctx {
cfg: std::sync::RwLock::new(Arc::new(cfg)),
db: db::Db::memory().unwrap(),
db: db::Db::memory().await.unwrap(),
client: reqwest::Client::new(),
out: Emitter::terminal(),
torrents: tokio::sync::OnceCell::new(),
@@ -1734,52 +1755,52 @@ mod tests {
}
}
#[test]
fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() {
#[tokio::test]
async fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() {
// davewiner: the OPML subscription left config.toml, but its 922 derived rows
// stayed in the database and kept being scanned under the no-parent fallback.
let ctx = test_ctx(config::Config::default());
ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").unwrap();
let ctx = test_ctx(config::Config::default()).await;
ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").await.unwrap();
assert!(
subscriptions(&ctx).unwrap().iter().all(|s| s.id != "child"),
subscriptions(&ctx).await.unwrap().iter().all(|s| s.id != "child"),
"a derived feed whose parent is gone from config must not be scanned"
);
}
#[test]
fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() {
let ctx = test_ctx(config::Config::default());
ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").unwrap();
ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").unwrap();
#[tokio::test]
async fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() {
let ctx = test_ctx(config::Config::default()).await;
ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").await.unwrap();
ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").await.unwrap();
let enc = feed::Enclosure { url: "http://x/ep.mp3".into(), mime: None, length: None };
ctx.db.record_enclosure("has-file", "g1", &enc).unwrap();
ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).unwrap();
ctx.db.record_enclosure("has-file", "g1", &enc).await.unwrap();
ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).await.unwrap();
retire_group(&ctx, "parent").unwrap();
retire_group(&ctx, "parent").await.unwrap();
let managed = ctx.db.managed_feeds().unwrap();
let managed = ctx.db.managed_feeds().await.unwrap();
assert!(!managed.iter().any(|m| m.id == "empty"), "nothing downloaded, so it is forgotten");
assert!(managed.iter().any(|m| m.id == "has-file"), "has a file on disk, so it is kept");
assert!(ctx.db.feed_summary("has-file").unwrap().orphaned, "and flagged as orphaned");
assert!(ctx.db.feed_summary("has-file").await.unwrap().orphaned, "and flagged as orphaned");
}
#[test]
fn a_stranded_group_is_retired_but_a_promoted_feed_keeps_its_entries() {
#[tokio::test]
async fn a_stranded_group_is_retired_but_a_promoted_feed_keeps_its_entries() {
// davewiner: the OPML left config before retire_group existed, and 11 of its feeds
// promoted to config since still said managed = 1.
let mut cfg = config::Config::default();
cfg.feeds.insert("promoted".into(), feed());
cfg.feeds.insert("live-opml".into(), feed());
let ctx = test_ctx(cfg);
ctx.db.upsert_managed("promoted", "http://x/p.xml", "Promoted", "gone-opml").unwrap();
ctx.db.upsert_managed("empty", "http://x/e.xml", "Empty", "gone-opml").unwrap();
ctx.db.upsert_managed("listed", "http://x/l.xml", "Listed", "live-opml").unwrap();
ctx.db.record_entry("promoted", &feed::Entry { guid: "g1".into(), ..Default::default() }).unwrap();
let ctx = test_ctx(cfg).await;
ctx.db.upsert_managed("promoted", "http://x/p.xml", "Promoted", "gone-opml").await.unwrap();
ctx.db.upsert_managed("empty", "http://x/e.xml", "Empty", "gone-opml").await.unwrap();
ctx.db.upsert_managed("listed", "http://x/l.xml", "Listed", "live-opml").await.unwrap();
ctx.db.record_entry("promoted", &feed::Entry { guid: "g1".into(), ..Default::default() }).await.unwrap();
assert_eq!(retire_stranded(&ctx).unwrap(), 2, "empty dropped, promoted unmanaged");
assert_eq!(retire_stranded(&ctx).await.unwrap(), 2, "empty dropped, promoted unmanaged");
let managed: Vec<String> = ctx.db.managed_feeds().unwrap().into_iter().map(|m| m.id).collect();
let managed: Vec<String> = ctx.db.managed_feeds().await.unwrap().into_iter().map(|m| m.id).collect();
assert_eq!(managed, ["listed"], "a group still in config is left alone");
assert_eq!(ctx.db.feed_summary("promoted").unwrap().entries, 1, "its entries survive");
assert_eq!(ctx.db.feed_summary("promoted").await.unwrap().entries, 1, "its entries survive");
}
}

View File

@@ -45,28 +45,28 @@ pub fn aged(candidates: &[Candidate], cutoff: i64) -> Vec<Candidate> {
.collect()
}
pub fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
pub async fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
let mut report = Report::default();
// Someone may have deleted a file by hand; the row must stop claiming it exists.
for (id, path) in db.missing_files()? {
for (id, path) in db.missing_files().await? {
if !dry_run {
db.mark_reaped(id)?;
db.mark_reaped(id).await?;
}
tracing::debug!(path, "file gone, row reaped");
report.reconciled += 1;
}
let candidates = db.reap_candidates()?;
let candidates = db.reap_candidates().await?;
if cfg.general.max_age_days > 0 {
let cutoff = now() - (cfg.general.max_age_days * 86_400) as i64;
report.aged_out = aged(&candidates, cutoff);
for c in &report.aged_out {
report.bytes_freed += remove(db, c, dry_run)?;
report.bytes_freed += remove(db, c, dry_run).await?;
}
if !dry_run {
report.entries_pruned = db.prune_entries(cutoff)?;
report.entries_pruned = db.prune_entries(cutoff).await?;
}
}
@@ -81,14 +81,14 @@ pub fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
let total: u64 = remaining.iter().map(|c| c.bytes.max(0) as u64).sum();
report.over_quota = pick(&remaining, total, limit);
for c in &report.over_quota {
report.bytes_freed += remove(db, c, dry_run)?;
report.bytes_freed += remove(db, c, dry_run).await?;
}
}
Ok(report)
}
fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
async fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
if dry_run {
return Ok(c.bytes.max(0) as u64);
}
@@ -100,7 +100,7 @@ fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
tracing::warn!(path = c.path, error = %e, "could not delete");
return Ok(0);
}
db.mark_reaped(c.id)?;
db.mark_reaped(c.id).await?;
Ok(size)
}
@@ -149,12 +149,12 @@ mod tests {
// age_key 0 means "never recorded" -- not the same as "infinitely old".
}
#[test]
fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
#[tokio::test]
async fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
// One file serves both subscribers, so it takes both of them to release it.
let db = Db::memory().unwrap();
let db = Db::memory().await.unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true),(2,'sam',false);
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
INSERT INTO entries (feed_id, guid, first_seen) VALUES
('f', 'keep', 0),
@@ -163,20 +163,20 @@ mod tests {
('f', 'read', 0);
-- Starred by one of the two, so it stays whatever the other thinks.
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
(1, 'f', 'keep', 1, 1),
(2, 'f', 'keep', 1, 0),
(1, 'f', 'half', 1, 0),
(1, 'f', 'read', 1, 0),
(2, 'f', 'read', 1, 0);
(1, 'f', 'keep', true, true),
(2, 'f', 'keep', true, false),
(1, 'f', 'half', true, false),
(1, 'f', 'read', true, false),
(2, 'f', 'read', true, false);
INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES
(1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10),
(2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20),
(3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30),
(4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);",
)
).await
.unwrap();
let got: Vec<i64> = db.reap_candidates().unwrap().iter().map(|c| c.id).collect();
let got: Vec<i64> = db.reap_candidates().await.unwrap().iter().map(|c| c.id).collect();
assert_eq!(
got,
vec![4, 2, 3],
@@ -185,12 +185,12 @@ mod tests {
);
}
#[test]
fn prune_keeps_entries_that_still_have_a_file() {
let db = Db::memory().unwrap();
#[tokio::test]
async fn prune_keeps_entries_that_still_have_a_file() {
let db = Db::memory().await.unwrap();
db.exec_for_test(
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true);
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',true);
INSERT INTO entries (feed_id, guid, first_seen) VALUES
('f', 'has-file', 100),
('f', 'no-file', 100),
@@ -198,9 +198,9 @@ mod tests {
('f', 'recent', 900);
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
(1, 'f', 'has-file', 'u1', '/tmp/x', 'done');",
)
).await
.unwrap();
assert_eq!(db.prune_entries(500).unwrap(), 1, "only the old, fileless, unflagged one");
assert_eq!(db.prune_entries(500).await.unwrap(), 1, "only the old, fileless, unflagged one");
}
}

View File

@@ -107,16 +107,16 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
let mut user = None;
if let Some(name) = vouched {
user = match state.ctx.db.user_by_name(&name) {
user = match state.ctx.db.user_by_name(&name).await {
Ok(Some(u)) => Some(u),
Ok(None) if cfg.web.auto_create_users => {
tracing::info!(user = %name, "creating an account for a name the proxy vouched for");
state
.ctx
.db
.create_user(&name, None, state.ctx.db.users().map(|u| u.is_empty()).unwrap_or(false))
.ok()
.and_then(|id| state.ctx.db.user_by_id(id).ok().flatten())
// The first account made is the admin.
let first = state.ctx.db.users().await.map(|u| u.is_empty()).unwrap_or(false);
match state.ctx.db.create_user(&name, None, first).await {
Ok(id) => state.ctx.db.user_by_id(id).await.ok().flatten(),
Err(_) => None,
}
}
Ok(None) => {
tracing::warn!(user = %name, "proxy vouched for an unknown name and auto_create_users is off");
@@ -130,7 +130,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
// Every request comes vouched for; signed_in keeps one an hour. Failing to note the time
// must not turn anyone away, so its error goes unanswered.
if let Some(u) = &user {
let _ = state.ctx.db.signed_in(u.id);
let _ = state.ctx.db.signed_in(u.id).await;
}
}
let by_proxy = user.is_some();
@@ -141,7 +141,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
user = state
.ctx
.db
.session_user(&sid, cfg.web.session_days.max(1) * 86_400)
.session_user(&sid, cfg.web.session_days.max(1) * 86_400).await
.unwrap_or(None);
}
}
@@ -155,11 +155,11 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
if user.is_none() && !token.is_empty() {
let supplied = from_query.clone().or_else(|| cookie(&req, COOKIE));
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
user = admin_user(&state);
user = admin_user(&state).await;
if from_query.is_some() {
// The token link is a sign-in; the cookie it leaves behind is not one each time.
if let Some(u) = &user {
let _ = state.ctx.db.signed_in(u.id);
let _ = state.ctx.db.signed_in(u.id).await;
}
set_cookie = Some(format!(
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
@@ -247,8 +247,8 @@ fn cookie(req: &Request, name: &str) -> Option<String> {
}
/// The account the shared token stands for: the first admin, or the first user at all.
fn admin_user(state: &WebState) -> Option<crate::db::User> {
let users = state.ctx.db.users().ok()?;
async fn admin_user(state: &WebState) -> Option<crate::db::User> {
let users = state.ctx.db.users().await.ok()?;
users
.iter()
.find(|u| u.is_admin)
@@ -267,7 +267,7 @@ async fn login(
Json(body): Json<Credentials>,
) -> Result<Response, ApiError> {
let name = body.name.trim().to_ascii_lowercase();
let user = state.ctx.db.user_by_name(&name)?;
let user = state.ctx.db.user_by_name(&name).await?;
// The same answer either way: whether a name exists is not something to leak.
let ok = user
.as_ref()
@@ -280,8 +280,8 @@ async fn login(
let user = user.expect("verified above");
let token = crate::auth::new_session_token();
state.ctx.db.create_session(user.id, &token)?;
state.ctx.db.signed_in(user.id)?;
state.ctx.db.create_session(user.id, &token).await?;
state.ctx.db.signed_in(user.id).await?;
tracing::info!(user = %user.name, "signed in");
let days = state.ctx.cfg().web.session_days.max(1);
@@ -298,7 +298,7 @@ async fn login(
async fn logout(State(state): State<WebState>, req: Request) -> Response {
if let Some(sid) = cookie(&req, SESSION_COOKIE) {
let _ = state.ctx.db.delete_session(&sid);
let _ = state.ctx.db.delete_session(&sid).await;
}
let mut resp = StatusCode::NO_CONTENT.into_response();
for c in [
@@ -320,7 +320,7 @@ async fn me(
) -> Json<serde_json::Value> {
let url = state.ctx.cfg().web.sign_out_url.clone();
let sign_out = (by_proxy && !url.is_empty()).then_some(url);
let (theme, mode) = state.ctx.db.theme(user.id).unwrap_or_default();
let (theme, mode) = state.ctx.db.theme(user.id).await.unwrap_or_default();
Json(serde_json::json!({
"name": user.name, "admin": user.is_admin, "sign_out": sign_out, "theme": theme, "mode": mode,
}))
@@ -343,7 +343,7 @@ async fn patch_me(
if !theme_ok(&body.theme, &body.mode) {
return Err(ApiError::bad_request("not a theme"));
}
state.ctx.db.set_theme(user.id, &body.theme, &body.mode)?;
state.ctx.db.set_theme(user.id, &body.theme, &body.mode).await?;
Ok(StatusCode::NO_CONTENT)
}
@@ -378,7 +378,7 @@ async fn list_users(
let users: Vec<_> = state
.ctx
.db
.users()?
.users().await?
.iter()
.map(|u| {
serde_json::json!({
@@ -409,7 +409,7 @@ async fn add_user(
let name = crate::auth::name_from_header(&body.name).ok_or_else(|| {
ApiError::bad_request("a name is required, without commas, semicolons or line breaks")
})?;
if state.ctx.db.user_by_name(&name)?.is_some() {
if state.ctx.db.user_by_name(&name).await?.is_some() {
return Err(ApiError::bad_request(format!("{name} already exists")));
}
// No password is someone the proxy signs in, as with `ipx user add --no-password`.
@@ -418,7 +418,7 @@ async fn add_user(
} else {
Some(crate::auth::hash_password(&body.password).map_err(|e| ApiError::bad_request(format!("{e:#}")))?)
};
state.ctx.db.create_user(&name, hash.as_deref(), body.admin)?;
state.ctx.db.create_user(&name, hash.as_deref(), body.admin).await?;
tracing::info!(by = %user.name, user = %name, admin = body.admin, "account added");
Ok(StatusCode::CREATED)
}
@@ -435,7 +435,7 @@ async fn patch_user(
Json(body): Json<UserPatch>,
) -> Result<StatusCode, ApiError> {
require_admin(&user)?;
let users = state.ctx.db.users()?;
let users = state.ctx.db.users().await?;
let target = users
.iter()
.find(|u| u.id == id)
@@ -446,7 +446,7 @@ async fn patch_user(
target.name
)));
}
state.ctx.db.set_admin(id, body.admin)?;
state.ctx.db.set_admin(id, body.admin).await?;
tracing::info!(by = %user.name, user = %target.name, admin = body.admin, "admin changed");
Ok(StatusCode::NO_CONTENT)
}
@@ -457,7 +457,7 @@ async fn remove_user(
Path(id): Path<i64>,
) -> Result<StatusCode, ApiError> {
require_admin(&user)?;
let users = state.ctx.db.users()?;
let users = state.ctx.db.users().await?;
let target = users
.iter()
.find(|u| u.id == id)
@@ -468,7 +468,7 @@ async fn remove_user(
target.name
)));
}
state.ctx.db.delete_user(id)?;
state.ctx.db.delete_user(id).await?;
tracing::info!(by = %user.name, user = %target.name, "account removed");
Ok(StatusCode::NO_CONTENT)
}
@@ -510,7 +510,7 @@ async fn admin_page(State(state): State<WebState>, user: crate::db::User) -> Res
if !user.is_admin {
return Redirect::to("/").into_response();
}
let theme = state.ctx.db.theme(user.id).unwrap_or_default();
let theme = state.ctx.db.theme(user.id).await.unwrap_or_default();
let page = with_theme(include_str!(concat!(env!("OUT_DIR"), "/admin.html")), theme);
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page)).into_response()
}
@@ -584,7 +584,7 @@ const ADMIN_LINK: &str = "<a id=admin ";
/// The page, with the log button left out for anyone but an admin. Hiding it from the page's
/// script instead showed it for a moment on every load, until /api/me answered.
async fn index(State(state): State<WebState>, user: crate::db::User) -> impl IntoResponse {
let theme = state.ctx.db.theme(user.id).unwrap_or_default();
let theme = state.ctx.db.theme(user.id).await.unwrap_or_default();
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page_for(user.is_admin, theme)))
}
@@ -672,16 +672,16 @@ async fn feeds(
let cfg = state.ctx.cfg();
// Config entries plus the feeds derived from OPML subscriptions -- the catalogue.
// What comes back is only the part of it this person subscribes to.
let subs = crate::subscriptions(&state.ctx)?;
let subs = crate::subscriptions(&state.ctx).await?;
let mine: std::collections::HashMap<String, crate::db::Sub> = state
.ctx
.db
.subscriptions_for(user.id)?
.subscriptions_for(user.id).await?
.into_iter()
.map(|s| (s.feed_id.clone(), s))
.collect();
let counts = state.ctx.db.subscriber_counts()?;
let pinned = state.ctx.db.pinned_feeds(user.id)?;
let counts = state.ctx.db.subscriber_counts().await?;
let pinned = state.ctx.db.pinned_feeds(user.id).await?;
let mut out = Vec::with_capacity(mine.len());
for sub in &subs {
let (id, feed) = (&sub.id, &sub.cfg);
@@ -689,8 +689,8 @@ async fn feeds(
// the same fallback the scanner uses (`Db::subscribers`).
let up = feed.group.as_deref().and_then(|g| mine.get(g));
let Some(mine) = mine.get(id) else { continue };
let s = state.ctx.db.feed_summary(id)?;
let st = state.ctx.db.http_state(id)?;
let s = state.ctx.db.feed_summary(id).await?;
let st = state.ctx.db.http_state(id).await?;
out.push(FeedRow {
id: id.clone(),
url: feed.url.clone(),
@@ -740,7 +740,7 @@ async fn feeds(
last_error: s.last_error,
entries: s.entries,
downloaded: s.downloaded,
unread: state.ctx.db.unread_count(user.id, id)?,
unread: state.ctx.db.unread_count(user.id, id).await?,
subscribers: counts.get(id).copied().unwrap_or(0),
pinned: pinned.contains(id),
});
@@ -800,13 +800,13 @@ struct PopularRow {
/// first. Popular is the top of it, the directory is all of it, and it is all that
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
/// feeds inside it and never itself: both lists are for finding a show.
fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
async fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
let db = &state.ctx.db;
let mine: std::collections::HashSet<String> =
db.subscriptions_for(user_id)?.into_iter().map(|s| s.feed_id).collect();
let counts = db.subscriber_counts()?;
let media = db.media_feeds()?;
let catalogue = crate::subscriptions(&state.ctx)?;
db.subscriptions_for(user_id).await?.into_iter().map(|s| s.feed_id).collect();
let counts = db.subscriber_counts().await?;
let media = db.media_feeds().await?;
let catalogue = crate::subscriptions(&state.ctx).await?;
let by_id: std::collections::HashMap<&str, &crate::config::Feed> =
catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect();
let is_folder: std::collections::HashSet<&str> =
@@ -823,7 +823,7 @@ fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
{
continue;
}
let sum = db.feed_summary(&s.id)?;
let sum = db.feed_summary(&s.id).await?;
let subscribed = mine.contains(&s.id);
out.push(PopularRow {
id: s.id.clone(),
@@ -844,7 +844,7 @@ async fn get_popular(
State(state): State<WebState>,
user: crate::db::User,
) -> Result<Json<Vec<PopularRow>>, ApiError> {
let mut rows = popular(&state, user.id)?;
let mut rows = popular(&state, user.id).await?;
rows.truncate(10);
Ok(Json(rows))
}
@@ -854,7 +854,7 @@ async fn get_directory(
State(state): State<WebState>,
user: crate::db::User,
) -> Result<Json<Vec<PopularRow>>, ApiError> {
let mut rows = popular(&state, user.id)?;
let mut rows = popular(&state, user.id).await?;
rows.sort_by_key(sort_name);
Ok(Json(rows))
}
@@ -870,10 +870,10 @@ async fn subscribe_popular(
user: crate::db::User,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> {
if !popular(&state, user.id)?.iter().any(|p| p.id == id) {
if !popular(&state, user.id).await?.iter().any(|p| p.id == id) {
return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
}
state.ctx.db.subscribe(user.id, &id)?;
state.ctx.db.subscribe(user.id, &id).await?;
Ok(Json(serde_json::json!({ "id": id })))
}
@@ -1124,7 +1124,7 @@ async fn entries(
user: crate::db::User,
Query(page): Query<Page>,
) -> Result<Json<EntryPage>, ApiError> {
entry_page(&state, user.id, Some(&id), &page)
entry_page(&state, user.id, Some(&id), &page).await
}
/// Every subscribed feed's items together, newest first: All Subscriptions.
@@ -1133,11 +1133,11 @@ async fn all_entries(
user: crate::db::User,
Query(page): Query<Page>,
) -> Result<Json<EntryPage>, ApiError> {
entry_page(&state, user.id, None, &page)
entry_page(&state, user.id, None, &page).await
}
/// One feed's page of items, or every subscribed feed's when `feed` is None.
fn entry_page(
async fn entry_page(
state: &WebState,
user_id: i64,
feed: Option<&str>,
@@ -1146,19 +1146,21 @@ fn entry_page(
let filter = crate::db::Filter::parse(page.filter.as_deref().unwrap_or("all"));
let search = page.q.as_deref().map(str::trim).filter(|q| !q.is_empty());
let db = &state.ctx.db;
// Currently Listening keeps its own order, pinned or not: it is what you are part-way through.
let order = crate::db::order_sql(
page.sort.as_deref().unwrap_or("published"),
page.dir.as_deref().unwrap_or("desc"),
filter != crate::db::Filter::InProgress,
);
let mut rows =
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order)?;
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order).await?;
let mut sanitizer = feed_sanitizer();
for row in &mut rows {
if let Some(d) = &row.description {
row.description = Some(clean_description(&mut sanitizer, d, row.link.as_deref()));
}
}
let total = db.count_in(user_id, feed, filter, search)?;
let total = db.count_in(user_id, feed, filter, search).await?;
Ok(Json(EntryPage { total, entries: rows }))
}
@@ -1200,10 +1202,10 @@ struct NewFeed {
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
/// goes on your subscription, and before the first scan, which would otherwise skip every
/// explicit item.
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
async fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
if allow {
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
state.ctx.db.set_subscription(user_id, &sub)?;
state.ctx.db.set_subscription(user_id, &sub).await?;
}
Ok(())
}
@@ -1217,14 +1219,14 @@ async fn add_feed(
let url = crate::feed::expand_input(&body.url);
// Someone else may already have it. Then adding costs nothing: no second fetch, no
// second copy on disk, just another name against the same feed.
if let Some(existing) = crate::subscriptions(&state.ctx)?
if let Some(existing) = crate::subscriptions(&state.ctx).await?
.into_iter()
.find(|s| crate::feed::same_feed(&s.cfg.url, &url))
{
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
state.ctx.db.subscribe(user.id, &existing.id)?;
let already = state.ctx.db.subscription(user.id, &existing.id).await?.is_some();
state.ctx.db.subscribe(user.id, &existing.id).await?;
if !already {
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?;
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit).await?;
}
scan_soon(&state, Some(existing.id.clone())).await;
return Ok(Json(
@@ -1234,8 +1236,8 @@ async fn add_feed(
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?;
state.ctx.db.subscribe(user.id, &id)?;
explicit_on_add(&state, user.id, &id, body.allow_explicit)?;
state.ctx.db.subscribe(user.id, &id).await?;
explicit_on_add(&state, user.id, &id, body.allow_explicit).await?;
scan_soon(&state, Some(id.clone())).await;
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
}
@@ -1288,17 +1290,17 @@ async fn patch_feed(
) -> Result<StatusCode, ApiError> {
// Pinning is yours alone too, and means nothing for a feed you do not subscribe to.
if let Some(on) = body.pinned
&& !state.ctx.db.set_pinned(user.id, &id, on)?
&& !state.ctx.db.set_pinned(user.id, &id, on).await?
{
return Err(ApiError::not_found("you do not subscribe to that feed"));
}
// What one person wants -- which items, whether to fetch them, how many at a time --
// is theirs. It goes on their subscription and nobody else sees the change.
if state.ctx.db.subscription(user.id, &id)?.is_some() {
if state.ctx.db.subscription(user.id, &id).await?.is_some() {
let mut mine = state
.ctx
.db
.subscription(user.id, &id)?
.subscription(user.id, &id).await?
.unwrap_or_else(|| crate::db::Sub { feed_id: id.clone(), ..Default::default() });
let mut touched = false;
if let Some(v) = body.keywords.clone() {
@@ -1318,7 +1320,7 @@ async fn patch_feed(
touched = true;
}
if touched {
state.ctx.db.set_subscription(user.id, &mine)?;
state.ctx.db.set_subscription(user.id, &mine).await?;
}
}
@@ -1339,13 +1341,13 @@ async fn patch_feed(
// Derived feeds have no config entry. Editing one is the moment it earns a real
// entry: promote it, so the config holds your decisions and nothing else.
if !cfg.feeds.contains_key(&id) {
let subs = crate::subscriptions(&state.ctx)?;
let subs = crate::subscriptions(&state.ctx).await?;
let found = subs
.iter()
.find(|s| s.id == id)
.ok_or_else(|| ApiError::bad_request(format!("no feed with id {id:?}")))?;
cfg.feeds.insert(id.clone(), found.cfg.clone());
state.ctx.db.unmanage(&id)?;
state.ctx.db.unmanage(&id).await?;
}
let checked = match &body.url {
@@ -1386,7 +1388,7 @@ async fn patch_feed(
if url_changed {
// Refreshing a rotated auth token is the common case; entries and download history
// are keyed by feed id, so they survive the change.
state.ctx.db.clear_validators(&id)?;
state.ctx.db.clear_validators(&id).await?;
}
Ok(StatusCode::NO_CONTENT)
}
@@ -1398,14 +1400,14 @@ async fn remove_feed(
) -> Result<StatusCode, ApiError> {
// Unsubscribing is personal: it takes the feed off your list and leaves everyone
// else's alone.
state.ctx.db.unsubscribe(user.id, &id)?;
for child in crate::subscriptions(&state.ctx)?
state.ctx.db.unsubscribe(user.id, &id).await?;
for child in crate::subscriptions(&state.ctx).await?
.iter()
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
{
state.ctx.db.unsubscribe(user.id, &child.id)?;
state.ctx.db.unsubscribe(user.id, &child.id).await?;
}
if state.ctx.db.subscriber_counts()?.contains_key(&id) {
if state.ctx.db.subscriber_counts().await?.contains_key(&id) {
return Ok(StatusCode::NO_CONTENT);
}
@@ -1415,12 +1417,12 @@ async fn remove_feed(
if cfg.feeds.remove(&id).is_none() {
// A derived feed: forget it here, though the OPML will list it again on the next
// read unless you unsubscribe from the OPML itself.
state.ctx.db.drop_managed(&id)?;
state.ctx.db.drop_managed(&id).await?;
return Ok(StatusCode::NO_CONTENT);
}
cfg.save(&state.config_path)?;
state.ctx.reload_cfg(&state.config_path)?;
crate::retire_group(&state.ctx, &id)?;
crate::retire_group(&state.ctx, &id).await?;
Ok(StatusCode::NO_CONTENT)
}
@@ -1438,10 +1440,10 @@ async fn set_flags(
) -> Result<StatusCode, ApiError> {
use crate::db::EntryFlag;
if let Some(v) = body.read {
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Read, v)?;
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Read, v).await?;
}
if let Some(v) = body.flagged {
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Flagged, v)?;
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Flagged, v).await?;
}
Ok(StatusCode::NO_CONTENT)
}
@@ -1456,12 +1458,12 @@ async fn download_now(
let enc = state
.ctx
.db
.enclosure(id)?
.enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
if enc.path.is_some() {
return Ok(StatusCode::NO_CONTENT); // Already here.
}
state.ctx.db.requeue(id)?;
state.ctx.db.requeue(id).await?;
state
.cmds
.send(Command::Download { enclosure: id })
@@ -1485,13 +1487,13 @@ async fn delete_file(
let enc = state
.ctx
.db
.enclosure(id)?
.enclosure(id).await?
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
// There is one copy of the file: deleting it deletes everyone's. Say so before doing
// it, once, and let them decide.
if !q.force {
let (starred, unread) = state.ctx.db.others_wanting(id, user.id)?;
let (starred, unread) = state.ctx.db.others_wanting(id, user.id).await?;
let people = |n: i64| if n == 1 { "person".to_string() } else { format!("{n} people") };
let complaint = match (starred, unread) {
(0, 0) => None,
@@ -1517,7 +1519,7 @@ async fn delete_file(
return Err(e.into());
}
// The row survives as 'reaped', which is what stops the next scan re-downloading it.
state.ctx.db.mark_reaped(id)?;
state.ctx.db.mark_reaped(id).await?;
state.events.send(Event::Reaped {
path: enc.path.unwrap_or_default(),
bytes: enc.length.unwrap_or(0).max(0) as u64,
@@ -1571,7 +1573,7 @@ async fn media(
Path(id): Path<i64>,
req: Request,
) -> Response {
let Ok(Some(enc)) = state.ctx.db.enclosure(id) else {
let Ok(Some(enc)) = state.ctx.db.enclosure(id).await else {
return (StatusCode::NOT_FOUND, "no such enclosure").into_response();
};
let Some(path) = enc.path else {
@@ -1596,7 +1598,7 @@ async fn set_position(
user: crate::db::User,
Json(body): Json<Position>,
) -> Result<StatusCode, ApiError> {
state.ctx.db.set_position(user.id, &feed_id, &guid, body.secs, body.duration)?;
state.ctx.db.set_position(user.id, &feed_id, &guid, body.secs, body.duration).await?;
Ok(StatusCode::NO_CONTENT)
}
@@ -1608,12 +1610,12 @@ async fn read_all(
// A subscription's own row has no entries, so marking it read means everything under it.
let mut ids = vec![id.clone()];
ids.extend(
crate::subscriptions(&state.ctx)?
crate::subscriptions(&state.ctx).await?
.into_iter()
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
.map(|s| s.id),
);
let n = state.ctx.db.mark_all_read(user.id, &ids)?;
let n = state.ctx.db.mark_all_read(user.id, &ids).await?;
Ok(Json(serde_json::json!({ "marked": n })))
}
@@ -1624,8 +1626,8 @@ async fn read_all_mine(
user: crate::db::User,
) -> Result<Json<serde_json::Value>, ApiError> {
let ids: Vec<String> =
state.ctx.db.subscriptions_for(user.id)?.into_iter().map(|s| s.feed_id).collect();
let n = state.ctx.db.mark_all_read(user.id, &ids)?;
state.ctx.db.subscriptions_for(user.id).await?.into_iter().map(|s| s.feed_id).collect();
let n = state.ctx.db.mark_all_read(user.id, &ids).await?;
Ok(Json(serde_json::json!({ "marked": n })))
}
@@ -1646,9 +1648,9 @@ async fn download_latest(
Path(id): Path<String>,
Json(body): Json<HowMany>,
) -> Result<Json<serde_json::Value>, ApiError> {
let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100))?;
let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100)).await?;
for enc in &ids {
state.ctx.db.requeue(*enc)?;
state.ctx.db.requeue(*enc).await?;
state
.cmds
.send(Command::Download { enclosure: *enc })
@@ -1666,7 +1668,7 @@ async fn export_opml(
// Yours, not the whole catalogue: other people's feeds, and any private URLs in them, are
// not yours to download. This used to export config.toml to whoever asked.
let mine: std::collections::HashSet<String> =
state.ctx.db.subscriptions_for(user.id)?.into_iter().map(|s| s.feed_id).collect();
state.ctx.db.subscriptions_for(user.id).await?.into_iter().map(|s| s.feed_id).collect();
let mut doc = opml::OPML {
head: Some(opml::Head {
title: Some("ipx subscriptions".into()),
@@ -1674,7 +1676,7 @@ async fn export_opml(
}),
..Default::default()
};
for s in crate::subscriptions(&state.ctx)? {
for s in crate::subscriptions(&state.ctx).await? {
// A feed from an OPML subscription comes back with the OPML itself.
if s.managed || !mine.contains(&s.id) {
continue;
@@ -1682,7 +1684,7 @@ async fn export_opml(
let title = state
.ctx
.db
.feed_summary(&s.id)
.feed_summary(&s.id).await
.ok()
.and_then(|sum| sum.title)
.unwrap_or_else(|| s.id.clone());
@@ -1716,7 +1718,7 @@ async fn import_opml(
// file arrives as text, is read here, and is gone when the request ends.
let doc = opml::OPML::from_str(&body.xml)
.map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?;
let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id)?;
let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id).await?;
if added > 0 {
scan_soon(&state, None).await;
}

View File

@@ -1299,3 +1299,20 @@ test('a pinned feed, even one from inside a folder, goes to the top of the list'
await expect(page.locator('.feed.pinned')).toHaveCount(0);
await expect(page.locator('.feed.child', { hasText: name })).toHaveCount(1);
});
test('a pinned item goes to the top of its list, and back when unpinned', async ({ page }) => {
await page.locator('.feed', { hasText: 'Test Show' }).click();
await page.locator('.tabs button', { hasText: 'All' }).first().click();
await expect(page.locator('.ep').nth(1)).toBeVisible({ timeout: 20_000 });
const guids = () => page.locator('.ep').evaluateAll(rows => rows.map(r => r.dataset.guid));
const before = await guids();
const last = before[before.length - 1];
const row = page.locator(`.ep[data-guid="${last}"]`);
await row.locator('[data-a="flag"]').click();
await expect.poll(async () => (await guids())[0]).toBe(last);
// It is the server's order, so it holds on a reload.
await page.reload();
await expect.poll(async () => (await guids())[0]).toBe(last);
await page.locator(`.ep[data-guid="${last}"] [data-a="flag"]`).click();
await expect.poll(guids).toEqual(before);
});

View File

@@ -314,7 +314,9 @@ async function epAction(a: string, e, el, encId?: number){
const path=`/api/entries/${encodeURIComponent(e.feed_id)}/${encodeURIComponent(e.guid)}`;
try{
if(a==='play') play(e, encId!=null ? enc : undefined);
if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); }
// A pinned item sits at the top of its list (the server sorts it there), so the list is
// asked for again rather than the row redrawn where it stands.
if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); loadEntries(); }
if(a==='read'){ await setRead(e,!e.read); redraw(); }
if(a==='get'){
if(!enc) return;