Merge seaorm: the database through SeaORM, on SQLite or Postgres (#18)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
10
CHANGELOG.md
10
CHANGELOG.md
@@ -7,8 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- ipx can keep its data in Postgres: set `IPX_DATABASE_URL` to a `postgres://` URL. Without it,
|
||||||
|
it is the SQLite `state.db` as before. `ipx copy-db <state.db>` moves an existing database
|
||||||
|
across, everything in one go.
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
|
- The database is reached through SeaORM, on the way to Postgres (issue #18); it is still the
|
||||||
|
same SQLite file, and nothing you see changes. On Postgres, sorting by title or feed follows
|
||||||
|
the language's order (an accented letter beside the plain one) rather than raw bytes. A database from before 0.7 has to be opened by a
|
||||||
|
0.7 release first, which brings its tables up to date.
|
||||||
- A pinned item sits at the top of its list, above everything else in whatever order you sort
|
- A pinned item sits at the top of its list, above everything else in whatever order you sort
|
||||||
by, and moves there the moment you pin it. Sorting by the pin column itself still goes both
|
by, and moves there the moment you pin it. Sorting by the pin column itself still goes both
|
||||||
ways, and Currently Listening keeps its own order.
|
ways, and Currently Listening keeps its own order.
|
||||||
|
|||||||
19
CLAUDE.md
19
CLAUDE.md
@@ -14,7 +14,8 @@ Arcane project `content`: `/mnt/fast/arcane/projects/content/compose.yaml`. That
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Image | `192.168.1.130:5000/ipodderx:latest` | |
|
| Image | `192.168.1.130:5000/ipodderx:latest` | |
|
||||||
| Config | `/mnt/fast/appdata/ipodderx/config.toml` | `/config/config.toml` |
|
| Config | `/mnt/fast/appdata/ipodderx/config.toml` | `/config/config.toml` |
|
||||||
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` |
|
| Database | Postgres 18, database `ipodderx`, login `ipodderx`, on the `postgres` container of the Arcane project `databases` (`192.168.1.130:5433`). The URL is in `/mnt/fast/appdata/ipodderx/database.env` (root-only), passed to the container as `IPX_DATABASE_URL` | |
|
||||||
|
| Old database | `/mnt/user/ipodderx/state.db`, SQLite, used until the move to Postgres on 2026-09-18 and kept for rollback | `/data/state.db` |
|
||||||
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
|
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
|
||||||
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
|
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
|
||||||
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
|
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
|
||||||
@@ -125,7 +126,7 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
|||||||
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
|
||||||
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
|
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
|
||||||
and `position` columns from before accounts; two bugs came from queries still reading them
|
and `position` columns from before accounts; two bugs came from queries still reading them
|
||||||
(retention, and the entry pruner), and `migrate()` now drops them.
|
(retention, and the entry pruner), and they were dropped in 0.5.
|
||||||
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once;
|
||||||
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
|
||||||
derived and never written to config.
|
derived and never written to config.
|
||||||
@@ -140,10 +141,16 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
|
|||||||
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
|
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
|
||||||
different task. `ipx status` checks the control socket and the database; to see the worker
|
different task. `ipx status` checks the control socket and the database; to see the worker
|
||||||
getting through its jobs, watch for `scan complete` in the log.
|
getting through its jobs, watch for `scan complete` in the log.
|
||||||
* **Every `ipx` command runs `migrate()` when it opens the database**, the healthcheck's
|
* **The database goes through SeaORM, and the entities in `src/entity.rs` are the schema.**
|
||||||
`ipx status` included. A migration that rewrites a big table (`DROP COLUMN`) takes seconds on
|
`Db::open` creates any missing table or index from them (`create_missing`), on every `ipx`
|
||||||
production, and a command run meanwhile fails with `migrating schema`. It changes nothing; wait
|
command, the healthcheck's `ipx status` included, so it must never write when nothing is
|
||||||
for `daemon started` in the log. Copy `state.db` aside before deploying one.
|
missing: SeaORM's experimental schema sync dropped and remade an index on every open, the
|
||||||
|
write lock that took made `ipx status` time out behind a busy daemon, and it was removed for
|
||||||
|
it. A new column on an existing table needs its own `ALTER`; nothing adds one for you.
|
||||||
|
* **SQL written by hand in `db.rs` has to run on SQLite and Postgres both** (issue #18): `$1`
|
||||||
|
parameters, bound only if used; `ON CONFLICT`, not `INSERT OR IGNORE`; yes/no columns tested
|
||||||
|
as themselves (`NOT coalesce(s.read, false)`) and written as `true`/`false`, never compared to
|
||||||
|
1; no `rowid`, `GLOB` or `UPDATE OR IGNORE`. `Args` in `db.rs` builds the parameters.
|
||||||
|
|
||||||
## House style
|
## House style
|
||||||
|
|
||||||
|
|||||||
873
Cargo.lock
generated
873
Cargo.lock
generated
File diff suppressed because it is too large
Load Diff
@@ -18,7 +18,7 @@ percent-encoding = "2.3.2"
|
|||||||
quick-xml = { version = "0.42.0", features = ["escape-html"] }
|
quick-xml = { version = "0.42.0", features = ["escape-html"] }
|
||||||
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] }
|
reqwest = { version = "0.13.5", default-features = false, features = ["rustls", "http2", "gzip", "stream", "json", "charset", "system-proxy"] }
|
||||||
rss = "2.1.1"
|
rss = "2.1.1"
|
||||||
rusqlite = { version = "0.40.2", features = ["bundled"] }
|
sea-orm = { version = "2.0.3", default-features = false, features = ["sqlx-sqlite", "sqlx-postgres", "runtime-tokio-rustls", "macros", "with-json", "sqlite-use-returning-for-3_35"] }
|
||||||
serde = { version = "1.0.229", features = ["derive"] }
|
serde = { version = "1.0.229", features = ["derive"] }
|
||||||
serde_json = "1.0.151"
|
serde_json = "1.0.151"
|
||||||
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "sync", "time", "signal"] }
|
||||||
|
|||||||
@@ -10,7 +10,8 @@ it to a running daemon.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
||||||
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
||||||
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
| `src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
||||||
|
| `src/entity.rs` | The tables, as SeaORM entities: the schema | — |
|
||||||
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
|
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
|
||||||
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
||||||
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
||||||
@@ -65,14 +66,14 @@ entry_state user_id, feed_id, guid, read, flagged, position
|
|||||||
```
|
```
|
||||||
|
|
||||||
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
|
||||||
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
|
before accounts; two bugs came from queries still reading them, and they were dropped in 0.5.
|
||||||
older database.
|
|
||||||
|
|
||||||
Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
|
Schema changes: the tables are the entities in `src/entity.rs`, and `Db::open` creates whatever
|
||||||
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
|
table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It
|
||||||
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
|
never alters a table that exists, so a new column on one needs its own `ALTER` in
|
||||||
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
|
`create_missing`, or `sea-orm-migration` once there are several. `Db::memory()` builds its
|
||||||
path as `Db::open`, so a migration cannot pass the tests while missing in production.
|
database the same way, so the tests run on the schema production gets. A database from before
|
||||||
|
0.7 takes its last columns from the old `migrate()`, so it upgrades through a 0.7 release first.
|
||||||
|
|
||||||
## Control socket
|
## Control socket
|
||||||
|
|
||||||
|
|||||||
@@ -11,8 +11,10 @@ effect without a restart. Default location `$XDG_CONFIG_HOME/ipx/config.toml`
|
|||||||
| Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` |
|
| Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` |
|
||||||
| Downloads | `[general] download_dir` | — |
|
| Downloads | `[general] download_dir` | — |
|
||||||
|
|
||||||
`~` is expanded in paths. The database is SQLite in WAL mode; back it up by copying `state.db`
|
`~` is expanded in paths. The database is SQLite in WAL mode unless `IPX_DATABASE_URL` names a
|
||||||
while the daemon is stopped, or with `sqlite3 state.db .backup`.
|
Postgres database instead. Back SQLite up by copying `state.db` while the daemon is stopped, or
|
||||||
|
with `sqlite3 state.db .backup`; back Postgres up with `pg_dump`. `ipx copy-db <state.db>` copies a
|
||||||
|
SQLite database into the empty Postgres one `IPX_DATABASE_URL` names.
|
||||||
|
|
||||||
## `[general]`
|
## `[general]`
|
||||||
|
|
||||||
@@ -116,6 +118,8 @@ they are re-derived on every scan. Editing one in the UI promotes it to a real c
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `IPX_CONFIG` | Config file path |
|
| `IPX_CONFIG` | Config file path |
|
||||||
| `IPX_DATA_DIR` | Directory holding `state.db` |
|
| `IPX_DATA_DIR` | Directory holding `state.db` |
|
||||||
|
| `IPX_DATABASE_URL` | A `postgres://user:password@host:port/database` URL: use that database instead of `state.db` |
|
||||||
|
| `IPX_TEST_DATABASE_URL` | For `cargo test`: run the database tests on this Postgres database too, each in a schema of its own |
|
||||||
| `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) |
|
| `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) |
|
||||||
| `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view |
|
| `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view |
|
||||||
| `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches |
|
| `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches |
|
||||||
|
|||||||
244
src/entity.rs
Normal file
244
src/entity.rs
Normal file
@@ -0,0 +1,244 @@
|
|||||||
|
//! The database's tables as SeaORM entities: the one description of the schema, from which
|
||||||
|
//! `Db::open` creates what a database is missing, on SQLite or Postgres alike (see
|
||||||
|
//! `db::create_missing`). Times are Unix seconds.
|
||||||
|
|
||||||
|
pub mod feeds {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "feeds")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub id: String,
|
||||||
|
#[sea_orm(column_type = "Text")]
|
||||||
|
pub url: String,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub title: Option<String>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub image: Option<String>,
|
||||||
|
/// The channel's first <itunes:category>, for the Directory.
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub category: Option<String>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub etag: Option<String>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub last_modified: Option<String>,
|
||||||
|
pub last_checked: Option<i64>,
|
||||||
|
pub ttl_mins: Option<i64>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub last_error: Option<String>,
|
||||||
|
/// When the current run of failures began; NULL while the feed is healthy. Kept through
|
||||||
|
/// repeated failures so the UI can tell a blip (macmanx: failed once, fine an hour
|
||||||
|
/// later) from a feed that has been down for a day.
|
||||||
|
pub error_since: Option<i64>,
|
||||||
|
/// Came from a subscribed OPML that no longer lists it, but has downloads, so kept.
|
||||||
|
#[sea_orm(default_value = false)]
|
||||||
|
pub orphaned: bool,
|
||||||
|
/// The OPML subscription this feed came from.
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub group_id: Option<String>,
|
||||||
|
/// Derived from an OPML and not written to config.toml. Writing 80-odd generated entries
|
||||||
|
/// into a hand-edited file made it unreadable; the OPML is the source of truth, so they
|
||||||
|
/// are re-derived instead. Customising one promotes it to config.
|
||||||
|
#[sea_orm(default_value = false)]
|
||||||
|
pub managed: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||||
|
pub enum Relation {}
|
||||||
|
|
||||||
|
impl ActiveModelBehavior for ActiveModel {}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub mod entries {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "entries")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub feed_id: String,
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub guid: String,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub title: Option<String>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub link: Option<String>,
|
||||||
|
pub published: Option<i64>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub description: Option<String>,
|
||||||
|
pub first_seen: i64,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub image: Option<String>,
|
||||||
|
pub duration: Option<i64>,
|
||||||
|
pub episode: Option<i64>,
|
||||||
|
pub season: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||||
|
pub enum Relation {}
|
||||||
|
|
||||||
|
impl ActiveModelBehavior for ActiveModel {}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub mod enclosures {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "enclosures")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key)]
|
||||||
|
pub id: i64,
|
||||||
|
#[sea_orm(column_type = "Text")]
|
||||||
|
pub feed_id: String,
|
||||||
|
#[sea_orm(column_type = "Text")]
|
||||||
|
pub guid: String,
|
||||||
|
/// The dedupe key, and the reason one file serves every subscriber. A reaped file keeps
|
||||||
|
/// its row with path NULL and state 'reaped', so a purged episode is never fetched again.
|
||||||
|
#[sea_orm(unique, column_type = "Text")]
|
||||||
|
pub url: String,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub mime: Option<String>,
|
||||||
|
pub length: Option<i64>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub path: Option<String>,
|
||||||
|
#[sea_orm(column_type = "Text")]
|
||||||
|
pub state: String,
|
||||||
|
#[sea_orm(default_value = 0)]
|
||||||
|
pub bytes_done: i64,
|
||||||
|
pub downloaded_at: Option<i64>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub last_error: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||||
|
pub enum Relation {}
|
||||||
|
|
||||||
|
impl ActiveModelBehavior for ActiveModel {}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub mod users {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "users")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key)]
|
||||||
|
pub id: i64,
|
||||||
|
/// Unique without regard to case: `db::create_missing` adds the index on lower(name), which
|
||||||
|
/// works the same on both databases where SQLite's COLLATE NOCASE does not.
|
||||||
|
#[sea_orm(column_type = "Text")]
|
||||||
|
pub name: String,
|
||||||
|
/// NULL for someone who only ever arrives through the proxy: there is no password to
|
||||||
|
/// check, and leaving it empty is not the same as leaving it unset.
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub pass_hash: Option<String>,
|
||||||
|
#[sea_orm(default_value = false)]
|
||||||
|
pub is_admin: bool,
|
||||||
|
/// For whoever maintains the server. NULL where it is not known.
|
||||||
|
pub created: Option<i64>,
|
||||||
|
pub last_login: Option<i64>,
|
||||||
|
/// The theme chosen in Settings, and light, dark or auto. NULL until one is chosen.
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub theme: Option<String>,
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub theme_mode: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||||
|
pub enum Relation {}
|
||||||
|
|
||||||
|
impl ActiveModelBehavior for ActiveModel {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A table of one person's rows, gone when they are.
|
||||||
|
macro_rules! owned_by_user {
|
||||||
|
() => {
|
||||||
|
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||||
|
pub enum Relation {
|
||||||
|
#[sea_orm(
|
||||||
|
belongs_to = "super::users::Entity",
|
||||||
|
from = "Column::UserId",
|
||||||
|
to = "super::users::Column::Id",
|
||||||
|
on_delete = "Cascade"
|
||||||
|
)]
|
||||||
|
User,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Related<super::users::Entity> for Entity {
|
||||||
|
fn to() -> RelationDef {
|
||||||
|
Relation::User.def()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ActiveModelBehavior for ActiveModel {}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What one person wants from a feed. The feed, its items and its files are shared; this is the
|
||||||
|
/// part that is not. NULL in a column means: follow the feed's own setting.
|
||||||
|
pub mod subscriptions {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "subscriptions")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key, auto_increment = false)]
|
||||||
|
pub user_id: i64,
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub feed_id: String,
|
||||||
|
/// JSON array of strings; NULL follows the feed.
|
||||||
|
#[sea_orm(column_type = "Text", nullable)]
|
||||||
|
pub keywords: Option<String>,
|
||||||
|
pub auto_download: Option<bool>,
|
||||||
|
pub allow_explicit: Option<bool>,
|
||||||
|
pub max_new_per_check: Option<i64>,
|
||||||
|
/// Pinned to the top of this person's feed list, a feed inside a folder included.
|
||||||
|
#[sea_orm(default_value = false)]
|
||||||
|
pub pinned: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
owned_by_user!();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read, kept and how far in. One row per person per item, created on first touch; an item
|
||||||
|
/// nobody has touched has no row at all, which is what unread means.
|
||||||
|
pub mod entry_state {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "entry_state")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key, auto_increment = false)]
|
||||||
|
pub user_id: i64,
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub feed_id: String,
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub guid: String,
|
||||||
|
#[sea_orm(default_value = false)]
|
||||||
|
pub read: bool,
|
||||||
|
#[sea_orm(default_value = false)]
|
||||||
|
pub flagged: bool,
|
||||||
|
#[sea_orm(default_value = 0)]
|
||||||
|
pub position: i64,
|
||||||
|
/// The length this person's player measured, beside the position it is measured against.
|
||||||
|
pub duration: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
owned_by_user!();
|
||||||
|
}
|
||||||
|
|
||||||
|
pub mod sessions {
|
||||||
|
use sea_orm::entity::prelude::*;
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||||
|
#[sea_orm(table_name = "sessions")]
|
||||||
|
pub struct Model {
|
||||||
|
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
|
||||||
|
pub token: String,
|
||||||
|
pub user_id: i64,
|
||||||
|
pub seen: i64,
|
||||||
|
}
|
||||||
|
|
||||||
|
owned_by_user!();
|
||||||
|
}
|
||||||
@@ -176,7 +176,9 @@ pub async fn daemon_is_live(path: &Path) -> bool {
|
|||||||
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
|
/// healthcheck left waiting behind a scan or a long download timed out and called a busy daemon
|
||||||
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
|
/// dead. The answer goes to the client that asked and no one else: broadcast, it ended any
|
||||||
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
|
/// `ipx fetch` that was watching a scan, since `status` is a terminal event.
|
||||||
pub type StatusFn = std::sync::Arc<dyn Fn() -> Event + Send + Sync>;
|
/// A future, since reading the counts is a database query.
|
||||||
|
pub type StatusFn =
|
||||||
|
std::sync::Arc<dyn Fn() -> std::pin::Pin<Box<dyn std::future::Future<Output = Event> + Send>> + Send + Sync>;
|
||||||
|
|
||||||
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
|
/// Accepts connections, feeding commands to `cmds` and events from `events` back out.
|
||||||
pub async fn serve(
|
pub async fn serve(
|
||||||
@@ -249,7 +251,7 @@ async fn handle(
|
|||||||
// Answered here, not queued behind whatever the worker is on: see StatusFn.
|
// Answered here, not queued behind whatever the worker is on: see StatusFn.
|
||||||
Ok(Command::Status) => {
|
Ok(Command::Status) => {
|
||||||
tracing::info!(target: "ipx::io", "-> {line}");
|
tracing::info!(target: "ipx::io", "-> {line}");
|
||||||
let ev = status();
|
let ev = status().await;
|
||||||
if let Ok(json) = serde_json::to_string(&ev) {
|
if let Ok(json) = serde_json::to_string(&ev) {
|
||||||
tracing::info!(target: "ipx::io", "<- {json}");
|
tracing::info!(target: "ipx::io", "<- {json}");
|
||||||
}
|
}
|
||||||
@@ -366,7 +368,8 @@ mod tests {
|
|||||||
// Another client, watching a scan: it must not be handed someone else's answer, which
|
// Another client, watching a scan: it must not be handed someone else's answer, which
|
||||||
// would end its session.
|
// would end its session.
|
||||||
let mut watcher = events.subscribe();
|
let mut watcher = events.subscribe();
|
||||||
let status: StatusFn = std::sync::Arc::new(|| Event::Status { feeds: 1, pending: 2, downloaded: 3 });
|
let status: StatusFn =
|
||||||
|
std::sync::Arc::new(|| Box::pin(async { Event::Status { feeds: 1, pending: 2, downloaded: 3 } }));
|
||||||
let (client, server) = UnixStream::pair().unwrap();
|
let (client, server) = UnixStream::pair().unwrap();
|
||||||
tokio::spawn(handle(server, events.subscribe(), cmds, status));
|
tokio::spawn(handle(server, events.subscribe(), cmds, status));
|
||||||
|
|
||||||
|
|||||||
319
src/main.rs
319
src/main.rs
@@ -1,6 +1,7 @@
|
|||||||
mod auth;
|
mod auth;
|
||||||
mod config;
|
mod config;
|
||||||
mod db;
|
mod db;
|
||||||
|
mod entity;
|
||||||
mod download;
|
mod download;
|
||||||
mod feed;
|
mod feed;
|
||||||
mod ipc;
|
mod ipc;
|
||||||
@@ -36,6 +37,12 @@ struct Cli {
|
|||||||
enum Command {
|
enum Command {
|
||||||
/// Show configured feeds and their state
|
/// Show configured feeds and their state
|
||||||
List,
|
List,
|
||||||
|
/// Copy everything from a SQLite state.db into the database IPX_DATABASE_URL names, which
|
||||||
|
/// must be empty: the one-off move to Postgres
|
||||||
|
CopyDb {
|
||||||
|
/// The SQLite file to copy from
|
||||||
|
from: PathBuf,
|
||||||
|
},
|
||||||
/// Scan feeds for new entries
|
/// Scan feeds for new entries
|
||||||
Fetch {
|
Fetch {
|
||||||
/// Only this feed id
|
/// Only this feed id
|
||||||
@@ -179,7 +186,7 @@ async fn main() -> Result<()> {
|
|||||||
|
|
||||||
let config_path = cli.config.clone().unwrap_or_else(config::config_path);
|
let config_path = cli.config.clone().unwrap_or_else(config::config_path);
|
||||||
let cfg = config::Config::load(&config_path)?;
|
let cfg = config::Config::load(&config_path)?;
|
||||||
let db = db::Db::open(&config::data_dir().join("state.db"))?;
|
let db = db::Db::open(&db::location()).await?;
|
||||||
|
|
||||||
// A daemon owns the state; don't have two processes downloading the same thing.
|
// A daemon owns the state; don't have two processes downloading the same thing.
|
||||||
let wire_cmd = match &cli.command {
|
let wire_cmd = match &cli.command {
|
||||||
@@ -194,7 +201,8 @@ async fn main() -> Result<()> {
|
|||||||
| Command::Add { .. }
|
| Command::Add { .. }
|
||||||
| Command::Rm { .. }
|
| Command::Rm { .. }
|
||||||
| Command::Import { .. }
|
| Command::Import { .. }
|
||||||
| Command::Export { .. } => None,
|
| Command::Export { .. }
|
||||||
|
| Command::CopyDb { .. } => None,
|
||||||
};
|
};
|
||||||
if let Some(cmd) = &wire_cmd
|
if let Some(cmd) = &wire_cmd
|
||||||
&& !cli.local
|
&& !cli.local
|
||||||
@@ -219,22 +227,23 @@ async fn main() -> Result<()> {
|
|||||||
});
|
});
|
||||||
|
|
||||||
match cli.command {
|
match cli.command {
|
||||||
Command::List => list(&ctx, &config_path),
|
Command::List => list(&ctx, &config_path).await,
|
||||||
Command::Daemon { web } => daemon(ctx, config_path, web, events).await,
|
Command::Daemon { web } => daemon(ctx, config_path, web, events).await,
|
||||||
Command::Add { url, folder, keywords } => {
|
Command::Add { url, folder, keywords } => {
|
||||||
add(&ctx, &config_path, &url, folder, keywords).await
|
add(&ctx, &config_path, &url, folder, keywords).await
|
||||||
}
|
}
|
||||||
Command::Rm { feed } => rm(&ctx, &config_path, &feed),
|
Command::Rm { feed } => rm(&ctx, &config_path, &feed).await,
|
||||||
Command::User { cmd } => user_cmd(&ctx, cmd),
|
Command::User { cmd } => user_cmd(&ctx, cmd).await,
|
||||||
Command::Import { file } => import(&ctx, &config_path, &file).await,
|
Command::Import { file } => import(&ctx, &config_path, &file).await,
|
||||||
Command::Export { file } => export(&ctx, &file),
|
Command::Export { file } => export(&ctx, &file).await,
|
||||||
|
Command::CopyDb { from } => copy_db(&ctx, &from).await,
|
||||||
_ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await,
|
_ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Accounts. Passwords come in on stdin so they never reach a shell history or a `ps`
|
/// Accounts. Passwords come in on stdin so they never reach a shell history or a `ps`
|
||||||
/// listing.
|
/// listing.
|
||||||
fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
async fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
||||||
let read_password = || -> Result<String> {
|
let read_password = || -> Result<String> {
|
||||||
use std::io::Read;
|
use std::io::Read;
|
||||||
let mut buf = String::new();
|
let mut buf = String::new();
|
||||||
@@ -252,7 +261,7 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
if name.is_empty() {
|
if name.is_empty() {
|
||||||
anyhow::bail!("a name is required");
|
anyhow::bail!("a name is required");
|
||||||
}
|
}
|
||||||
if ctx.db.user_by_name(&name)?.is_some() {
|
if ctx.db.user_by_name(&name).await?.is_some() {
|
||||||
anyhow::bail!("{name} already exists");
|
anyhow::bail!("{name} already exists");
|
||||||
}
|
}
|
||||||
let hash = if no_password {
|
let hash = if no_password {
|
||||||
@@ -261,8 +270,8 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
Some(crate::auth::hash_password(&read_password()?)?)
|
Some(crate::auth::hash_password(&read_password()?)?)
|
||||||
};
|
};
|
||||||
// The first account runs the place; there is nobody else to grant it.
|
// The first account runs the place; there is nobody else to grant it.
|
||||||
let first = ctx.db.users()?.is_empty();
|
let first = ctx.db.users().await?.is_empty();
|
||||||
ctx.db.create_user(&name, hash.as_deref(), admin || first)?;
|
ctx.db.create_user(&name, hash.as_deref(), admin || first).await?;
|
||||||
println!(
|
println!(
|
||||||
"added {name}{}{}",
|
"added {name}{}{}",
|
||||||
if admin || first { " (admin)" } else { "" },
|
if admin || first { " (admin)" } else { "" },
|
||||||
@@ -271,7 +280,7 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
UserCmd::List => {
|
UserCmd::List => {
|
||||||
let users = ctx.db.users()?;
|
let users = ctx.db.users().await?;
|
||||||
if users.is_empty() {
|
if users.is_empty() {
|
||||||
println!("no accounts yet: ipx user add <name>");
|
println!("no accounts yet: ipx user add <name>");
|
||||||
}
|
}
|
||||||
@@ -294,9 +303,9 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
let name = name.trim().to_ascii_lowercase();
|
let name = name.trim().to_ascii_lowercase();
|
||||||
let user = ctx
|
let user = ctx
|
||||||
.db
|
.db
|
||||||
.user_by_name(&name)?
|
.user_by_name(&name).await?
|
||||||
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
||||||
ctx.db.set_password(user.id, &crate::auth::hash_password(&read_password()?)?)?;
|
ctx.db.set_password(user.id, &crate::auth::hash_password(&read_password()?)?).await?;
|
||||||
println!("password changed for {name}");
|
println!("password changed for {name}");
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -307,12 +316,12 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
|
.ok_or_else(|| anyhow::anyhow!("not a usable name: no commas, semicolons or line breaks"))?;
|
||||||
let user = ctx
|
let user = ctx
|
||||||
.db
|
.db
|
||||||
.user_by_name(&name)?
|
.user_by_name(&name).await?
|
||||||
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
||||||
if ctx.db.user_by_name(&new_name)?.is_some() {
|
if ctx.db.user_by_name(&new_name).await?.is_some() {
|
||||||
anyhow::bail!("{new_name} already exists");
|
anyhow::bail!("{new_name} already exists");
|
||||||
}
|
}
|
||||||
ctx.db.rename_user(user.id, &new_name)?;
|
ctx.db.rename_user(user.id, &new_name).await?;
|
||||||
println!("renamed {name} to {new_name}");
|
println!("renamed {name} to {new_name}");
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -320,9 +329,9 @@ fn user_cmd(ctx: &Arc<Ctx>, cmd: UserCmd) -> Result<()> {
|
|||||||
let name = name.trim().to_ascii_lowercase();
|
let name = name.trim().to_ascii_lowercase();
|
||||||
let user = ctx
|
let user = ctx
|
||||||
.db
|
.db
|
||||||
.user_by_name(&name)?
|
.user_by_name(&name).await?
|
||||||
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
.ok_or_else(|| anyhow::anyhow!("no such account: {name}"))?;
|
||||||
ctx.db.delete_user(user.id)?;
|
ctx.db.delete_user(user.id).await?;
|
||||||
println!("removed {name}");
|
println!("removed {name}");
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -333,13 +342,13 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
|
|||||||
match cmd {
|
match cmd {
|
||||||
Cmd::Fetch { feed, force } => {
|
Cmd::Fetch { feed, force } => {
|
||||||
// Make room before pulling more down, as the original did per download.
|
// Make room before pulling more down, as the original did per download.
|
||||||
reap(ctx, false, false)?;
|
reap(ctx, false, false).await?;
|
||||||
fetch(ctx, feed.as_deref(), force).await
|
fetch(ctx, feed.as_deref(), force).await
|
||||||
}
|
}
|
||||||
Cmd::Reap { dry_run } => reap(ctx, dry_run, true),
|
Cmd::Reap { dry_run } => reap(ctx, dry_run, true).await,
|
||||||
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
|
Cmd::Download { enclosure } => download_one(ctx, enclosure).await,
|
||||||
Cmd::Status => {
|
Cmd::Status => {
|
||||||
ctx.out.emit(status(ctx));
|
ctx.out.emit(status(ctx).await);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -347,10 +356,10 @@ async fn run(ctx: &Arc<Ctx>, cmd: Cmd) -> Result<()> {
|
|||||||
|
|
||||||
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
|
/// The counts `ipx status` prints. A running daemon's socket answers with this directly rather
|
||||||
/// than through the job queue.
|
/// than through the job queue.
|
||||||
fn status(ctx: &Ctx) -> Event {
|
async fn status(ctx: &Ctx) -> Event {
|
||||||
match ctx.db.counts() {
|
match ctx.db.counts().await {
|
||||||
Ok((pending, downloaded)) => {
|
Ok((pending, downloaded)) => {
|
||||||
let feeds = subscriptions(ctx).map(|s| s.len()).unwrap_or(0);
|
let feeds = subscriptions(ctx).await.map(|s| s.len()).unwrap_or(0);
|
||||||
Event::Status { feeds, pending, downloaded }
|
Event::Status { feeds, pending, downloaded }
|
||||||
}
|
}
|
||||||
Err(e) => Event::Error { msg: format!("{e:#}") },
|
Err(e) => Event::Error { msg: format!("{e:#}") },
|
||||||
@@ -369,30 +378,30 @@ async fn daemon(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// A database with nobody in it cannot be signed into.
|
// A database with nobody in it cannot be signed into.
|
||||||
if ctx.db.users()?.is_empty() {
|
if ctx.db.users().await?.is_empty() {
|
||||||
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true)?;
|
ctx.db.create_user("admin", Some(&crate::auth::hash_password(DEFAULT_PASSWORD)?), true).await?;
|
||||||
tracing::warn!(
|
tracing::warn!(
|
||||||
"no accounts yet: created 'admin' with the default password '{DEFAULT_PASSWORD}'. \
|
"no accounts yet: created 'admin' with the default password '{DEFAULT_PASSWORD}'. \
|
||||||
Change it with `echo -n <password> | ipx user passwd admin`"
|
Change it with `echo -n <password> | ipx user passwd admin`"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
if let Some(admin) = ctx.db.users()?.into_iter().find(|u| u.is_admin) {
|
if let Some(admin) = ctx.db.users().await?.into_iter().find(|u| u.is_admin) {
|
||||||
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
let catalogue: Vec<String> = ctx.cfg().feeds.keys().cloned().collect();
|
||||||
match ctx.db.adopt_catalogue(admin.id, &catalogue) {
|
match ctx.db.adopt_catalogue(admin.id, &catalogue).await {
|
||||||
Ok(0) => {}
|
Ok(0) => {}
|
||||||
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
|
Ok(n) => tracing::info!(user = %admin.name, feeds = n, "subscribed the first admin to the catalogue"),
|
||||||
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
|
Err(e) => tracing::error!(error = %e, "could not subscribe the first admin to the catalogue"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
match ctx.db.requeue_interrupted() {
|
match ctx.db.requeue_interrupted().await {
|
||||||
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
|
Ok(n) if n > 0 => tracing::info!(count = n, "requeued downloads interrupted by a restart"),
|
||||||
Ok(_) => {}
|
Ok(_) => {}
|
||||||
Err(e) => tracing::warn!(error = ?e, "could not requeue interrupted downloads"),
|
Err(e) => tracing::warn!(error = ?e, "could not requeue interrupted downloads"),
|
||||||
}
|
}
|
||||||
|
|
||||||
match retire_stranded(&ctx) {
|
match retire_stranded(&ctx).await {
|
||||||
Ok(0) => {}
|
Ok(0) => {}
|
||||||
Ok(n) => tracing::info!(feeds = n, "retired feeds whose OPML is no longer in config"),
|
Ok(n) => tracing::info!(feeds = n, "retired feeds whose OPML is no longer in config"),
|
||||||
Err(e) => tracing::warn!(error = ?e, "could not retire feeds whose OPML is no longer in config"),
|
Err(e) => tracing::warn!(error = ?e, "could not retire feeds whose OPML is no longer in config"),
|
||||||
@@ -400,7 +409,7 @@ async fn daemon(
|
|||||||
|
|
||||||
// Before the parser knew WordPress's numbered player URLs, a file it listed twice was
|
// Before the parser knew WordPress's numbered player URLs, a file it listed twice was
|
||||||
// downloaded twice. The repeats fold into the first, and their spare copies are deleted.
|
// downloaded twice. The repeats fold into the first, and their spare copies are deleted.
|
||||||
match ctx.db.merge_repeated_enclosures(feed::same_file_key) {
|
match ctx.db.merge_repeated_enclosures(feed::same_file_key).await {
|
||||||
Ok((0, _)) => {}
|
Ok((0, _)) => {}
|
||||||
Ok((n, spare)) => {
|
Ok((n, spare)) => {
|
||||||
for path in &spare {
|
for path in &spare {
|
||||||
@@ -419,7 +428,10 @@ async fn daemon(
|
|||||||
// status is answered by the socket itself; everything else waits its turn in the queue.
|
// status is answered by the socket itself; everything else waits its turn in the queue.
|
||||||
let answer: ipc::StatusFn = {
|
let answer: ipc::StatusFn = {
|
||||||
let ctx = ctx.clone();
|
let ctx = ctx.clone();
|
||||||
Arc::new(move || status(&ctx))
|
Arc::new(move || {
|
||||||
|
let ctx = ctx.clone();
|
||||||
|
Box::pin(async move { status(&ctx).await })
|
||||||
|
})
|
||||||
};
|
};
|
||||||
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
|
let server = tokio::spawn(ipc::serve(socket.clone(), events.clone(), tx_cmd, answer));
|
||||||
|
|
||||||
@@ -427,7 +439,7 @@ async fn daemon(
|
|||||||
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
let mut ticker = tokio::time::interval(std::time::Duration::from_secs(60));
|
||||||
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
feeds = subscriptions(&ctx).map(|s| s.len()).unwrap_or(0),
|
feeds = subscriptions(&ctx).await.map(|s| s.len()).unwrap_or(0),
|
||||||
"daemon started"
|
"daemon started"
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -571,7 +583,7 @@ async fn add(
|
|||||||
let mut cfg = (*ctx.cfg()).clone();
|
let mut cfg = (*ctx.cfg()).clone();
|
||||||
let url = &feed::expand_input(url);
|
let url = &feed::expand_input(url);
|
||||||
// Includes feeds derived from an OPML, or the same show could be added twice.
|
// Includes feeds derived from an OPML, or the same show could be added twice.
|
||||||
if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
|
if let Some(existing) = subscriptions(ctx).await?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) {
|
||||||
anyhow::bail!("already subscribed as {:?}", existing.id);
|
anyhow::bail!("already subscribed as {:?}", existing.id);
|
||||||
}
|
}
|
||||||
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
|
||||||
@@ -623,13 +635,13 @@ pub async fn add_one(
|
|||||||
|
|
||||||
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
// Slugs must be unique across derived feeds too, or a new feed can collide with one
|
||||||
// an OPML already introduced.
|
// an OPML already introduced.
|
||||||
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)?
|
let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx).await?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|s| (s.id, s.cfg))
|
.map(|s| (s.id, s.cfg))
|
||||||
.collect();
|
.collect();
|
||||||
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
|
// A removed feed keeps its rows, so its id is only free again for the same feed: re-adding
|
||||||
// it gets its history back, and a different feed does not inherit someone else's.
|
// it gets its history back, and a different feed does not inherit someone else's.
|
||||||
for (id, other) in ctx.db.feed_urls()? {
|
for (id, other) in ctx.db.feed_urls().await? {
|
||||||
if !feed::same_feed(&other, url) {
|
if !feed::same_feed(&other, url) {
|
||||||
taken.entry(id).or_insert_with(|| probe.clone());
|
taken.entry(id).or_insert_with(|| probe.clone());
|
||||||
}
|
}
|
||||||
@@ -647,19 +659,19 @@ fn url_stem(url: &str) -> String {
|
|||||||
.unwrap_or_else(|| url.to_owned())
|
.unwrap_or_else(|| url.to_owned())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
|
async fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> {
|
||||||
let mut cfg = (*ctx.cfg()).clone();
|
let mut cfg = (*ctx.cfg()).clone();
|
||||||
if cfg.feeds.remove(feed).is_none() {
|
if cfg.feeds.remove(feed).is_none() {
|
||||||
// Derived from an OPML: drop it here, though the subscription will list it again
|
// Derived from an OPML: drop it here, though the subscription will list it again
|
||||||
// on the next read unless the OPML itself goes.
|
// on the next read unless the OPML itself goes.
|
||||||
ctx.db.drop_managed(feed)?;
|
ctx.db.drop_managed(feed).await?;
|
||||||
println!("removed {feed}; it came from an OPML subscription and may return on the next read");
|
println!("removed {feed}; it came from an OPML subscription and may return on the next read");
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
cfg.save(config_path)?;
|
cfg.save(config_path)?;
|
||||||
// State and files stay: re-adding the feed should not re-download its back catalogue.
|
// State and files stay: re-adding the feed should not re-download its back catalogue.
|
||||||
println!("removed {feed}; downloads and history kept");
|
println!("removed {feed}; downloads and history kept");
|
||||||
retire_group(ctx, feed)?;
|
retire_group(ctx, feed).await?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -669,13 +681,13 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
|
|||||||
// The CLI speaks for the operator, as the shared web token does.
|
// The CLI speaks for the operator, as the shared web token does.
|
||||||
let admin = ctx
|
let admin = ctx
|
||||||
.db
|
.db
|
||||||
.users()?
|
.users().await?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.find(|u| u.is_admin)
|
.find(|u| u.is_admin)
|
||||||
.ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?;
|
.ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?;
|
||||||
let doc = opml::OPML::from_str(&text)
|
let doc = opml::OPML::from_str(&text)
|
||||||
.map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?;
|
.map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?;
|
||||||
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id)?;
|
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id).await?;
|
||||||
println!("subscribed {} to {added} feed(s); {had} already there", admin.name);
|
println!("subscribed {} to {added} feed(s); {had} already there", admin.name);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -690,7 +702,7 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
|
|||||||
///
|
///
|
||||||
/// The caller parses the document, so each refuses a file that is not OPML in its own terms,
|
/// The caller parses the document, so each refuses a file that is not OPML in its own terms,
|
||||||
/// before anything is touched: a 400 from the web, a message from the CLI.
|
/// before anything is touched: a 400 from the web, a message from the CLI.
|
||||||
pub fn subscribe_opml(
|
pub async fn subscribe_opml(
|
||||||
ctx: &Ctx,
|
ctx: &Ctx,
|
||||||
config_path: &std::path::Path,
|
config_path: &std::path::Path,
|
||||||
doc: &opml::OPML,
|
doc: &opml::OPML,
|
||||||
@@ -699,7 +711,7 @@ pub fn subscribe_opml(
|
|||||||
let mut found = vec![];
|
let mut found = vec![];
|
||||||
collect_outlines(&doc.body.outlines, &mut found);
|
collect_outlines(&doc.body.outlines, &mut found);
|
||||||
|
|
||||||
let known = subscriptions(ctx)?;
|
let known = subscriptions(ctx).await?;
|
||||||
let mut cfg = (*ctx.cfg()).clone();
|
let mut cfg = (*ctx.cfg()).clone();
|
||||||
let mut ids = vec![];
|
let mut ids = vec![];
|
||||||
let mut grew = false;
|
let mut grew = false;
|
||||||
@@ -745,10 +757,10 @@ pub fn subscribe_opml(
|
|||||||
|
|
||||||
let (mut added, mut had) = (0, 0);
|
let (mut added, mut had) = (0, 0);
|
||||||
for id in ids {
|
for id in ids {
|
||||||
if ctx.db.subscription(user_id, &id)?.is_some() {
|
if ctx.db.subscription(user_id, &id).await?.is_some() {
|
||||||
had += 1;
|
had += 1;
|
||||||
} else {
|
} else {
|
||||||
ctx.db.subscribe(user_id, &id)?;
|
ctx.db.subscribe(user_id, &id).await?;
|
||||||
added += 1;
|
added += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -766,7 +778,16 @@ pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, Strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
|
async fn copy_db(ctx: &Ctx, from: &std::path::Path) -> Result<()> {
|
||||||
|
anyhow::ensure!(from.exists(), "{} does not exist", from.display());
|
||||||
|
let source = db::Db::open(&from.display().to_string()).await?;
|
||||||
|
for (table, n) in ctx.db.copy_from(&source).await? {
|
||||||
|
println!("{table:14} {n}");
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
|
||||||
let mut doc = opml::OPML::default();
|
let mut doc = opml::OPML::default();
|
||||||
doc.head = Some(opml::Head {
|
doc.head = Some(opml::Head {
|
||||||
title: Some("ipx subscriptions".into()),
|
title: Some("ipx subscriptions".into()),
|
||||||
@@ -775,7 +796,7 @@ fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
|
|||||||
for (id, feed) in &ctx.cfg().feeds {
|
for (id, feed) in &ctx.cfg().feeds {
|
||||||
let title = ctx
|
let title = ctx
|
||||||
.db
|
.db
|
||||||
.feed_summary(id)
|
.feed_summary(id).await
|
||||||
.ok()
|
.ok()
|
||||||
.and_then(|s| s.title)
|
.and_then(|s| s.title)
|
||||||
.unwrap_or_else(|| id.clone());
|
.unwrap_or_else(|| id.clone());
|
||||||
@@ -787,14 +808,14 @@ fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
|
async fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
if cfg.feeds.is_empty() {
|
if cfg.feeds.is_empty() {
|
||||||
println!("No feeds configured in {}", config_path.display());
|
println!("No feeds configured in {}", config_path.display());
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
for (id, feed) in &cfg.feeds {
|
for (id, feed) in &cfg.feeds {
|
||||||
let s = ctx.db.feed_summary(id)?;
|
let s = ctx.db.feed_summary(id).await?;
|
||||||
println!("{id} {}", s.title.as_deref().unwrap_or("-"));
|
println!("{id} {}", s.title.as_deref().unwrap_or("-"));
|
||||||
println!(" url {}", feed.url);
|
println!(" url {}", feed.url);
|
||||||
println!(" last checked {}", ago(s.last_checked));
|
println!(" last checked {}", ago(s.last_checked));
|
||||||
@@ -809,8 +830,8 @@ fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> {
|
|||||||
/// `standalone` false means this is the sweep that runs before a scan: it reports what it
|
/// `standalone` false means this is the sweep that runs before a scan: it reports what it
|
||||||
/// deleted, but must not emit the terminal ReapDone, or a client waiting on its `fetch`
|
/// deleted, but must not emit the terminal ReapDone, or a client waiting on its `fetch`
|
||||||
/// would stop reading before the scan had even started.
|
/// would stop reading before the scan had even started.
|
||||||
fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
|
async fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
|
||||||
let r = retention::run(&ctx.cfg(), &ctx.db, dry_run)?;
|
let r = retention::run(&ctx.cfg(), &ctx.db, dry_run).await?;
|
||||||
for c in r.aged_out.iter().chain(r.over_quota.iter()) {
|
for c in r.aged_out.iter().chain(r.over_quota.iter()) {
|
||||||
ctx.out.emit(Event::Reaped {
|
ctx.out.emit(Event::Reaped {
|
||||||
path: c.path.clone(),
|
path: c.path.clone(),
|
||||||
@@ -828,7 +849,7 @@ fn reap(ctx: &Ctx, dry_run: bool, standalone: bool) -> Result<()> {
|
|||||||
|
|
||||||
async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let subs = subscriptions(ctx)?;
|
let subs = subscriptions(ctx).await?;
|
||||||
if let Some(id) = only
|
if let Some(id) = only
|
||||||
&& !subs.iter().any(|s| s.id == id)
|
&& !subs.iter().any(|s| s.id == id)
|
||||||
{
|
{
|
||||||
@@ -839,7 +860,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
|||||||
let mut fresh: Vec<String> = vec![];
|
let mut fresh: Vec<String> = vec![];
|
||||||
for sub in subs.iter().filter(|s| only.is_none_or(|o| o == s.id)) {
|
for sub in subs.iter().filter(|s| only.is_none_or(|o| o == s.id)) {
|
||||||
let (id, feed_cfg) = (&sub.id, &sub.cfg);
|
let (id, feed_cfg) = (&sub.id, &sub.cfg);
|
||||||
let state = ctx.db.http_state(id)?;
|
let state = ctx.db.http_state(id).await?;
|
||||||
|
|
||||||
if !force && let Some(last) = state.last_checked {
|
if !force && let Some(last) = state.last_checked {
|
||||||
let due = last + due_after(&cfg, feed_cfg, state.ttl_mins) as i64;
|
let due = last + due_after(&cfg, feed_cfg, state.ttl_mins) as i64;
|
||||||
@@ -886,20 +907,20 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
|||||||
// One bad feed must not end the scan.
|
// One bad feed must not end the scan.
|
||||||
let msg = format!("{e:#}");
|
let msg = format!("{e:#}");
|
||||||
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
|
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
|
||||||
ctx.db.set_feed_error(id, &feed_cfg.url, &msg)?;
|
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Feeds a subscribed OPML just introduced: scan them now, in this run.
|
// Feeds a subscribed OPML just introduced: scan them now, in this run.
|
||||||
if !fresh.is_empty() {
|
if !fresh.is_empty() {
|
||||||
let subs = subscriptions(ctx)?;
|
let subs = subscriptions(ctx).await?;
|
||||||
for id in &fresh {
|
for id in &fresh {
|
||||||
let Some(feed_cfg) = subs.iter().find(|s| &s.id == id).map(|s| &s.cfg) else {
|
let Some(feed_cfg) = subs.iter().find(|s| &s.id == id).map(|s| &s.cfg) else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
scanned += 1;
|
scanned += 1;
|
||||||
ctx.out.emit(Event::FeedStart { feed: id.clone() });
|
ctx.out.emit(Event::FeedStart { feed: id.clone() });
|
||||||
let state = ctx.db.http_state(id)?;
|
let state = ctx.db.http_state(id).await?;
|
||||||
match scan_one(ctx, id, feed_cfg, &state).await {
|
match scan_one(ctx, id, feed_cfg, &state).await {
|
||||||
Ok(Outcome::Feed(s)) => ctx.out.emit(Event::FeedDone {
|
Ok(Outcome::Feed(s)) => ctx.out.emit(Event::FeedDone {
|
||||||
feed: id.clone(),
|
feed: id.clone(),
|
||||||
@@ -912,7 +933,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> {
|
|||||||
Err(e) => {
|
Err(e) => {
|
||||||
let msg = format!("{e:#}");
|
let msg = format!("{e:#}");
|
||||||
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
|
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
|
||||||
ctx.db.set_feed_error(id, &feed_cfg.url, &msg)?;
|
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -934,7 +955,7 @@ pub struct Sub {
|
|||||||
///
|
///
|
||||||
/// A derived feed borrows its parent's settings wholesale. That is why it needs no config
|
/// A derived feed borrows its parent's settings wholesale. That is why it needs no config
|
||||||
/// entry -- there is nothing to store but its URL and where it came from.
|
/// entry -- there is nothing to store but its URL and where it came from.
|
||||||
pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
pub async fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let mut out: Vec<Sub> = cfg
|
let mut out: Vec<Sub> = cfg
|
||||||
.feeds
|
.feeds
|
||||||
@@ -942,7 +963,7 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
|||||||
.map(|(id, f)| Sub { id: id.clone(), cfg: f.clone(), managed: false })
|
.map(|(id, f)| Sub { id: id.clone(), cfg: f.clone(), managed: false })
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
for m in ctx.db.managed_feeds()? {
|
for m in ctx.db.managed_feeds().await? {
|
||||||
if cfg.feeds.contains_key(&m.id) {
|
if cfg.feeds.contains_key(&m.id) {
|
||||||
continue; // promoted to config at some point; that entry wins
|
continue; // promoted to config at some point; that entry wins
|
||||||
}
|
}
|
||||||
@@ -953,10 +974,10 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
|||||||
// skip them here regardless so a row that slips through is never scanned.
|
// skip them here regardless so a row that slips through is never scanned.
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let base = parent
|
let base = match parent.and_then(|p| p.folder.clone()) {
|
||||||
.and_then(|p| p.folder.clone())
|
Some(folder) => folder,
|
||||||
.or_else(|| ctx.db.feed_summary(&m.group_id).ok().and_then(|s| s.title))
|
None => ctx.db.feed_summary(&m.group_id).await.ok().and_then(|s| s.title).unwrap_or_else(|| m.group_id.clone()),
|
||||||
.unwrap_or_else(|| m.group_id.clone());
|
};
|
||||||
let title = m.title.clone().unwrap_or_else(|| m.id.clone());
|
let title = m.title.clone().unwrap_or_else(|| m.id.clone());
|
||||||
out.push(Sub {
|
out.push(Sub {
|
||||||
id: m.id.clone(),
|
id: m.id.clone(),
|
||||||
@@ -988,17 +1009,17 @@ pub fn subscriptions(ctx: &Ctx) -> Result<Vec<Sub>> {
|
|||||||
/// itself is removed, since `subscriptions()` would otherwise keep scanning them under a
|
/// itself is removed, since `subscriptions()` would otherwise keep scanning them under a
|
||||||
/// fallback policy meant for a feed with no parent at all. A feed promoted to config is not
|
/// fallback policy meant for a feed with no parent at all. A feed promoted to config is not
|
||||||
/// derived any more, so it is only unmanaged.
|
/// derived any more, so it is only unmanaged.
|
||||||
pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
|
pub async fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
for m in ctx.db.managed_feeds()?.into_iter().filter(|m| m.group_id == parent_id) {
|
for m in ctx.db.managed_feeds().await?.into_iter().filter(|m| m.group_id == parent_id) {
|
||||||
if cfg.feeds.contains_key(&m.id) {
|
if cfg.feeds.contains_key(&m.id) {
|
||||||
// Scanned from its config entry and still read. Dropped as derived, its stored
|
// Scanned from its config entry and still read. Dropped as derived, its stored
|
||||||
// entries would go with it: davewiner's 11 were promoted without being unmanaged.
|
// entries would go with it: davewiner's 11 were promoted without being unmanaged.
|
||||||
ctx.db.unmanage(&m.id)?;
|
ctx.db.unmanage(&m.id).await?;
|
||||||
} else if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 {
|
} else if ctx.db.downloaded_count(&m.id).await.unwrap_or(1) > 0 {
|
||||||
ctx.db.set_orphaned(&m.id, true)?;
|
ctx.db.set_orphaned(&m.id, true).await?;
|
||||||
} else {
|
} else {
|
||||||
ctx.db.drop_managed(&m.id)?;
|
ctx.db.drop_managed(&m.id).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
@@ -1008,18 +1029,18 @@ pub fn retire_group(ctx: &Ctx, parent_id: &str) -> Result<()> {
|
|||||||
/// dropped or unmanaged. An OPML removed before `retire_group` existed left its feeds behind:
|
/// dropped or unmanaged. An OPML removed before `retire_group` existed left its feeds behind:
|
||||||
/// davewiner's 922 were skipped by every scan and never cleared, and their stale errors were
|
/// davewiner's 922 were skipped by every scan and never cleared, and their stale errors were
|
||||||
/// most of the ones stored.
|
/// most of the ones stored.
|
||||||
fn retire_stranded(ctx: &Ctx) -> Result<usize> {
|
async fn retire_stranded(ctx: &Ctx) -> Result<usize> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let before = ctx.db.managed_feeds()?;
|
let before = ctx.db.managed_feeds().await?;
|
||||||
let stranded: std::collections::BTreeSet<&str> = before
|
let stranded: std::collections::BTreeSet<&str> = before
|
||||||
.iter()
|
.iter()
|
||||||
.map(|m| m.group_id.as_str())
|
.map(|m| m.group_id.as_str())
|
||||||
.filter(|g| !cfg.feeds.contains_key(*g))
|
.filter(|g| !cfg.feeds.contains_key(*g))
|
||||||
.collect();
|
.collect();
|
||||||
for group in stranded {
|
for group in stranded {
|
||||||
retire_group(ctx, group)?;
|
retire_group(ctx, group).await?;
|
||||||
}
|
}
|
||||||
Ok(before.len() - ctx.db.managed_feeds()?.len())
|
Ok(before.len() - ctx.db.managed_feeds().await?.len())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Seconds to wait before re-checking a feed.
|
/// Seconds to wait before re-checking a feed.
|
||||||
@@ -1064,19 +1085,19 @@ async fn scan_one(
|
|||||||
if feed::is_patreon_creator(&feed_cfg.url) {
|
if feed::is_patreon_creator(&feed_cfg.url) {
|
||||||
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
|
match feed::patreon_shows(&ctx.client, &feed_cfg.url).await {
|
||||||
Ok((name, shows)) if shows.len() > 1 => {
|
Ok((name, shows)) if shows.len() > 1 => {
|
||||||
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
ctx.db.touch_feed(id, &feed_cfg.url).await?;
|
||||||
if let Some(name) = name {
|
if let Some(name) = name {
|
||||||
ctx.db.set_title(id, &name)?;
|
ctx.db.set_title(id, &name).await?;
|
||||||
}
|
}
|
||||||
// Read as one feed before it was split, it listed every show's items in one
|
// Read as one feed before it was split, it listed every show's items in one
|
||||||
// heap. The items go; its files and read state move to each show as the show
|
// heap. The items go; its files and read state move to each show as the show
|
||||||
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
|
// lists them (`Db::adopt`), so no show comes up empty for want of a URL.
|
||||||
ctx.db.clear_entries(id)?;
|
ctx.db.clear_entries(id).await?;
|
||||||
return sync_group(ctx, id, feed_cfg, &shows).await;
|
return sync_group(ctx, id, feed_cfg, &shows).await;
|
||||||
}
|
}
|
||||||
Ok(_) => {} // One show: the creator's feed is that show.
|
Ok(_) => {} // One show: the creator's feed is that show.
|
||||||
// Already split: keep the shows it has rather than read the creator as one heap.
|
// Already split: keep the shows it has rather than read the creator as one heap.
|
||||||
Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e),
|
Err(e) if ctx.db.managed_feeds().await?.iter().any(|m| m.group_id == id) => return Err(e),
|
||||||
Err(e) => tracing::warn!(
|
Err(e) => tracing::warn!(
|
||||||
feed = id,
|
feed = id,
|
||||||
error = %format!("{e:#}"),
|
error = %format!("{e:#}"),
|
||||||
@@ -1097,29 +1118,29 @@ async fn scan_one(
|
|||||||
// from backup, a manual edit, a cleanup that removed entries. Believe the database over
|
// from backup, a manual edit, a cleanup that removed entries. Believe the database over
|
||||||
// the validator: drop it and ask again, or the feed stays empty until the publisher
|
// the validator: drop it and ask again, or the feed stays empty until the publisher
|
||||||
// happens to change something.
|
// happens to change something.
|
||||||
if matches!(fetched, feed::Fetched::NotModified) && ctx.db.feed_summary(id)?.entries == 0 {
|
if matches!(fetched, feed::Fetched::NotModified) && ctx.db.feed_summary(id).await?.entries == 0 {
|
||||||
tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator");
|
tracing::info!(feed = id, "not modified, but nothing stored; refetching without the validator");
|
||||||
ctx.db.clear_validators(id)?;
|
ctx.db.clear_validators(id).await?;
|
||||||
fetched = feed::fetch(&ctx.client, feed_cfg, None, None).await?;
|
fetched = feed::fetch(&ctx.client, feed_cfg, None, None).await?;
|
||||||
}
|
}
|
||||||
|
|
||||||
let (bytes, etag, last_modified) = match fetched {
|
let (bytes, etag, last_modified) = match fetched {
|
||||||
feed::Fetched::NotModified => {
|
feed::Fetched::NotModified => {
|
||||||
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
ctx.db.touch_feed(id, &feed_cfg.url).await?;
|
||||||
return Ok(Outcome::NotModified);
|
return Ok(Outcome::NotModified);
|
||||||
}
|
}
|
||||||
feed::Fetched::Body { bytes, etag, last_modified } => (bytes, etag, last_modified),
|
feed::Fetched::Body { bytes, etag, last_modified } => (bytes, etag, last_modified),
|
||||||
};
|
};
|
||||||
|
|
||||||
if bytes.iter().all(u8::is_ascii_whitespace) {
|
if bytes.iter().all(u8::is_ascii_whitespace) {
|
||||||
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
ctx.db.touch_feed(id, &feed_cfg.url).await?;
|
||||||
return Ok(Outcome::Empty);
|
return Ok(Outcome::Empty);
|
||||||
}
|
}
|
||||||
|
|
||||||
// A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml"
|
// A subscribed OPML is a list of feeds, not a feed. The original matched on a ".opml"
|
||||||
// URL; sniffing the body also catches one served from a URL without that extension.
|
// URL; sniffing the body also catches one served from a URL without that extension.
|
||||||
if feed::is_opml(&bytes) {
|
if feed::is_opml(&bytes) {
|
||||||
ctx.db.touch_feed(id, &feed_cfg.url)?;
|
ctx.db.touch_feed(id, &feed_cfg.url).await?;
|
||||||
return sync_opml(ctx, id, feed_cfg, &bytes).await;
|
return sync_opml(ctx, id, feed_cfg, &bytes).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1133,29 +1154,29 @@ async fn scan_one(
|
|||||||
parsed.ttl_mins,
|
parsed.ttl_mins,
|
||||||
parsed.image.as_deref(),
|
parsed.image.as_deref(),
|
||||||
parsed.category.as_deref(),
|
parsed.category.as_deref(),
|
||||||
)?;
|
).await?;
|
||||||
|
|
||||||
let policy = policy_for(ctx, id, feed_cfg)?;
|
let policy = policy_for(ctx, id, feed_cfg).await?;
|
||||||
if let Some(parent) = &feed_cfg.group {
|
if let Some(parent) = &feed_cfg.group {
|
||||||
let listed: Vec<(&str, &str)> = parsed
|
let listed: Vec<(&str, &str)> = parsed
|
||||||
.entries
|
.entries
|
||||||
.iter()
|
.iter()
|
||||||
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
|
.flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str())))
|
||||||
.collect();
|
.collect();
|
||||||
ctx.db.adopt(parent, id, &listed)?;
|
ctx.db.adopt(parent, id, &listed).await?;
|
||||||
}
|
}
|
||||||
// Verdicts are recorded in `state`, so the download queue below is just "everything still
|
// Verdicts are recorded in `state`, so the download queue below is just "everything still
|
||||||
// pending". A filter's verdict is looked at again on every scan, though: made once, at
|
// pending". A filter's verdict is looked at again on every scan, though: made once, at
|
||||||
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
|
// discovery, it outlived the setting behind it, and allowing explicit items afterwards
|
||||||
// changed nothing however often the feed was scanned.
|
// changed nothing however often the feed was scanned.
|
||||||
let skipped = ctx.db.skipped_by_filter(id)?;
|
let skipped = ctx.db.skipped_by_filter(id).await?;
|
||||||
let mut scan = Scan::default();
|
let mut scan = Scan::default();
|
||||||
for entry in &parsed.entries {
|
for entry in &parsed.entries {
|
||||||
if ctx.db.record_entry(id, entry)? {
|
if ctx.db.record_entry(id, entry).await? {
|
||||||
scan.new_entries += 1;
|
scan.new_entries += 1;
|
||||||
}
|
}
|
||||||
for enc in &entry.enclosures {
|
for enc in &entry.enclosures {
|
||||||
let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? {
|
let was = if ctx.db.record_enclosure(id, &entry.guid, enc).await? {
|
||||||
None
|
None
|
||||||
} else if let Some(reason) = skipped.get(&enc.url) {
|
} else if let Some(reason) = skipped.get(&enc.url) {
|
||||||
Some(reason.as_str())
|
Some(reason.as_str())
|
||||||
@@ -1165,8 +1186,8 @@ async fn scan_one(
|
|||||||
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
|
let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc);
|
||||||
if now != was {
|
if now != was {
|
||||||
match now {
|
match now {
|
||||||
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?,
|
Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason)).await?,
|
||||||
None => ctx.db.mark_enclosure(&enc.url, "pending", None)?,
|
None => ctx.db.mark_enclosure(&enc.url, "pending", None).await?,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1178,10 +1199,10 @@ async fn scan_one(
|
|||||||
let folder = download::folder_for(&cfg, id, feed_cfg, parsed.title.as_deref());
|
let folder = download::folder_for(&cfg, id, feed_cfg, parsed.title.as_deref());
|
||||||
let dest_dir = cfg.general.download_dir.join(&folder);
|
let dest_dir = cfg.general.download_dir.join(&folder);
|
||||||
|
|
||||||
for item in ctx.db.pending(id, budget)? {
|
for item in ctx.db.pending(id, budget).await? {
|
||||||
if download::looks_like_torrent(&item.url, item.mime.as_deref()) {
|
if download::looks_like_torrent(&item.url, item.mime.as_deref()) {
|
||||||
if !ctx.cfg().torrent.enabled {
|
if !ctx.cfg().torrent.enabled {
|
||||||
ctx.db.mark_enclosure(&item.url, "skipped", Some("torrents disabled"))?;
|
ctx.db.mark_enclosure(&item.url, "skipped", Some("torrents disabled")).await?;
|
||||||
ctx.out.emit(Event::TorrentDeferred {
|
ctx.out.emit(Event::TorrentDeferred {
|
||||||
feed: id.to_string(),
|
feed: id.to_string(),
|
||||||
url: item.url.clone(),
|
url: item.url.clone(),
|
||||||
@@ -1191,14 +1212,14 @@ async fn scan_one(
|
|||||||
}
|
}
|
||||||
if ctx.detach_torrents {
|
if ctx.detach_torrents {
|
||||||
// 'downloading' keeps the next scan from queueing it a second time.
|
// 'downloading' keeps the next scan from queueing it a second time.
|
||||||
ctx.db.mark_enclosure(&item.url, "downloading", None)?;
|
ctx.db.mark_enclosure(&item.url, "downloading", None).await?;
|
||||||
spawn_torrent(ctx, id.to_string(), item.id, item.url.clone(), dest_dir.clone());
|
spawn_torrent(ctx, id.to_string(), item.id, item.url.clone(), dest_dir.clone());
|
||||||
scan.torrents += 1;
|
scan.torrents += 1;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
match torrent_one(ctx, id, item.id, &item.url, &dest_dir).await {
|
match torrent_one(ctx, id, item.id, &item.url, &dest_dir).await {
|
||||||
Ok((path, bytes)) => {
|
Ok((path, bytes)) => {
|
||||||
ctx.db.mark_downloaded(&item.url, &path, bytes)?;
|
ctx.db.mark_downloaded(&item.url, &path, bytes).await?;
|
||||||
ctx.out.emit(Event::DownloadDone {
|
ctx.out.emit(Event::DownloadDone {
|
||||||
feed: id.to_string(),
|
feed: id.to_string(),
|
||||||
enclosure: item.id,
|
enclosure: item.id,
|
||||||
@@ -1216,7 +1237,7 @@ async fn scan_one(
|
|||||||
url: item.url.clone(),
|
url: item.url.clone(),
|
||||||
msg: msg.clone(),
|
msg: msg.clone(),
|
||||||
});
|
});
|
||||||
ctx.db.mark_enclosure(&item.url, "error", Some(&msg))?;
|
ctx.db.mark_enclosure(&item.url, "error", Some(&msg)).await?;
|
||||||
scan.failed += 1;
|
scan.failed += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1241,7 +1262,7 @@ async fn scan_one(
|
|||||||
url: item.url.clone(),
|
url: item.url.clone(),
|
||||||
msg: msg.clone(),
|
msg: msg.clone(),
|
||||||
});
|
});
|
||||||
ctx.db.mark_enclosure(&item.url, "error", Some(&msg))?;
|
ctx.db.mark_enclosure(&item.url, "error", Some(&msg)).await?;
|
||||||
scan.failed += 1;
|
scan.failed += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1259,7 +1280,7 @@ async fn sync_opml(
|
|||||||
) -> Result<Outcome> {
|
) -> Result<Outcome> {
|
||||||
let listed = feed::parse_opml(bytes)?;
|
let listed = feed::parse_opml(bytes)?;
|
||||||
if let Some(title) = feed::opml_title(bytes) {
|
if let Some(title) = feed::opml_title(bytes) {
|
||||||
ctx.db.set_title(parent_id, &title)?;
|
ctx.db.set_title(parent_id, &title).await?;
|
||||||
}
|
}
|
||||||
sync_group(ctx, parent_id, parent, &listed).await
|
sync_group(ctx, parent_id, parent, &listed).await
|
||||||
}
|
}
|
||||||
@@ -1278,13 +1299,13 @@ async fn sync_group(
|
|||||||
listed: &[(String, String)],
|
listed: &[(String, String)],
|
||||||
) -> Result<Outcome> {
|
) -> Result<Outcome> {
|
||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let existing = ctx.db.managed_feeds()?;
|
let existing = ctx.db.managed_feeds().await?;
|
||||||
let mut added = vec![];
|
let mut added = vec![];
|
||||||
|
|
||||||
for (title, url) in listed {
|
for (title, url) in listed {
|
||||||
// Already known, whether derived or promoted into the config.
|
// Already known, whether derived or promoted into the config.
|
||||||
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
if let Some(m) = existing.iter().find(|m| &m.url == url) {
|
||||||
ctx.db.upsert_managed(&m.id, url, title, parent_id)?;
|
ctx.db.upsert_managed(&m.id, url, title, parent_id).await?;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
// A Patreon show you added by hand may be spelled differently from the one listed.
|
// A Patreon show you added by hand may be spelled differently from the one listed.
|
||||||
@@ -1292,7 +1313,7 @@ async fn sync_group(
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
// A removed feed keeps its rows, so its id is only free again for the same feed.
|
// A removed feed keeps its rows, so its id is only free again for the same feed.
|
||||||
let known = ctx.db.feed_urls()?;
|
let known = ctx.db.feed_urls().await?;
|
||||||
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
let taken: std::collections::BTreeMap<String, config::Feed> = cfg
|
||||||
.feeds
|
.feeds
|
||||||
.keys()
|
.keys()
|
||||||
@@ -1302,7 +1323,7 @@ async fn sync_group(
|
|||||||
.map(|id| (id.clone(), parent.clone()))
|
.map(|id| (id.clone(), parent.clone()))
|
||||||
.collect();
|
.collect();
|
||||||
let id = config::unique_slug(title, &taken);
|
let id = config::unique_slug(title, &taken);
|
||||||
ctx.db.upsert_managed(&id, url, title, parent_id)?;
|
ctx.db.upsert_managed(&id, url, title, parent_id).await?;
|
||||||
added.push(id);
|
added.push(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1310,15 +1331,15 @@ async fn sync_group(
|
|||||||
// subscription means. Their own feeds are untouched.
|
// subscription means. Their own feeds are untouched.
|
||||||
for id in ctx
|
for id in ctx
|
||||||
.db
|
.db
|
||||||
.managed_feeds()?
|
.managed_feeds().await?
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|m| m.group_id == parent_id)
|
.filter(|m| m.group_id == parent_id)
|
||||||
.map(|m| m.id.clone())
|
.map(|m| m.id.clone())
|
||||||
.chain(std::iter::once(parent_id.to_string()))
|
.chain(std::iter::once(parent_id.to_string()))
|
||||||
{
|
{
|
||||||
for user in ctx.db.users()? {
|
for user in ctx.db.users().await? {
|
||||||
if ctx.db.subscription(user.id, parent_id)?.is_some() {
|
if ctx.db.subscription(user.id, parent_id).await?.is_some() {
|
||||||
ctx.db.subscribe(user.id, &id)?;
|
ctx.db.subscribe(user.id, &id).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1330,13 +1351,13 @@ async fn sync_group(
|
|||||||
if listed.iter().any(|(_, u)| u == &m.url) {
|
if listed.iter().any(|(_, u)| u == &m.url) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if ctx.db.downloaded_count(&m.id).unwrap_or(1) > 0 {
|
if ctx.db.downloaded_count(&m.id).await.unwrap_or(1) > 0 {
|
||||||
// Never orphan a downloaded file: keep the feed and say why in the UI.
|
// Never orphan a downloaded file: keep the feed and say why in the UI.
|
||||||
ctx.db.set_orphaned(&m.id, true)?;
|
ctx.db.set_orphaned(&m.id, true).await?;
|
||||||
kept += 1;
|
kept += 1;
|
||||||
tracing::info!(feed = %m.id, "dropped from the OPML but has downloads; keeping it");
|
tracing::info!(feed = %m.id, "dropped from the OPML but has downloads; keeping it");
|
||||||
} else {
|
} else {
|
||||||
ctx.db.drop_managed(&m.id)?;
|
ctx.db.drop_managed(&m.id).await?;
|
||||||
removed += 1;
|
removed += 1;
|
||||||
tracing::info!(feed = %m.id, "dropped from the OPML with nothing downloaded; removed");
|
tracing::info!(feed = %m.id, "dropped from the OPML with nothing downloaded; removed");
|
||||||
}
|
}
|
||||||
@@ -1403,9 +1424,9 @@ pub struct Policy {
|
|||||||
pub budget: usize,
|
pub budget: usize,
|
||||||
}
|
}
|
||||||
|
|
||||||
fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
async fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> {
|
||||||
let global = ctx.cfg().general.max_new_per_check;
|
let global = ctx.cfg().general.max_new_per_check;
|
||||||
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global))
|
Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref()).await?, feed_cfg, global))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy {
|
||||||
@@ -1484,14 +1505,14 @@ async fn fetch_one(
|
|||||||
// as if it were an episode.
|
// as if it were an episode.
|
||||||
let _ = tokio::fs::remove_file(&got.tmp).await;
|
let _ = tokio::fs::remove_file(&got.tmp).await;
|
||||||
if !ctx.cfg().torrent.enabled {
|
if !ctx.cfg().torrent.enabled {
|
||||||
ctx.db.mark_enclosure(url, "skipped", Some("torrents disabled"))?;
|
ctx.db.mark_enclosure(url, "skipped", Some("torrents disabled")).await?;
|
||||||
anyhow::bail!("body is a torrent and torrents are disabled");
|
anyhow::bail!("body is a torrent and torrents are disabled");
|
||||||
}
|
}
|
||||||
return torrent_one(ctx, feed_id, enclosure, url, dest_dir).await;
|
return torrent_one(ctx, feed_id, enclosure, url, dest_dir).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
let path = download::place(&got, dest_dir).await?;
|
let path = download::place(&got, dest_dir).await?;
|
||||||
ctx.db.mark_downloaded(url, &path, got.bytes)?;
|
ctx.db.mark_downloaded(url, &path, got.bytes).await?;
|
||||||
Ok((path, got.bytes))
|
Ok((path, got.bytes))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1509,7 +1530,7 @@ fn spawn_torrent(ctx: &Arc<Ctx>, feed_id: String, enclosure: i64, url: String, d
|
|||||||
let db = &ctx.db;
|
let db = &ctx.db;
|
||||||
match outcome {
|
match outcome {
|
||||||
Ok((path, bytes)) => {
|
Ok((path, bytes)) => {
|
||||||
if let Err(e) = db.mark_downloaded(&url, &path, bytes) {
|
if let Err(e) = db.mark_downloaded(&url, &path, bytes).await {
|
||||||
tracing::warn!(error = ?e, "could not record the finished torrent");
|
tracing::warn!(error = ?e, "could not record the finished torrent");
|
||||||
}
|
}
|
||||||
ctx.out.emit(Event::DownloadDone {
|
ctx.out.emit(Event::DownloadDone {
|
||||||
@@ -1522,7 +1543,7 @@ fn spawn_torrent(ctx: &Arc<Ctx>, feed_id: String, enclosure: i64, url: String, d
|
|||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let msg = format!("{e:#}");
|
let msg = format!("{e:#}");
|
||||||
let _ = db.mark_enclosure(&url, "error", Some(&msg));
|
let _ = db.mark_enclosure(&url, "error", Some(&msg)).await;
|
||||||
ctx.out.emit(Event::DownloadError { feed: feed_id, enclosure, url, msg });
|
ctx.out.emit(Event::DownloadError { feed: feed_id, enclosure, url, msg });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1535,14 +1556,14 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
|
|||||||
let cfg = ctx.cfg();
|
let cfg = ctx.cfg();
|
||||||
let enc = ctx
|
let enc = ctx
|
||||||
.db
|
.db
|
||||||
.enclosure(id)?
|
.enclosure(id).await?
|
||||||
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
|
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
|
||||||
if enc.path.is_some() {
|
if enc.path.is_some() {
|
||||||
return Ok(()); // Already here.
|
return Ok(()); // Already here.
|
||||||
}
|
}
|
||||||
// Must look through the derived feeds too: anything inside an OPML subscription has
|
// Must look through the derived feeds too: anything inside an OPML subscription has
|
||||||
// no config entry, so a config-only lookup called every one of them "unsubscribed".
|
// no config entry, so a config-only lookup called every one of them "unsubscribed".
|
||||||
let subs = subscriptions(ctx)?;
|
let subs = subscriptions(ctx).await?;
|
||||||
let feed_cfg = subs
|
let feed_cfg = subs
|
||||||
.iter()
|
.iter()
|
||||||
.find(|s| s.id == enc.feed_id)
|
.find(|s| s.id == enc.feed_id)
|
||||||
@@ -1550,14 +1571,14 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
|
|||||||
.ok_or_else(|| anyhow::anyhow!("enclosure {id} belongs to unsubscribed feed {:?}", enc.feed_id))?;
|
.ok_or_else(|| anyhow::anyhow!("enclosure {id} belongs to unsubscribed feed {:?}", enc.feed_id))?;
|
||||||
let feed_cfg = &feed_cfg;
|
let feed_cfg = &feed_cfg;
|
||||||
|
|
||||||
let title = ctx.db.feed_summary(&enc.feed_id)?.title;
|
let title = ctx.db.feed_summary(&enc.feed_id).await?.title;
|
||||||
let folder = download::folder_for(&cfg, &enc.feed_id, feed_cfg, title.as_deref());
|
let folder = download::folder_for(&cfg, &enc.feed_id, feed_cfg, title.as_deref());
|
||||||
let dest_dir = cfg.general.download_dir.join(&folder);
|
let dest_dir = cfg.general.download_dir.join(&folder);
|
||||||
|
|
||||||
ctx.out.emit(Event::FeedStart { feed: enc.feed_id.clone() });
|
ctx.out.emit(Event::FeedStart { feed: enc.feed_id.clone() });
|
||||||
let is_torrent = download::looks_like_torrent(&enc.url, enc.mime.as_deref());
|
let is_torrent = download::looks_like_torrent(&enc.url, enc.mime.as_deref());
|
||||||
if is_torrent && cfg.torrent.enabled && ctx.detach_torrents {
|
if is_torrent && cfg.torrent.enabled && ctx.detach_torrents {
|
||||||
ctx.db.mark_enclosure(&enc.url, "downloading", None)?;
|
ctx.db.mark_enclosure(&enc.url, "downloading", None).await?;
|
||||||
spawn_torrent(ctx, enc.feed_id.clone(), enc.id, enc.url.clone(), dest_dir);
|
spawn_torrent(ctx, enc.feed_id.clone(), enc.id, enc.url.clone(), dest_dir);
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
@@ -1573,7 +1594,7 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
|
|||||||
|
|
||||||
match result {
|
match result {
|
||||||
Ok((path, bytes)) => {
|
Ok((path, bytes)) => {
|
||||||
ctx.db.mark_downloaded(&enc.url, &path, bytes)?;
|
ctx.db.mark_downloaded(&enc.url, &path, bytes).await?;
|
||||||
ctx.out.emit(Event::DownloadDone {
|
ctx.out.emit(Event::DownloadDone {
|
||||||
feed: enc.feed_id.clone(),
|
feed: enc.feed_id.clone(),
|
||||||
enclosure: enc.id,
|
enclosure: enc.id,
|
||||||
@@ -1584,7 +1605,7 @@ async fn download_one(ctx: &Arc<Ctx>, id: i64) -> Result<()> {
|
|||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let msg = format!("{e:#}");
|
let msg = format!("{e:#}");
|
||||||
ctx.db.mark_enclosure(&enc.url, "error", Some(&msg))?;
|
ctx.db.mark_enclosure(&enc.url, "error", Some(&msg)).await?;
|
||||||
ctx.out.emit(Event::DownloadError {
|
ctx.out.emit(Event::DownloadError {
|
||||||
feed: enc.feed_id.clone(),
|
feed: enc.feed_id.clone(),
|
||||||
enclosure: enc.id,
|
enclosure: enc.id,
|
||||||
@@ -1690,8 +1711,8 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[tokio::test]
|
||||||
fn a_shared_feed_is_fetched_for_whoever_wants_the_most() {
|
async fn a_shared_feed_is_fetched_for_whoever_wants_the_most() {
|
||||||
// Nobody subscribed: the feed's own settings stand, as in a single-user install.
|
// Nobody subscribed: the feed's own settings stand, as in a single-user install.
|
||||||
let p = merge_policy(&[], &feed(), 3);
|
let p = merge_policy(&[], &feed(), 3);
|
||||||
assert!(p.auto_download);
|
assert!(p.auto_download);
|
||||||
@@ -1721,10 +1742,10 @@ mod tests {
|
|||||||
assert!(p.auto_download);
|
assert!(p.auto_download);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn test_ctx(cfg: config::Config) -> Ctx {
|
async fn test_ctx(cfg: config::Config) -> Ctx {
|
||||||
Ctx {
|
Ctx {
|
||||||
cfg: std::sync::RwLock::new(Arc::new(cfg)),
|
cfg: std::sync::RwLock::new(Arc::new(cfg)),
|
||||||
db: db::Db::memory().unwrap(),
|
db: db::Db::memory().await.unwrap(),
|
||||||
client: reqwest::Client::new(),
|
client: reqwest::Client::new(),
|
||||||
out: Emitter::terminal(),
|
out: Emitter::terminal(),
|
||||||
torrents: tokio::sync::OnceCell::new(),
|
torrents: tokio::sync::OnceCell::new(),
|
||||||
@@ -1734,52 +1755,52 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[tokio::test]
|
||||||
fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() {
|
async fn a_derived_feed_is_not_scanned_once_its_opml_leaves_config() {
|
||||||
// davewiner: the OPML subscription left config.toml, but its 922 derived rows
|
// davewiner: the OPML subscription left config.toml, but its 922 derived rows
|
||||||
// stayed in the database and kept being scanned under the no-parent fallback.
|
// stayed in the database and kept being scanned under the no-parent fallback.
|
||||||
let ctx = test_ctx(config::Config::default());
|
let ctx = test_ctx(config::Config::default()).await;
|
||||||
ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").unwrap();
|
ctx.db.upsert_managed("child", "http://x/child.xml", "Child", "gone-opml").await.unwrap();
|
||||||
assert!(
|
assert!(
|
||||||
subscriptions(&ctx).unwrap().iter().all(|s| s.id != "child"),
|
subscriptions(&ctx).await.unwrap().iter().all(|s| s.id != "child"),
|
||||||
"a derived feed whose parent is gone from config must not be scanned"
|
"a derived feed whose parent is gone from config must not be scanned"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[tokio::test]
|
||||||
fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() {
|
async fn retiring_a_group_drops_what_was_never_downloaded_and_orphans_the_rest() {
|
||||||
let ctx = test_ctx(config::Config::default());
|
let ctx = test_ctx(config::Config::default()).await;
|
||||||
ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").unwrap();
|
ctx.db.upsert_managed("empty", "http://x/empty.xml", "Empty", "parent").await.unwrap();
|
||||||
ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").unwrap();
|
ctx.db.upsert_managed("has-file", "http://x/has-file.xml", "Has File", "parent").await.unwrap();
|
||||||
let enc = feed::Enclosure { url: "http://x/ep.mp3".into(), mime: None, length: None };
|
let enc = feed::Enclosure { url: "http://x/ep.mp3".into(), mime: None, length: None };
|
||||||
ctx.db.record_enclosure("has-file", "g1", &enc).unwrap();
|
ctx.db.record_enclosure("has-file", "g1", &enc).await.unwrap();
|
||||||
ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).unwrap();
|
ctx.db.mark_downloaded(&enc.url, std::path::Path::new("/downloads/ep.mp3"), 1).await.unwrap();
|
||||||
|
|
||||||
retire_group(&ctx, "parent").unwrap();
|
retire_group(&ctx, "parent").await.unwrap();
|
||||||
|
|
||||||
let managed = ctx.db.managed_feeds().unwrap();
|
let managed = ctx.db.managed_feeds().await.unwrap();
|
||||||
assert!(!managed.iter().any(|m| m.id == "empty"), "nothing downloaded, so it is forgotten");
|
assert!(!managed.iter().any(|m| m.id == "empty"), "nothing downloaded, so it is forgotten");
|
||||||
assert!(managed.iter().any(|m| m.id == "has-file"), "has a file on disk, so it is kept");
|
assert!(managed.iter().any(|m| m.id == "has-file"), "has a file on disk, so it is kept");
|
||||||
assert!(ctx.db.feed_summary("has-file").unwrap().orphaned, "and flagged as orphaned");
|
assert!(ctx.db.feed_summary("has-file").await.unwrap().orphaned, "and flagged as orphaned");
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[tokio::test]
|
||||||
fn a_stranded_group_is_retired_but_a_promoted_feed_keeps_its_entries() {
|
async fn a_stranded_group_is_retired_but_a_promoted_feed_keeps_its_entries() {
|
||||||
// davewiner: the OPML left config before retire_group existed, and 11 of its feeds
|
// davewiner: the OPML left config before retire_group existed, and 11 of its feeds
|
||||||
// promoted to config since still said managed = 1.
|
// promoted to config since still said managed = 1.
|
||||||
let mut cfg = config::Config::default();
|
let mut cfg = config::Config::default();
|
||||||
cfg.feeds.insert("promoted".into(), feed());
|
cfg.feeds.insert("promoted".into(), feed());
|
||||||
cfg.feeds.insert("live-opml".into(), feed());
|
cfg.feeds.insert("live-opml".into(), feed());
|
||||||
let ctx = test_ctx(cfg);
|
let ctx = test_ctx(cfg).await;
|
||||||
ctx.db.upsert_managed("promoted", "http://x/p.xml", "Promoted", "gone-opml").unwrap();
|
ctx.db.upsert_managed("promoted", "http://x/p.xml", "Promoted", "gone-opml").await.unwrap();
|
||||||
ctx.db.upsert_managed("empty", "http://x/e.xml", "Empty", "gone-opml").unwrap();
|
ctx.db.upsert_managed("empty", "http://x/e.xml", "Empty", "gone-opml").await.unwrap();
|
||||||
ctx.db.upsert_managed("listed", "http://x/l.xml", "Listed", "live-opml").unwrap();
|
ctx.db.upsert_managed("listed", "http://x/l.xml", "Listed", "live-opml").await.unwrap();
|
||||||
ctx.db.record_entry("promoted", &feed::Entry { guid: "g1".into(), ..Default::default() }).unwrap();
|
ctx.db.record_entry("promoted", &feed::Entry { guid: "g1".into(), ..Default::default() }).await.unwrap();
|
||||||
|
|
||||||
assert_eq!(retire_stranded(&ctx).unwrap(), 2, "empty dropped, promoted unmanaged");
|
assert_eq!(retire_stranded(&ctx).await.unwrap(), 2, "empty dropped, promoted unmanaged");
|
||||||
|
|
||||||
let managed: Vec<String> = ctx.db.managed_feeds().unwrap().into_iter().map(|m| m.id).collect();
|
let managed: Vec<String> = ctx.db.managed_feeds().await.unwrap().into_iter().map(|m| m.id).collect();
|
||||||
assert_eq!(managed, ["listed"], "a group still in config is left alone");
|
assert_eq!(managed, ["listed"], "a group still in config is left alone");
|
||||||
assert_eq!(ctx.db.feed_summary("promoted").unwrap().entries, 1, "its entries survive");
|
assert_eq!(ctx.db.feed_summary("promoted").await.unwrap().entries, 1, "its entries survive");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -45,28 +45,28 @@ pub fn aged(candidates: &[Candidate], cutoff: i64) -> Vec<Candidate> {
|
|||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
|
pub async fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
|
||||||
let mut report = Report::default();
|
let mut report = Report::default();
|
||||||
|
|
||||||
// Someone may have deleted a file by hand; the row must stop claiming it exists.
|
// Someone may have deleted a file by hand; the row must stop claiming it exists.
|
||||||
for (id, path) in db.missing_files()? {
|
for (id, path) in db.missing_files().await? {
|
||||||
if !dry_run {
|
if !dry_run {
|
||||||
db.mark_reaped(id)?;
|
db.mark_reaped(id).await?;
|
||||||
}
|
}
|
||||||
tracing::debug!(path, "file gone, row reaped");
|
tracing::debug!(path, "file gone, row reaped");
|
||||||
report.reconciled += 1;
|
report.reconciled += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
let candidates = db.reap_candidates()?;
|
let candidates = db.reap_candidates().await?;
|
||||||
|
|
||||||
if cfg.general.max_age_days > 0 {
|
if cfg.general.max_age_days > 0 {
|
||||||
let cutoff = now() - (cfg.general.max_age_days * 86_400) as i64;
|
let cutoff = now() - (cfg.general.max_age_days * 86_400) as i64;
|
||||||
report.aged_out = aged(&candidates, cutoff);
|
report.aged_out = aged(&candidates, cutoff);
|
||||||
for c in &report.aged_out {
|
for c in &report.aged_out {
|
||||||
report.bytes_freed += remove(db, c, dry_run)?;
|
report.bytes_freed += remove(db, c, dry_run).await?;
|
||||||
}
|
}
|
||||||
if !dry_run {
|
if !dry_run {
|
||||||
report.entries_pruned = db.prune_entries(cutoff)?;
|
report.entries_pruned = db.prune_entries(cutoff).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -81,14 +81,14 @@ pub fn run(cfg: &Config, db: &Db, dry_run: bool) -> Result<Report> {
|
|||||||
let total: u64 = remaining.iter().map(|c| c.bytes.max(0) as u64).sum();
|
let total: u64 = remaining.iter().map(|c| c.bytes.max(0) as u64).sum();
|
||||||
report.over_quota = pick(&remaining, total, limit);
|
report.over_quota = pick(&remaining, total, limit);
|
||||||
for c in &report.over_quota {
|
for c in &report.over_quota {
|
||||||
report.bytes_freed += remove(db, c, dry_run)?;
|
report.bytes_freed += remove(db, c, dry_run).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(report)
|
Ok(report)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
|
async fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
|
||||||
if dry_run {
|
if dry_run {
|
||||||
return Ok(c.bytes.max(0) as u64);
|
return Ok(c.bytes.max(0) as u64);
|
||||||
}
|
}
|
||||||
@@ -100,7 +100,7 @@ fn remove(db: &Db, c: &Candidate, dry_run: bool) -> Result<u64> {
|
|||||||
tracing::warn!(path = c.path, error = %e, "could not delete");
|
tracing::warn!(path = c.path, error = %e, "could not delete");
|
||||||
return Ok(0);
|
return Ok(0);
|
||||||
}
|
}
|
||||||
db.mark_reaped(c.id)?;
|
db.mark_reaped(c.id).await?;
|
||||||
Ok(size)
|
Ok(size)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,12 +149,12 @@ mod tests {
|
|||||||
// age_key 0 means "never recorded" -- not the same as "infinitely old".
|
// age_key 0 means "never recorded" -- not the same as "infinitely old".
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[tokio::test]
|
||||||
fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
|
async fn query_never_offers_a_file_anyone_starred_and_prefers_ones_everyone_read() {
|
||||||
// One file serves both subscribers, so it takes both of them to release it.
|
// One file serves both subscribers, so it takes both of them to release it.
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().await.unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1),(2,'sam',0);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true),(2,'sam',false);
|
||||||
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
|
INSERT INTO subscriptions (user_id, feed_id) VALUES (1,'f'),(2,'f');
|
||||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||||
('f', 'keep', 0),
|
('f', 'keep', 0),
|
||||||
@@ -163,20 +163,20 @@ mod tests {
|
|||||||
('f', 'read', 0);
|
('f', 'read', 0);
|
||||||
-- Starred by one of the two, so it stays whatever the other thinks.
|
-- Starred by one of the two, so it stays whatever the other thinks.
|
||||||
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
|
INSERT INTO entry_state (user_id, feed_id, guid, read, flagged) VALUES
|
||||||
(1, 'f', 'keep', 1, 1),
|
(1, 'f', 'keep', true, true),
|
||||||
(2, 'f', 'keep', 1, 0),
|
(2, 'f', 'keep', true, false),
|
||||||
(1, 'f', 'half', 1, 0),
|
(1, 'f', 'half', true, false),
|
||||||
(1, 'f', 'read', 1, 0),
|
(1, 'f', 'read', true, false),
|
||||||
(2, 'f', 'read', 1, 0);
|
(2, 'f', 'read', true, false);
|
||||||
INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES
|
INSERT INTO enclosures (id, feed_id, guid, url, path, bytes_done, state, downloaded_at) VALUES
|
||||||
(1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10),
|
(1, 'f', 'keep', 'u1', '/tmp/keep', 10, 'done', 10),
|
||||||
(2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20),
|
(2, 'f', 'half', 'u2', '/tmp/half', 10, 'done', 20),
|
||||||
(3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30),
|
(3, 'f', 'unread', 'u3', '/tmp/unread', 10, 'done', 30),
|
||||||
(4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);",
|
(4, 'f', 'read', 'u4', '/tmp/read', 10, 'done', 40);",
|
||||||
)
|
).await
|
||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
let got: Vec<i64> = db.reap_candidates().unwrap().iter().map(|c| c.id).collect();
|
let got: Vec<i64> = db.reap_candidates().await.unwrap().iter().map(|c| c.id).collect();
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
got,
|
got,
|
||||||
vec![4, 2, 3],
|
vec![4, 2, 3],
|
||||||
@@ -185,12 +185,12 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[tokio::test]
|
||||||
fn prune_keeps_entries_that_still_have_a_file() {
|
async fn prune_keeps_entries_that_still_have_a_file() {
|
||||||
let db = Db::memory().unwrap();
|
let db = Db::memory().await.unwrap();
|
||||||
db.exec_for_test(
|
db.exec_for_test(
|
||||||
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',1);
|
"INSERT INTO users (id, name, is_admin) VALUES (1,'ray',true);
|
||||||
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',1);
|
INSERT INTO entry_state (user_id, feed_id, guid, flagged) VALUES (1,'f','flagged',true);
|
||||||
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
INSERT INTO entries (feed_id, guid, first_seen) VALUES
|
||||||
('f', 'has-file', 100),
|
('f', 'has-file', 100),
|
||||||
('f', 'no-file', 100),
|
('f', 'no-file', 100),
|
||||||
@@ -198,9 +198,9 @@ mod tests {
|
|||||||
('f', 'recent', 900);
|
('f', 'recent', 900);
|
||||||
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
INSERT INTO enclosures (id, feed_id, guid, url, path, state) VALUES
|
||||||
(1, 'f', 'has-file', 'u1', '/tmp/x', 'done');",
|
(1, 'f', 'has-file', 'u1', '/tmp/x', 'done');",
|
||||||
)
|
).await
|
||||||
.unwrap();
|
.unwrap();
|
||||||
|
|
||||||
assert_eq!(db.prune_entries(500).unwrap(), 1, "only the old, fileless, unflagged one");
|
assert_eq!(db.prune_entries(500).await.unwrap(), 1, "only the old, fileless, unflagged one");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
180
src/web.rs
180
src/web.rs
@@ -107,16 +107,16 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
let mut user = None;
|
let mut user = None;
|
||||||
|
|
||||||
if let Some(name) = vouched {
|
if let Some(name) = vouched {
|
||||||
user = match state.ctx.db.user_by_name(&name) {
|
user = match state.ctx.db.user_by_name(&name).await {
|
||||||
Ok(Some(u)) => Some(u),
|
Ok(Some(u)) => Some(u),
|
||||||
Ok(None) if cfg.web.auto_create_users => {
|
Ok(None) if cfg.web.auto_create_users => {
|
||||||
tracing::info!(user = %name, "creating an account for a name the proxy vouched for");
|
tracing::info!(user = %name, "creating an account for a name the proxy vouched for");
|
||||||
state
|
// The first account made is the admin.
|
||||||
.ctx
|
let first = state.ctx.db.users().await.map(|u| u.is_empty()).unwrap_or(false);
|
||||||
.db
|
match state.ctx.db.create_user(&name, None, first).await {
|
||||||
.create_user(&name, None, state.ctx.db.users().map(|u| u.is_empty()).unwrap_or(false))
|
Ok(id) => state.ctx.db.user_by_id(id).await.ok().flatten(),
|
||||||
.ok()
|
Err(_) => None,
|
||||||
.and_then(|id| state.ctx.db.user_by_id(id).ok().flatten())
|
}
|
||||||
}
|
}
|
||||||
Ok(None) => {
|
Ok(None) => {
|
||||||
tracing::warn!(user = %name, "proxy vouched for an unknown name and auto_create_users is off");
|
tracing::warn!(user = %name, "proxy vouched for an unknown name and auto_create_users is off");
|
||||||
@@ -130,7 +130,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
// Every request comes vouched for; signed_in keeps one an hour. Failing to note the time
|
// Every request comes vouched for; signed_in keeps one an hour. Failing to note the time
|
||||||
// must not turn anyone away, so its error goes unanswered.
|
// must not turn anyone away, so its error goes unanswered.
|
||||||
if let Some(u) = &user {
|
if let Some(u) = &user {
|
||||||
let _ = state.ctx.db.signed_in(u.id);
|
let _ = state.ctx.db.signed_in(u.id).await;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let by_proxy = user.is_some();
|
let by_proxy = user.is_some();
|
||||||
@@ -141,7 +141,7 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
user = state
|
user = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.session_user(&sid, cfg.web.session_days.max(1) * 86_400)
|
.session_user(&sid, cfg.web.session_days.max(1) * 86_400).await
|
||||||
.unwrap_or(None);
|
.unwrap_or(None);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -155,11 +155,11 @@ async fn auth(State(state): State<WebState>, mut req: Request, next: Next) -> Re
|
|||||||
if user.is_none() && !token.is_empty() {
|
if user.is_none() && !token.is_empty() {
|
||||||
let supplied = from_query.clone().or_else(|| cookie(&req, COOKIE));
|
let supplied = from_query.clone().or_else(|| cookie(&req, COOKIE));
|
||||||
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
|
if supplied.is_some_and(|t| constant_time_eq(&t, &token)) {
|
||||||
user = admin_user(&state);
|
user = admin_user(&state).await;
|
||||||
if from_query.is_some() {
|
if from_query.is_some() {
|
||||||
// The token link is a sign-in; the cookie it leaves behind is not one each time.
|
// The token link is a sign-in; the cookie it leaves behind is not one each time.
|
||||||
if let Some(u) = &user {
|
if let Some(u) = &user {
|
||||||
let _ = state.ctx.db.signed_in(u.id);
|
let _ = state.ctx.db.signed_in(u.id).await;
|
||||||
}
|
}
|
||||||
set_cookie = Some(format!(
|
set_cookie = Some(format!(
|
||||||
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
|
"{COOKIE}={token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000"
|
||||||
@@ -247,8 +247,8 @@ fn cookie(req: &Request, name: &str) -> Option<String> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// The account the shared token stands for: the first admin, or the first user at all.
|
/// The account the shared token stands for: the first admin, or the first user at all.
|
||||||
fn admin_user(state: &WebState) -> Option<crate::db::User> {
|
async fn admin_user(state: &WebState) -> Option<crate::db::User> {
|
||||||
let users = state.ctx.db.users().ok()?;
|
let users = state.ctx.db.users().await.ok()?;
|
||||||
users
|
users
|
||||||
.iter()
|
.iter()
|
||||||
.find(|u| u.is_admin)
|
.find(|u| u.is_admin)
|
||||||
@@ -267,7 +267,7 @@ async fn login(
|
|||||||
Json(body): Json<Credentials>,
|
Json(body): Json<Credentials>,
|
||||||
) -> Result<Response, ApiError> {
|
) -> Result<Response, ApiError> {
|
||||||
let name = body.name.trim().to_ascii_lowercase();
|
let name = body.name.trim().to_ascii_lowercase();
|
||||||
let user = state.ctx.db.user_by_name(&name)?;
|
let user = state.ctx.db.user_by_name(&name).await?;
|
||||||
// The same answer either way: whether a name exists is not something to leak.
|
// The same answer either way: whether a name exists is not something to leak.
|
||||||
let ok = user
|
let ok = user
|
||||||
.as_ref()
|
.as_ref()
|
||||||
@@ -280,8 +280,8 @@ async fn login(
|
|||||||
|
|
||||||
let user = user.expect("verified above");
|
let user = user.expect("verified above");
|
||||||
let token = crate::auth::new_session_token();
|
let token = crate::auth::new_session_token();
|
||||||
state.ctx.db.create_session(user.id, &token)?;
|
state.ctx.db.create_session(user.id, &token).await?;
|
||||||
state.ctx.db.signed_in(user.id)?;
|
state.ctx.db.signed_in(user.id).await?;
|
||||||
tracing::info!(user = %user.name, "signed in");
|
tracing::info!(user = %user.name, "signed in");
|
||||||
|
|
||||||
let days = state.ctx.cfg().web.session_days.max(1);
|
let days = state.ctx.cfg().web.session_days.max(1);
|
||||||
@@ -298,7 +298,7 @@ async fn login(
|
|||||||
|
|
||||||
async fn logout(State(state): State<WebState>, req: Request) -> Response {
|
async fn logout(State(state): State<WebState>, req: Request) -> Response {
|
||||||
if let Some(sid) = cookie(&req, SESSION_COOKIE) {
|
if let Some(sid) = cookie(&req, SESSION_COOKIE) {
|
||||||
let _ = state.ctx.db.delete_session(&sid);
|
let _ = state.ctx.db.delete_session(&sid).await;
|
||||||
}
|
}
|
||||||
let mut resp = StatusCode::NO_CONTENT.into_response();
|
let mut resp = StatusCode::NO_CONTENT.into_response();
|
||||||
for c in [
|
for c in [
|
||||||
@@ -320,7 +320,7 @@ async fn me(
|
|||||||
) -> Json<serde_json::Value> {
|
) -> Json<serde_json::Value> {
|
||||||
let url = state.ctx.cfg().web.sign_out_url.clone();
|
let url = state.ctx.cfg().web.sign_out_url.clone();
|
||||||
let sign_out = (by_proxy && !url.is_empty()).then_some(url);
|
let sign_out = (by_proxy && !url.is_empty()).then_some(url);
|
||||||
let (theme, mode) = state.ctx.db.theme(user.id).unwrap_or_default();
|
let (theme, mode) = state.ctx.db.theme(user.id).await.unwrap_or_default();
|
||||||
Json(serde_json::json!({
|
Json(serde_json::json!({
|
||||||
"name": user.name, "admin": user.is_admin, "sign_out": sign_out, "theme": theme, "mode": mode,
|
"name": user.name, "admin": user.is_admin, "sign_out": sign_out, "theme": theme, "mode": mode,
|
||||||
}))
|
}))
|
||||||
@@ -343,7 +343,7 @@ async fn patch_me(
|
|||||||
if !theme_ok(&body.theme, &body.mode) {
|
if !theme_ok(&body.theme, &body.mode) {
|
||||||
return Err(ApiError::bad_request("not a theme"));
|
return Err(ApiError::bad_request("not a theme"));
|
||||||
}
|
}
|
||||||
state.ctx.db.set_theme(user.id, &body.theme, &body.mode)?;
|
state.ctx.db.set_theme(user.id, &body.theme, &body.mode).await?;
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -378,7 +378,7 @@ async fn list_users(
|
|||||||
let users: Vec<_> = state
|
let users: Vec<_> = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.users()?
|
.users().await?
|
||||||
.iter()
|
.iter()
|
||||||
.map(|u| {
|
.map(|u| {
|
||||||
serde_json::json!({
|
serde_json::json!({
|
||||||
@@ -409,7 +409,7 @@ async fn add_user(
|
|||||||
let name = crate::auth::name_from_header(&body.name).ok_or_else(|| {
|
let name = crate::auth::name_from_header(&body.name).ok_or_else(|| {
|
||||||
ApiError::bad_request("a name is required, without commas, semicolons or line breaks")
|
ApiError::bad_request("a name is required, without commas, semicolons or line breaks")
|
||||||
})?;
|
})?;
|
||||||
if state.ctx.db.user_by_name(&name)?.is_some() {
|
if state.ctx.db.user_by_name(&name).await?.is_some() {
|
||||||
return Err(ApiError::bad_request(format!("{name} already exists")));
|
return Err(ApiError::bad_request(format!("{name} already exists")));
|
||||||
}
|
}
|
||||||
// No password is someone the proxy signs in, as with `ipx user add --no-password`.
|
// No password is someone the proxy signs in, as with `ipx user add --no-password`.
|
||||||
@@ -418,7 +418,7 @@ async fn add_user(
|
|||||||
} else {
|
} else {
|
||||||
Some(crate::auth::hash_password(&body.password).map_err(|e| ApiError::bad_request(format!("{e:#}")))?)
|
Some(crate::auth::hash_password(&body.password).map_err(|e| ApiError::bad_request(format!("{e:#}")))?)
|
||||||
};
|
};
|
||||||
state.ctx.db.create_user(&name, hash.as_deref(), body.admin)?;
|
state.ctx.db.create_user(&name, hash.as_deref(), body.admin).await?;
|
||||||
tracing::info!(by = %user.name, user = %name, admin = body.admin, "account added");
|
tracing::info!(by = %user.name, user = %name, admin = body.admin, "account added");
|
||||||
Ok(StatusCode::CREATED)
|
Ok(StatusCode::CREATED)
|
||||||
}
|
}
|
||||||
@@ -435,7 +435,7 @@ async fn patch_user(
|
|||||||
Json(body): Json<UserPatch>,
|
Json(body): Json<UserPatch>,
|
||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
require_admin(&user)?;
|
require_admin(&user)?;
|
||||||
let users = state.ctx.db.users()?;
|
let users = state.ctx.db.users().await?;
|
||||||
let target = users
|
let target = users
|
||||||
.iter()
|
.iter()
|
||||||
.find(|u| u.id == id)
|
.find(|u| u.id == id)
|
||||||
@@ -446,7 +446,7 @@ async fn patch_user(
|
|||||||
target.name
|
target.name
|
||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
state.ctx.db.set_admin(id, body.admin)?;
|
state.ctx.db.set_admin(id, body.admin).await?;
|
||||||
tracing::info!(by = %user.name, user = %target.name, admin = body.admin, "admin changed");
|
tracing::info!(by = %user.name, user = %target.name, admin = body.admin, "admin changed");
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
@@ -457,7 +457,7 @@ async fn remove_user(
|
|||||||
Path(id): Path<i64>,
|
Path(id): Path<i64>,
|
||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
require_admin(&user)?;
|
require_admin(&user)?;
|
||||||
let users = state.ctx.db.users()?;
|
let users = state.ctx.db.users().await?;
|
||||||
let target = users
|
let target = users
|
||||||
.iter()
|
.iter()
|
||||||
.find(|u| u.id == id)
|
.find(|u| u.id == id)
|
||||||
@@ -468,7 +468,7 @@ async fn remove_user(
|
|||||||
target.name
|
target.name
|
||||||
)));
|
)));
|
||||||
}
|
}
|
||||||
state.ctx.db.delete_user(id)?;
|
state.ctx.db.delete_user(id).await?;
|
||||||
tracing::info!(by = %user.name, user = %target.name, "account removed");
|
tracing::info!(by = %user.name, user = %target.name, "account removed");
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
@@ -510,7 +510,7 @@ async fn admin_page(State(state): State<WebState>, user: crate::db::User) -> Res
|
|||||||
if !user.is_admin {
|
if !user.is_admin {
|
||||||
return Redirect::to("/").into_response();
|
return Redirect::to("/").into_response();
|
||||||
}
|
}
|
||||||
let theme = state.ctx.db.theme(user.id).unwrap_or_default();
|
let theme = state.ctx.db.theme(user.id).await.unwrap_or_default();
|
||||||
let page = with_theme(include_str!(concat!(env!("OUT_DIR"), "/admin.html")), theme);
|
let page = with_theme(include_str!(concat!(env!("OUT_DIR"), "/admin.html")), theme);
|
||||||
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page)).into_response()
|
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page)).into_response()
|
||||||
}
|
}
|
||||||
@@ -584,7 +584,7 @@ const ADMIN_LINK: &str = "<a id=admin ";
|
|||||||
/// The page, with the log button left out for anyone but an admin. Hiding it from the page's
|
/// The page, with the log button left out for anyone but an admin. Hiding it from the page's
|
||||||
/// script instead showed it for a moment on every load, until /api/me answered.
|
/// script instead showed it for a moment on every load, until /api/me answered.
|
||||||
async fn index(State(state): State<WebState>, user: crate::db::User) -> impl IntoResponse {
|
async fn index(State(state): State<WebState>, user: crate::db::User) -> impl IntoResponse {
|
||||||
let theme = state.ctx.db.theme(user.id).unwrap_or_default();
|
let theme = state.ctx.db.theme(user.id).await.unwrap_or_default();
|
||||||
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page_for(user.is_admin, theme)))
|
([(header::CACHE_CONTROL, PAGE_CACHE)], Html(page_for(user.is_admin, theme)))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -672,16 +672,16 @@ async fn feeds(
|
|||||||
let cfg = state.ctx.cfg();
|
let cfg = state.ctx.cfg();
|
||||||
// Config entries plus the feeds derived from OPML subscriptions -- the catalogue.
|
// Config entries plus the feeds derived from OPML subscriptions -- the catalogue.
|
||||||
// What comes back is only the part of it this person subscribes to.
|
// What comes back is only the part of it this person subscribes to.
|
||||||
let subs = crate::subscriptions(&state.ctx)?;
|
let subs = crate::subscriptions(&state.ctx).await?;
|
||||||
let mine: std::collections::HashMap<String, crate::db::Sub> = state
|
let mine: std::collections::HashMap<String, crate::db::Sub> = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.subscriptions_for(user.id)?
|
.subscriptions_for(user.id).await?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|s| (s.feed_id.clone(), s))
|
.map(|s| (s.feed_id.clone(), s))
|
||||||
.collect();
|
.collect();
|
||||||
let counts = state.ctx.db.subscriber_counts()?;
|
let counts = state.ctx.db.subscriber_counts().await?;
|
||||||
let pinned = state.ctx.db.pinned_feeds(user.id)?;
|
let pinned = state.ctx.db.pinned_feeds(user.id).await?;
|
||||||
let mut out = Vec::with_capacity(mine.len());
|
let mut out = Vec::with_capacity(mine.len());
|
||||||
for sub in &subs {
|
for sub in &subs {
|
||||||
let (id, feed) = (&sub.id, &sub.cfg);
|
let (id, feed) = (&sub.id, &sub.cfg);
|
||||||
@@ -689,8 +689,8 @@ async fn feeds(
|
|||||||
// the same fallback the scanner uses (`Db::subscribers`).
|
// the same fallback the scanner uses (`Db::subscribers`).
|
||||||
let up = feed.group.as_deref().and_then(|g| mine.get(g));
|
let up = feed.group.as_deref().and_then(|g| mine.get(g));
|
||||||
let Some(mine) = mine.get(id) else { continue };
|
let Some(mine) = mine.get(id) else { continue };
|
||||||
let s = state.ctx.db.feed_summary(id)?;
|
let s = state.ctx.db.feed_summary(id).await?;
|
||||||
let st = state.ctx.db.http_state(id)?;
|
let st = state.ctx.db.http_state(id).await?;
|
||||||
out.push(FeedRow {
|
out.push(FeedRow {
|
||||||
id: id.clone(),
|
id: id.clone(),
|
||||||
url: feed.url.clone(),
|
url: feed.url.clone(),
|
||||||
@@ -740,7 +740,7 @@ async fn feeds(
|
|||||||
last_error: s.last_error,
|
last_error: s.last_error,
|
||||||
entries: s.entries,
|
entries: s.entries,
|
||||||
downloaded: s.downloaded,
|
downloaded: s.downloaded,
|
||||||
unread: state.ctx.db.unread_count(user.id, id)?,
|
unread: state.ctx.db.unread_count(user.id, id).await?,
|
||||||
subscribers: counts.get(id).copied().unwrap_or(0),
|
subscribers: counts.get(id).copied().unwrap_or(0),
|
||||||
pinned: pinned.contains(id),
|
pinned: pinned.contains(id),
|
||||||
});
|
});
|
||||||
@@ -800,13 +800,13 @@ struct PopularRow {
|
|||||||
/// first. Popular is the top of it, the directory is all of it, and it is all that
|
/// first. Popular is the top of it, the directory is all of it, and it is all that
|
||||||
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
|
/// `subscribe_popular` will subscribe you to. An OPML or a Patreon creator is listed as the
|
||||||
/// feeds inside it and never itself: both lists are for finding a show.
|
/// feeds inside it and never itself: both lists are for finding a show.
|
||||||
fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
|
async fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
|
||||||
let db = &state.ctx.db;
|
let db = &state.ctx.db;
|
||||||
let mine: std::collections::HashSet<String> =
|
let mine: std::collections::HashSet<String> =
|
||||||
db.subscriptions_for(user_id)?.into_iter().map(|s| s.feed_id).collect();
|
db.subscriptions_for(user_id).await?.into_iter().map(|s| s.feed_id).collect();
|
||||||
let counts = db.subscriber_counts()?;
|
let counts = db.subscriber_counts().await?;
|
||||||
let media = db.media_feeds()?;
|
let media = db.media_feeds().await?;
|
||||||
let catalogue = crate::subscriptions(&state.ctx)?;
|
let catalogue = crate::subscriptions(&state.ctx).await?;
|
||||||
let by_id: std::collections::HashMap<&str, &crate::config::Feed> =
|
let by_id: std::collections::HashMap<&str, &crate::config::Feed> =
|
||||||
catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect();
|
catalogue.iter().map(|s| (s.id.as_str(), &s.cfg)).collect();
|
||||||
let is_folder: std::collections::HashSet<&str> =
|
let is_folder: std::collections::HashSet<&str> =
|
||||||
@@ -823,7 +823,7 @@ fn popular(state: &WebState, user_id: i64) -> Result<Vec<PopularRow>> {
|
|||||||
{
|
{
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let sum = db.feed_summary(&s.id)?;
|
let sum = db.feed_summary(&s.id).await?;
|
||||||
let subscribed = mine.contains(&s.id);
|
let subscribed = mine.contains(&s.id);
|
||||||
out.push(PopularRow {
|
out.push(PopularRow {
|
||||||
id: s.id.clone(),
|
id: s.id.clone(),
|
||||||
@@ -844,7 +844,7 @@ async fn get_popular(
|
|||||||
State(state): State<WebState>,
|
State(state): State<WebState>,
|
||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
) -> Result<Json<Vec<PopularRow>>, ApiError> {
|
) -> Result<Json<Vec<PopularRow>>, ApiError> {
|
||||||
let mut rows = popular(&state, user.id)?;
|
let mut rows = popular(&state, user.id).await?;
|
||||||
rows.truncate(10);
|
rows.truncate(10);
|
||||||
Ok(Json(rows))
|
Ok(Json(rows))
|
||||||
}
|
}
|
||||||
@@ -854,7 +854,7 @@ async fn get_directory(
|
|||||||
State(state): State<WebState>,
|
State(state): State<WebState>,
|
||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
) -> Result<Json<Vec<PopularRow>>, ApiError> {
|
) -> Result<Json<Vec<PopularRow>>, ApiError> {
|
||||||
let mut rows = popular(&state, user.id)?;
|
let mut rows = popular(&state, user.id).await?;
|
||||||
rows.sort_by_key(sort_name);
|
rows.sort_by_key(sort_name);
|
||||||
Ok(Json(rows))
|
Ok(Json(rows))
|
||||||
}
|
}
|
||||||
@@ -870,10 +870,10 @@ async fn subscribe_popular(
|
|||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
Path(id): Path<String>,
|
Path(id): Path<String>,
|
||||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||||
if !popular(&state, user.id)?.iter().any(|p| p.id == id) {
|
if !popular(&state, user.id).await?.iter().any(|p| p.id == id) {
|
||||||
return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
|
return Err(ApiError::bad_request(format!("{id:?} is not in the directory")));
|
||||||
}
|
}
|
||||||
state.ctx.db.subscribe(user.id, &id)?;
|
state.ctx.db.subscribe(user.id, &id).await?;
|
||||||
Ok(Json(serde_json::json!({ "id": id })))
|
Ok(Json(serde_json::json!({ "id": id })))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1124,7 +1124,7 @@ async fn entries(
|
|||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
Query(page): Query<Page>,
|
Query(page): Query<Page>,
|
||||||
) -> Result<Json<EntryPage>, ApiError> {
|
) -> Result<Json<EntryPage>, ApiError> {
|
||||||
entry_page(&state, user.id, Some(&id), &page)
|
entry_page(&state, user.id, Some(&id), &page).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Every subscribed feed's items together, newest first: All Subscriptions.
|
/// Every subscribed feed's items together, newest first: All Subscriptions.
|
||||||
@@ -1133,11 +1133,11 @@ async fn all_entries(
|
|||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
Query(page): Query<Page>,
|
Query(page): Query<Page>,
|
||||||
) -> Result<Json<EntryPage>, ApiError> {
|
) -> Result<Json<EntryPage>, ApiError> {
|
||||||
entry_page(&state, user.id, None, &page)
|
entry_page(&state, user.id, None, &page).await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One feed's page of items, or every subscribed feed's when `feed` is None.
|
/// One feed's page of items, or every subscribed feed's when `feed` is None.
|
||||||
fn entry_page(
|
async fn entry_page(
|
||||||
state: &WebState,
|
state: &WebState,
|
||||||
user_id: i64,
|
user_id: i64,
|
||||||
feed: Option<&str>,
|
feed: Option<&str>,
|
||||||
@@ -1153,14 +1153,14 @@ fn entry_page(
|
|||||||
filter != crate::db::Filter::InProgress,
|
filter != crate::db::Filter::InProgress,
|
||||||
);
|
);
|
||||||
let mut rows =
|
let mut rows =
|
||||||
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order)?;
|
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order).await?;
|
||||||
let mut sanitizer = feed_sanitizer();
|
let mut sanitizer = feed_sanitizer();
|
||||||
for row in &mut rows {
|
for row in &mut rows {
|
||||||
if let Some(d) = &row.description {
|
if let Some(d) = &row.description {
|
||||||
row.description = Some(clean_description(&mut sanitizer, d, row.link.as_deref()));
|
row.description = Some(clean_description(&mut sanitizer, d, row.link.as_deref()));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let total = db.count_in(user_id, feed, filter, search)?;
|
let total = db.count_in(user_id, feed, filter, search).await?;
|
||||||
Ok(Json(EntryPage { total, entries: rows }))
|
Ok(Json(EntryPage { total, entries: rows }))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1202,10 +1202,10 @@ struct NewFeed {
|
|||||||
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
|
/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it
|
||||||
/// goes on your subscription, and before the first scan, which would otherwise skip every
|
/// goes on your subscription, and before the first scan, which would otherwise skip every
|
||||||
/// explicit item.
|
/// explicit item.
|
||||||
fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
|
async fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> {
|
||||||
if allow {
|
if allow {
|
||||||
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
|
let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() };
|
||||||
state.ctx.db.set_subscription(user_id, &sub)?;
|
state.ctx.db.set_subscription(user_id, &sub).await?;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
@@ -1219,14 +1219,14 @@ async fn add_feed(
|
|||||||
let url = crate::feed::expand_input(&body.url);
|
let url = crate::feed::expand_input(&body.url);
|
||||||
// Someone else may already have it. Then adding costs nothing: no second fetch, no
|
// Someone else may already have it. Then adding costs nothing: no second fetch, no
|
||||||
// second copy on disk, just another name against the same feed.
|
// second copy on disk, just another name against the same feed.
|
||||||
if let Some(existing) = crate::subscriptions(&state.ctx)?
|
if let Some(existing) = crate::subscriptions(&state.ctx).await?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.find(|s| crate::feed::same_feed(&s.cfg.url, &url))
|
.find(|s| crate::feed::same_feed(&s.cfg.url, &url))
|
||||||
{
|
{
|
||||||
let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some();
|
let already = state.ctx.db.subscription(user.id, &existing.id).await?.is_some();
|
||||||
state.ctx.db.subscribe(user.id, &existing.id)?;
|
state.ctx.db.subscribe(user.id, &existing.id).await?;
|
||||||
if !already {
|
if !already {
|
||||||
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?;
|
explicit_on_add(&state, user.id, &existing.id, body.allow_explicit).await?;
|
||||||
}
|
}
|
||||||
scan_soon(&state, Some(existing.id.clone())).await;
|
scan_soon(&state, Some(existing.id.clone())).await;
|
||||||
return Ok(Json(
|
return Ok(Json(
|
||||||
@@ -1236,8 +1236,8 @@ async fn add_feed(
|
|||||||
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
|
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
|
||||||
cfg.save(&state.config_path)?;
|
cfg.save(&state.config_path)?;
|
||||||
state.ctx.reload_cfg(&state.config_path)?;
|
state.ctx.reload_cfg(&state.config_path)?;
|
||||||
state.ctx.db.subscribe(user.id, &id)?;
|
state.ctx.db.subscribe(user.id, &id).await?;
|
||||||
explicit_on_add(&state, user.id, &id, body.allow_explicit)?;
|
explicit_on_add(&state, user.id, &id, body.allow_explicit).await?;
|
||||||
scan_soon(&state, Some(id.clone())).await;
|
scan_soon(&state, Some(id.clone())).await;
|
||||||
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
|
Ok(Json(serde_json::json!({ "id": id, "existing": false })))
|
||||||
}
|
}
|
||||||
@@ -1290,17 +1290,17 @@ async fn patch_feed(
|
|||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
// Pinning is yours alone too, and means nothing for a feed you do not subscribe to.
|
// Pinning is yours alone too, and means nothing for a feed you do not subscribe to.
|
||||||
if let Some(on) = body.pinned
|
if let Some(on) = body.pinned
|
||||||
&& !state.ctx.db.set_pinned(user.id, &id, on)?
|
&& !state.ctx.db.set_pinned(user.id, &id, on).await?
|
||||||
{
|
{
|
||||||
return Err(ApiError::not_found("you do not subscribe to that feed"));
|
return Err(ApiError::not_found("you do not subscribe to that feed"));
|
||||||
}
|
}
|
||||||
// What one person wants -- which items, whether to fetch them, how many at a time --
|
// What one person wants -- which items, whether to fetch them, how many at a time --
|
||||||
// is theirs. It goes on their subscription and nobody else sees the change.
|
// is theirs. It goes on their subscription and nobody else sees the change.
|
||||||
if state.ctx.db.subscription(user.id, &id)?.is_some() {
|
if state.ctx.db.subscription(user.id, &id).await?.is_some() {
|
||||||
let mut mine = state
|
let mut mine = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.subscription(user.id, &id)?
|
.subscription(user.id, &id).await?
|
||||||
.unwrap_or_else(|| crate::db::Sub { feed_id: id.clone(), ..Default::default() });
|
.unwrap_or_else(|| crate::db::Sub { feed_id: id.clone(), ..Default::default() });
|
||||||
let mut touched = false;
|
let mut touched = false;
|
||||||
if let Some(v) = body.keywords.clone() {
|
if let Some(v) = body.keywords.clone() {
|
||||||
@@ -1320,7 +1320,7 @@ async fn patch_feed(
|
|||||||
touched = true;
|
touched = true;
|
||||||
}
|
}
|
||||||
if touched {
|
if touched {
|
||||||
state.ctx.db.set_subscription(user.id, &mine)?;
|
state.ctx.db.set_subscription(user.id, &mine).await?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1341,13 +1341,13 @@ async fn patch_feed(
|
|||||||
// Derived feeds have no config entry. Editing one is the moment it earns a real
|
// Derived feeds have no config entry. Editing one is the moment it earns a real
|
||||||
// entry: promote it, so the config holds your decisions and nothing else.
|
// entry: promote it, so the config holds your decisions and nothing else.
|
||||||
if !cfg.feeds.contains_key(&id) {
|
if !cfg.feeds.contains_key(&id) {
|
||||||
let subs = crate::subscriptions(&state.ctx)?;
|
let subs = crate::subscriptions(&state.ctx).await?;
|
||||||
let found = subs
|
let found = subs
|
||||||
.iter()
|
.iter()
|
||||||
.find(|s| s.id == id)
|
.find(|s| s.id == id)
|
||||||
.ok_or_else(|| ApiError::bad_request(format!("no feed with id {id:?}")))?;
|
.ok_or_else(|| ApiError::bad_request(format!("no feed with id {id:?}")))?;
|
||||||
cfg.feeds.insert(id.clone(), found.cfg.clone());
|
cfg.feeds.insert(id.clone(), found.cfg.clone());
|
||||||
state.ctx.db.unmanage(&id)?;
|
state.ctx.db.unmanage(&id).await?;
|
||||||
}
|
}
|
||||||
|
|
||||||
let checked = match &body.url {
|
let checked = match &body.url {
|
||||||
@@ -1388,7 +1388,7 @@ async fn patch_feed(
|
|||||||
if url_changed {
|
if url_changed {
|
||||||
// Refreshing a rotated auth token is the common case; entries and download history
|
// Refreshing a rotated auth token is the common case; entries and download history
|
||||||
// are keyed by feed id, so they survive the change.
|
// are keyed by feed id, so they survive the change.
|
||||||
state.ctx.db.clear_validators(&id)?;
|
state.ctx.db.clear_validators(&id).await?;
|
||||||
}
|
}
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
@@ -1400,14 +1400,14 @@ async fn remove_feed(
|
|||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
// Unsubscribing is personal: it takes the feed off your list and leaves everyone
|
// Unsubscribing is personal: it takes the feed off your list and leaves everyone
|
||||||
// else's alone.
|
// else's alone.
|
||||||
state.ctx.db.unsubscribe(user.id, &id)?;
|
state.ctx.db.unsubscribe(user.id, &id).await?;
|
||||||
for child in crate::subscriptions(&state.ctx)?
|
for child in crate::subscriptions(&state.ctx).await?
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
|
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
|
||||||
{
|
{
|
||||||
state.ctx.db.unsubscribe(user.id, &child.id)?;
|
state.ctx.db.unsubscribe(user.id, &child.id).await?;
|
||||||
}
|
}
|
||||||
if state.ctx.db.subscriber_counts()?.contains_key(&id) {
|
if state.ctx.db.subscriber_counts().await?.contains_key(&id) {
|
||||||
return Ok(StatusCode::NO_CONTENT);
|
return Ok(StatusCode::NO_CONTENT);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1417,12 +1417,12 @@ async fn remove_feed(
|
|||||||
if cfg.feeds.remove(&id).is_none() {
|
if cfg.feeds.remove(&id).is_none() {
|
||||||
// A derived feed: forget it here, though the OPML will list it again on the next
|
// A derived feed: forget it here, though the OPML will list it again on the next
|
||||||
// read unless you unsubscribe from the OPML itself.
|
// read unless you unsubscribe from the OPML itself.
|
||||||
state.ctx.db.drop_managed(&id)?;
|
state.ctx.db.drop_managed(&id).await?;
|
||||||
return Ok(StatusCode::NO_CONTENT);
|
return Ok(StatusCode::NO_CONTENT);
|
||||||
}
|
}
|
||||||
cfg.save(&state.config_path)?;
|
cfg.save(&state.config_path)?;
|
||||||
state.ctx.reload_cfg(&state.config_path)?;
|
state.ctx.reload_cfg(&state.config_path)?;
|
||||||
crate::retire_group(&state.ctx, &id)?;
|
crate::retire_group(&state.ctx, &id).await?;
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1440,10 +1440,10 @@ async fn set_flags(
|
|||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
use crate::db::EntryFlag;
|
use crate::db::EntryFlag;
|
||||||
if let Some(v) = body.read {
|
if let Some(v) = body.read {
|
||||||
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Read, v)?;
|
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Read, v).await?;
|
||||||
}
|
}
|
||||||
if let Some(v) = body.flagged {
|
if let Some(v) = body.flagged {
|
||||||
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Flagged, v)?;
|
state.ctx.db.set_entry_flag(user.id, &feed_id, &guid, EntryFlag::Flagged, v).await?;
|
||||||
}
|
}
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
@@ -1458,12 +1458,12 @@ async fn download_now(
|
|||||||
let enc = state
|
let enc = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.enclosure(id)?
|
.enclosure(id).await?
|
||||||
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
|
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
|
||||||
if enc.path.is_some() {
|
if enc.path.is_some() {
|
||||||
return Ok(StatusCode::NO_CONTENT); // Already here.
|
return Ok(StatusCode::NO_CONTENT); // Already here.
|
||||||
}
|
}
|
||||||
state.ctx.db.requeue(id)?;
|
state.ctx.db.requeue(id).await?;
|
||||||
state
|
state
|
||||||
.cmds
|
.cmds
|
||||||
.send(Command::Download { enclosure: id })
|
.send(Command::Download { enclosure: id })
|
||||||
@@ -1487,13 +1487,13 @@ async fn delete_file(
|
|||||||
let enc = state
|
let enc = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.enclosure(id)?
|
.enclosure(id).await?
|
||||||
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
|
.ok_or_else(|| anyhow::anyhow!("no enclosure {id}"))?;
|
||||||
|
|
||||||
// There is one copy of the file: deleting it deletes everyone's. Say so before doing
|
// There is one copy of the file: deleting it deletes everyone's. Say so before doing
|
||||||
// it, once, and let them decide.
|
// it, once, and let them decide.
|
||||||
if !q.force {
|
if !q.force {
|
||||||
let (starred, unread) = state.ctx.db.others_wanting(id, user.id)?;
|
let (starred, unread) = state.ctx.db.others_wanting(id, user.id).await?;
|
||||||
let people = |n: i64| if n == 1 { "person".to_string() } else { format!("{n} people") };
|
let people = |n: i64| if n == 1 { "person".to_string() } else { format!("{n} people") };
|
||||||
let complaint = match (starred, unread) {
|
let complaint = match (starred, unread) {
|
||||||
(0, 0) => None,
|
(0, 0) => None,
|
||||||
@@ -1519,7 +1519,7 @@ async fn delete_file(
|
|||||||
return Err(e.into());
|
return Err(e.into());
|
||||||
}
|
}
|
||||||
// The row survives as 'reaped', which is what stops the next scan re-downloading it.
|
// The row survives as 'reaped', which is what stops the next scan re-downloading it.
|
||||||
state.ctx.db.mark_reaped(id)?;
|
state.ctx.db.mark_reaped(id).await?;
|
||||||
state.events.send(Event::Reaped {
|
state.events.send(Event::Reaped {
|
||||||
path: enc.path.unwrap_or_default(),
|
path: enc.path.unwrap_or_default(),
|
||||||
bytes: enc.length.unwrap_or(0).max(0) as u64,
|
bytes: enc.length.unwrap_or(0).max(0) as u64,
|
||||||
@@ -1573,7 +1573,7 @@ async fn media(
|
|||||||
Path(id): Path<i64>,
|
Path(id): Path<i64>,
|
||||||
req: Request,
|
req: Request,
|
||||||
) -> Response {
|
) -> Response {
|
||||||
let Ok(Some(enc)) = state.ctx.db.enclosure(id) else {
|
let Ok(Some(enc)) = state.ctx.db.enclosure(id).await else {
|
||||||
return (StatusCode::NOT_FOUND, "no such enclosure").into_response();
|
return (StatusCode::NOT_FOUND, "no such enclosure").into_response();
|
||||||
};
|
};
|
||||||
let Some(path) = enc.path else {
|
let Some(path) = enc.path else {
|
||||||
@@ -1598,7 +1598,7 @@ async fn set_position(
|
|||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
Json(body): Json<Position>,
|
Json(body): Json<Position>,
|
||||||
) -> Result<StatusCode, ApiError> {
|
) -> Result<StatusCode, ApiError> {
|
||||||
state.ctx.db.set_position(user.id, &feed_id, &guid, body.secs, body.duration)?;
|
state.ctx.db.set_position(user.id, &feed_id, &guid, body.secs, body.duration).await?;
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1610,12 +1610,12 @@ async fn read_all(
|
|||||||
// A subscription's own row has no entries, so marking it read means everything under it.
|
// A subscription's own row has no entries, so marking it read means everything under it.
|
||||||
let mut ids = vec![id.clone()];
|
let mut ids = vec![id.clone()];
|
||||||
ids.extend(
|
ids.extend(
|
||||||
crate::subscriptions(&state.ctx)?
|
crate::subscriptions(&state.ctx).await?
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
|
.filter(|s| s.cfg.group.as_deref() == Some(id.as_str()))
|
||||||
.map(|s| s.id),
|
.map(|s| s.id),
|
||||||
);
|
);
|
||||||
let n = state.ctx.db.mark_all_read(user.id, &ids)?;
|
let n = state.ctx.db.mark_all_read(user.id, &ids).await?;
|
||||||
Ok(Json(serde_json::json!({ "marked": n })))
|
Ok(Json(serde_json::json!({ "marked": n })))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1626,8 +1626,8 @@ async fn read_all_mine(
|
|||||||
user: crate::db::User,
|
user: crate::db::User,
|
||||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||||
let ids: Vec<String> =
|
let ids: Vec<String> =
|
||||||
state.ctx.db.subscriptions_for(user.id)?.into_iter().map(|s| s.feed_id).collect();
|
state.ctx.db.subscriptions_for(user.id).await?.into_iter().map(|s| s.feed_id).collect();
|
||||||
let n = state.ctx.db.mark_all_read(user.id, &ids)?;
|
let n = state.ctx.db.mark_all_read(user.id, &ids).await?;
|
||||||
Ok(Json(serde_json::json!({ "marked": n })))
|
Ok(Json(serde_json::json!({ "marked": n })))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1648,9 +1648,9 @@ async fn download_latest(
|
|||||||
Path(id): Path<String>,
|
Path(id): Path<String>,
|
||||||
Json(body): Json<HowMany>,
|
Json(body): Json<HowMany>,
|
||||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||||
let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100))?;
|
let ids = state.ctx.db.undownloaded(&id, body.count.clamp(1, 100)).await?;
|
||||||
for enc in &ids {
|
for enc in &ids {
|
||||||
state.ctx.db.requeue(*enc)?;
|
state.ctx.db.requeue(*enc).await?;
|
||||||
state
|
state
|
||||||
.cmds
|
.cmds
|
||||||
.send(Command::Download { enclosure: *enc })
|
.send(Command::Download { enclosure: *enc })
|
||||||
@@ -1668,7 +1668,7 @@ async fn export_opml(
|
|||||||
// Yours, not the whole catalogue: other people's feeds, and any private URLs in them, are
|
// Yours, not the whole catalogue: other people's feeds, and any private URLs in them, are
|
||||||
// not yours to download. This used to export config.toml to whoever asked.
|
// not yours to download. This used to export config.toml to whoever asked.
|
||||||
let mine: std::collections::HashSet<String> =
|
let mine: std::collections::HashSet<String> =
|
||||||
state.ctx.db.subscriptions_for(user.id)?.into_iter().map(|s| s.feed_id).collect();
|
state.ctx.db.subscriptions_for(user.id).await?.into_iter().map(|s| s.feed_id).collect();
|
||||||
let mut doc = opml::OPML {
|
let mut doc = opml::OPML {
|
||||||
head: Some(opml::Head {
|
head: Some(opml::Head {
|
||||||
title: Some("ipx subscriptions".into()),
|
title: Some("ipx subscriptions".into()),
|
||||||
@@ -1676,7 +1676,7 @@ async fn export_opml(
|
|||||||
}),
|
}),
|
||||||
..Default::default()
|
..Default::default()
|
||||||
};
|
};
|
||||||
for s in crate::subscriptions(&state.ctx)? {
|
for s in crate::subscriptions(&state.ctx).await? {
|
||||||
// A feed from an OPML subscription comes back with the OPML itself.
|
// A feed from an OPML subscription comes back with the OPML itself.
|
||||||
if s.managed || !mine.contains(&s.id) {
|
if s.managed || !mine.contains(&s.id) {
|
||||||
continue;
|
continue;
|
||||||
@@ -1684,7 +1684,7 @@ async fn export_opml(
|
|||||||
let title = state
|
let title = state
|
||||||
.ctx
|
.ctx
|
||||||
.db
|
.db
|
||||||
.feed_summary(&s.id)
|
.feed_summary(&s.id).await
|
||||||
.ok()
|
.ok()
|
||||||
.and_then(|sum| sum.title)
|
.and_then(|sum| sum.title)
|
||||||
.unwrap_or_else(|| s.id.clone());
|
.unwrap_or_else(|| s.id.clone());
|
||||||
@@ -1718,7 +1718,7 @@ async fn import_opml(
|
|||||||
// file arrives as text, is read here, and is gone when the request ends.
|
// file arrives as text, is read here, and is gone when the request ends.
|
||||||
let doc = opml::OPML::from_str(&body.xml)
|
let doc = opml::OPML::from_str(&body.xml)
|
||||||
.map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?;
|
.map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?;
|
||||||
let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id)?;
|
let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id).await?;
|
||||||
if added > 0 {
|
if added > 0 {
|
||||||
scan_soon(&state, None).await;
|
scan_soon(&state, None).await;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user