8 Commits

Author SHA1 Message Date
5f6bdbfbcb Release 0.8.0
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 22:31:56 +00:00
8849ba6bef Merge config-db: the feed catalogue and server settings in the database (#18)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 22:29:16 +00:00
1cafd8d6e3 Keep the feed catalogue and server settings in the database
Phase 3 of #18. Two tables: catalogue (each feed's config::Feed as JSON, so a
new feed setting needs no column) and settings (general: the five server
settings the admin page edits). config.toml keeps what is needed before the
database is reached, or decides who gets in: paths, [torrent], [web].

ipx still runs from one in-memory Config, assembled at start from both
(assemble_config). The eight places that saved config.toml and re-read it now
call Ctx::store_cfg, which writes the database and swaps the copy in memory; the
first-run web token, which is config.toml's, is written there.

The first start on a database with no catalogue imports config.toml's feeds and
settings in one transaction whose first insert is the settings row, so two ipx
starting at once cannot both import; it then trims config.toml, keeping the
original as config.toml.pre-database. After that, feeds written into the file are
ignored with a warning. copy-db skips it, and copies both tables.

Rehearsed on a clone of production's database with production's config: all 130
feeds imported, the file trimmed, and the feed list, settings and directory
identical to the live server's.

Postgres connections now ask for no notices. Every CREATE ... IF NOT EXISTS on an
existing table sends one, eleven per open; sqlx logs them, and
tracing-subscriber 0.3.23's per-layer filters then dropped the next line ipx
logged -- the import's own message went missing that way. Proved by toggling it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 22:28:49 +00:00
f09bb4a11c Revert pinned items rising to the top of their list
Sorting by the pin column, or the Pinned tab, was enough. order_sql loses its
pinned_first option, pinning no longer reloads the list, and the tests and
changelog line for #35 go. The NULLS FIRST/LAST ordering from the Postgres work
stays.

Closes #36.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 22:08:37 +00:00
973ebdd33a The database URL's env file lives beside the compose file
Arcane runs compose in its own container, where /mnt/fast/appdata does not
exist, so an absolute env_file path there failed its update with 'env file not
found'. The file is now ipodderx.env in the content project, referred to
relatively.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 21:56:17 +00:00
bc7491a377 docker-compose.yml: the database URL, as production has it
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 21:45:51 +00:00
c8df148546 Merge seaorm: the database through SeaORM, on SQLite or Postgres (#18)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 21:42:27 +00:00
c1c06229c8 Docs: Postgres in production
Where the database now is and how to reach it, IPX_DATABASE_URL and
IPX_TEST_DATABASE_URL, copy-db, backups, and the title sort on Postgres.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 21:42:27 +00:00
13 changed files with 468 additions and 119 deletions

View File

@@ -7,14 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
## [0.8.0] - 2026-09-18
### Added
- ipx can keep its data in Postgres: set `IPX_DATABASE_URL` to a `postgres://` URL. Without it,
it is the SQLite `state.db` as before. `ipx copy-db <state.db>` moves an existing database
across, everything in one go.
- The catalogue of feeds and the server settings the admin page edits are kept in the database
rather than config.toml, which keeps where things are, the torrent settings and who may sign
in. The first start takes them from config.toml and trims it, keeping the original as
`config.toml.pre-database`; feeds added to config.toml after that are ignored, with a warning.
### Changed ### Changed
- The database is reached through SeaORM, on the way to Postgres (issue #18); it is still the - The database is reached through SeaORM, which is what lets it be SQLite or Postgres; on SQLite
same SQLite file, and nothing you see changes. A database from before 0.7 has to be opened by a nothing you see changes. On Postgres, sorting by title or feed follows the language's order (an
0.7 release first, which brings its tables up to date. accented letter beside the plain one) rather than raw bytes. A database from before 0.7 has to
- A pinned item sits at the top of its list, above everything else in whatever order you sort be opened by a 0.7 release first, which brings its tables up to date.
by, and moves there the moment you pin it. Sorting by the pin column itself still goes both
ways, and Currently Listening keeps its own order.
## [0.7.0] - 2026-09-18 ## [0.7.0] - 2026-09-18
@@ -493,6 +503,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`. - `ipx import` and `ipx export` for OPML, and systemd units in `contrib/`.
[unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.6.1...main [unreleased]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.6.1...main
[0.8.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.7.0...v0.8.0
[0.7.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.6.1...v0.7.0 [0.7.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.6.1...v0.7.0
[0.6.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.6.0...v0.6.1 [0.6.1]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.6.0...v0.6.1
[0.6.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.5...v0.6.0 [0.6.0]: https://git.sdf1.net/rays/ipodderx-rs/compare/v0.5.5...v0.6.0

View File

@@ -13,8 +13,9 @@ Arcane project `content`: `/mnt/fast/arcane/projects/content/compose.yaml`. That
| | Host | In the container | | | Host | In the container |
|---|---|---| |---|---|---|
| Image | `192.168.1.130:5000/ipodderx:latest` | | | Image | `192.168.1.130:5000/ipodderx:latest` | |
| Config | `/mnt/fast/appdata/ipodderx/config.toml` | `/config/config.toml` | | Config | `/mnt/fast/appdata/ipodderx/config.toml`: bind address, token, trusted proxies, torrent, paths. The feeds and server settings are in the database | `/config/config.toml` |
| Database | `/mnt/user/ipodderx/state.db` | `/data/state.db` | | Database | Postgres 18, database `ipodderx`, login `ipodderx`, on the `postgres` container of the Arcane project `databases` (`192.168.1.130:5433`). The URL is in `ipodderx.env` beside the compose file (`/mnt/fast/arcane/projects/content/ipodderx.env`, mode 600), passed to the container as `IPX_DATABASE_URL`. A relative `env_file`: Arcane runs compose in its own container, where `/mnt/fast/appdata` does not exist | |
| Old database | `/mnt/user/ipodderx/state.db`, SQLite, used until the move to Postgres on 2026-09-18 and kept for rollback | `/data/state.db` |
| Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` | | Downloads | `/mnt/user/ipodderx/downloads` | `/downloads` |
| Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` | | Web UI | `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel | `0.0.0.0:8099` |
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway | | Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | trusts `Cf-Access-Authenticated-User-Email` from `192.168.16.1`, the `content_default` gateway |
@@ -126,9 +127,17 @@ Non-trivial logic leaves one runnable check behind. Pure functions (`merge_polic
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged` * **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
and `position` columns from before accounts; two bugs came from queries still reading them and `position` columns from before accounts; two bugs came from queries still reading them
(retention, and the entry pruner), and they were dropped in 0.5. (retention, and the entry pruner), and they were dropped in 0.5.
* **The catalogue is config.toml; the subscriptions are in the database.** A feed exists once; * **The catalogue and the server settings are in the database, not config.toml** (issue #18):
tables `catalogue` (each feed's `config::Feed` as JSON) and `settings` (`general`:
`config::Stored`). ipx still runs from one in-memory `Config`, config.toml for where things are
and who gets in, the database for the rest (`assemble_config`); a change goes through
`Ctx::store_cfg`, never a write to the file. The first start on a database without them imports
config.toml's and trims the file, keeping `config.toml.pre-database`. A feed exists once;
`subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are `subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
derived and never written to config. derived and never in the catalogue.
* **Postgres connections ask for no notices** (`client_min_messages=warning`, `db::url_for`).
Postgres sends one for every `CREATE ... IF NOT EXISTS` on something existing, sqlx logs each,
and tracing-subscriber's per-layer filters then dropped the next line ipx logged.
* **One fetch serves everyone**, so scan policy is a union of subscribers' wants (`merge_policy`). * **One fetch serves everyone**, so scan policy is a union of subscribers' wants (`merge_policy`).
Anyone wanting an item is enough to fetch it. Anyone wanting an item is enough to fetch it.
* **The UI hiding a control is not enforcement.** Admin-only actions check `user.is_admin` in the * **The UI hiding a control is not enforcement.** Admin-only actions check `user.is_admin` in the

2
Cargo.lock generated
View File

@@ -1820,7 +1820,7 @@ checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
[[package]] [[package]]
name = "ipx" name = "ipx"
version = "0.7.0" version = "0.8.0"
dependencies = [ dependencies = [
"ammonia", "ammonia",
"anyhow", "anyhow",

View File

@@ -1,6 +1,6 @@
[package] [package]
name = "ipx" name = "ipx"
version = "0.7.0" version = "0.8.0"
edition = "2024" edition = "2024"
[dependencies] [dependencies]

View File

@@ -8,6 +8,9 @@ services:
PGID: "100" PGID: "100"
TZ: "America/Toronto" TZ: "America/Toronto"
IPX_LOG: "ipx=info" IPX_LOG: "ipx=info"
# IPX_DATABASE_URL=postgres://... to use Postgres; without it, /data/state.db (SQLite).
env_file:
- ipodderx.env # relative: Arcane resolves it inside its own container
ports: ports:
- "8099:8099" # web UI - "8099:8099" # web UI
- "6881:6881/tcp" # BitTorrent peers - "6881:6881/tcp" # BitTorrent peers

View File

@@ -1,7 +1,19 @@
# Configuration # Configuration
One TOML file, read at startup and re-read whenever the web UI writes to it — most changes take Two places. **config.toml** holds what ipx needs before it reaches its database, and what decides
effect without a restart. Default location `$XDG_CONFIG_HOME/ipx/config.toml` who gets in: where things are (`download_dir`, `socket`, `organize`), `[torrent]` and `[web]`.
**The database** holds the catalogue of feeds (`[feeds.<id>]` below) and the server settings the
admin page edits (`schedule`, `max_total_gb`, `max_age_days`, `max_new_per_check`,
`media_types`). Change those in the web UI, or with `ipx add`, `ipx rm` and `ipx import`; they
take effect without a restart.
The first time ipx meets a database that holds no catalogue, it takes the feeds and those
settings from config.toml, then rewrites config.toml without them, keeping the original beside it
as `config.toml.pre-database`. After that, feeds or those settings written into config.toml are
ignored, with a warning in the log saying so. The sections below describe them as they were
written in config.toml, which is still how a fresh install begins.
config.toml's default location is `$XDG_CONFIG_HOME/ipx/config.toml`
(`~/.config/ipx/config.toml`), overridden with `--config` or `$IPX_CONFIG`. (`~/.config/ipx/config.toml`), overridden with `--config` or `$IPX_CONFIG`.
| What | Where | Override | | What | Where | Override |
@@ -11,8 +23,10 @@ effect without a restart. Default location `$XDG_CONFIG_HOME/ipx/config.toml`
| Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` | | Control socket | `$XDG_RUNTIME_DIR/ipx.sock` | `[general] socket` |
| Downloads | `[general] download_dir` | — | | Downloads | `[general] download_dir` | — |
`~` is expanded in paths. The database is SQLite in WAL mode; back it up by copying `state.db` `~` is expanded in paths. The database is SQLite in WAL mode unless `IPX_DATABASE_URL` names a
while the daemon is stopped, or with `sqlite3 state.db .backup`. Postgres database instead. Back SQLite up by copying `state.db` while the daemon is stopped, or
with `sqlite3 state.db .backup`; back Postgres up with `pg_dump`. `ipx copy-db <state.db>` copies a
SQLite database into the empty Postgres one `IPX_DATABASE_URL` names.
## `[general]` ## `[general]`
@@ -28,6 +42,9 @@ max_new_per_check = 3 # per feed, per scan. 0 = unlimited
media_types = ["audio", "video"] media_types = ["audio", "video"]
``` ```
`schedule`, `max_total_gb`, `max_age_days`, `max_new_per_check` and `media_types` move into the
database as described above; `download_dir`, `socket` and `organize` stay in config.toml.
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to * **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI. be polled *less* often, and a per-feed `schedule` overrides both. Admin-only from the UI.
* **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`. * **`organize`** — `feed` files downloads under the feed's folder; `date` under `YYYY-MM-DD`.
@@ -88,8 +105,9 @@ itself carry a credential. Put TLS in front of it if that matters.
## `[feeds.<id>]` ## `[feeds.<id>]`
The table key is the feed id: stable, human-readable, and used in paths and the API. `ipx add` Kept in the database once ipx has moved them in: a feed's settings are changed in the web UI, and
derives it from the feed title. feeds come and go with `ipx add`, `ipx rm` and `ipx import`. The table key is the feed id: stable,
human-readable, and used in paths and the API. `ipx add` derives it from the feed title.
```toml ```toml
[feeds.atp] [feeds.atp]
@@ -107,8 +125,8 @@ With more than one account, **`keywords`, `auto_download`, `allow_explicit` and
config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself
and are the same for everyone. See [users.md](users.md). and are the same for everyone. See [users.md](users.md).
Feeds derived from a subscribed OPML are **not** written here: the OPML is the source of truth and Feeds derived from a subscribed OPML are **not** in the catalogue: the OPML is the source of truth
they are re-derived on every scan. Editing one in the UI promotes it to a real config entry. and they are re-derived on every scan. Editing one in the UI promotes it to a catalogue entry.
## Environment ## Environment
@@ -116,6 +134,8 @@ they are re-derived on every scan. Editing one in the UI promotes it to a real c
|---|---| |---|---|
| `IPX_CONFIG` | Config file path | | `IPX_CONFIG` | Config file path |
| `IPX_DATA_DIR` | Directory holding `state.db` | | `IPX_DATA_DIR` | Directory holding `state.db` |
| `IPX_DATABASE_URL` | A `postgres://user:password@host:port/database` URL: use that database instead of `state.db` |
| `IPX_TEST_DATABASE_URL` | For `cargo test`: run the database tests on this Postgres database too, each in a schema of its own |
| `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) | | `IPX_LOG` | What reaches stderr (`ipx=debug`, `ipx::scan=debug`, …) |
| `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view | | `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view |
| `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches | | `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches |

View File

@@ -261,21 +261,86 @@ impl Config {
Ok(cfg) Ok(cfg)
} }
pub fn save(&self, path: &Path) -> Result<()> { }
if let Some(dir) = path.parent() {
std::fs::create_dir_all(dir) /// What the database keeps of the configuration (issue #18): the server settings the admin page
.with_context(|| format!("creating {}", dir.display()))?; /// edits, and, beside them in `Db::stored_config`, the catalogue of feeds. The rest -- where
/// things are, who may sign in, the torrent session -- is needed before the database is reached,
/// or decides who gets in, and stays in config.toml.
#[derive(Debug, Clone, PartialEq, Deserialize, Serialize)]
pub struct Stored {
pub schedule: String,
pub max_total_gb: f64,
pub max_age_days: u64,
pub max_new_per_check: usize,
pub media_types: Vec<String>,
}
/// `[general]` keys that live in the database once it holds the configuration.
const STORED_KEYS: [&str; 5] = ["schedule", "max_total_gb", "max_age_days", "max_new_per_check", "media_types"];
impl Stored {
pub fn of(cfg: &Config) -> Self {
let g = &cfg.general;
Self {
schedule: g.schedule.clone(),
max_total_gb: g.max_total_gb,
max_age_days: g.max_age_days,
max_new_per_check: g.max_new_per_check,
media_types: g.media_types.clone(),
} }
let text = toml::to_string_pretty(self)?;
std::fs::write(path, text).with_context(|| format!("writing {}", path.display()))?;
// Passwords may live in here.
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))?;
}
Ok(())
} }
pub fn apply(self, cfg: &mut Config) {
let g = &mut cfg.general;
g.schedule = self.schedule;
g.max_total_gb = self.max_total_gb;
g.max_age_days = self.max_age_days;
g.max_new_per_check = self.max_new_per_check;
g.media_types = self.media_types;
}
}
impl Config {
/// config.toml as it is kept once the database holds the feeds and server settings: the same
/// file without `[feeds]` or the `[general]` keys in `Stored`.
pub fn save_bootstrap(&self, path: &Path) -> Result<()> {
let mut v = toml::Value::try_from(self)?;
if let Some(t) = v.as_table_mut() {
t.remove("feeds");
if let Some(g) = t.get_mut("general").and_then(|g| g.as_table_mut()) {
for k in STORED_KEYS {
g.remove(k);
}
}
}
write_private(path, &toml::to_string_pretty(&v)?)
}
/// Whether config.toml still lists feeds or server settings, which the database now holds:
/// an edit there would otherwise go unnoticed.
pub fn file_holds_stored(path: &Path) -> bool {
let Ok(text) = std::fs::read_to_string(path) else { return false };
let Ok(v) = text.parse::<toml::Table>() else { return false };
v.get("feeds").and_then(|f| f.as_table()).is_some_and(|f| !f.is_empty())
|| v.get("general")
.and_then(|g| g.as_table())
.is_some_and(|g| STORED_KEYS.iter().any(|k| g.contains_key(*k)))
}
}
/// Writes a config file readable by its owner alone: feed passwords have lived in it.
fn write_private(path: &Path, text: &str) -> Result<()> {
if let Some(dir) = path.parent() {
std::fs::create_dir_all(dir).with_context(|| format!("creating {}", dir.display()))?;
}
std::fs::write(path, text).with_context(|| format!("writing {}", path.display()))?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))?;
}
Ok(())
} }
/// `$IPX_CONFIG`, else `$XDG_CONFIG_HOME/ipx/config.toml`. /// `$IPX_CONFIG`, else `$XDG_CONFIG_HOME/ipx/config.toml`.
@@ -370,6 +435,36 @@ fn expand_tilde(p: &Path) -> PathBuf {
mod tests { mod tests {
use super::*; use super::*;
#[test]
fn the_file_kept_beside_the_database_has_no_feeds_or_server_settings() {
let cfg: Config = toml::from_str(
r#"
[general]
download_dir = "/downloads"
schedule = "every 2h"
max_new_per_check = 7
media_types = ["audio"]
[web]
bind = "0.0.0.0:8099"
token = "t"
[feeds.show]
url = "http://x/show.xml"
"#,
)
.unwrap();
let path = std::env::temp_dir().join(format!("ipx-bootstrap-{}.toml", std::process::id()));
std::fs::write(&path, toml::to_string(&cfg).unwrap()).unwrap();
assert!(Config::file_holds_stored(&path), "a whole config.toml holds them");
cfg.save_bootstrap(&path).unwrap();
let text = std::fs::read_to_string(&path).unwrap();
assert!(!Config::file_holds_stored(&path), "{text}");
let back: Config = toml::from_str(&text).unwrap();
assert!(back.feeds.is_empty());
assert_eq!(back.web.token, "t", "who may sign in stays in the file");
assert_eq!(back.general.download_dir, PathBuf::from("/downloads"), "where things are, too");
std::fs::remove_file(&path).unwrap();
}
#[test] #[test]
fn parses_a_config_and_applies_defaults() { fn parses_a_config_and_applies_defaults() {
let cfg: Config = toml::from_str( let cfg: Config = toml::from_str(

183
src/db.rs
View File

@@ -2,7 +2,7 @@
//! per-feed .ipxd plists, history.dat and qmcache.dat. //! per-feed .ipxd plists, history.dat and qmcache.dat.
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use crate::entity::{enclosures, entries, feeds, sessions, subscriptions, users}; use crate::entity::{catalogue, enclosures, entries, feeds, sessions, settings, subscriptions, users};
use sea_orm::sea_query::{Expr, Func}; use sea_orm::sea_query::{Expr, Func};
use sea_orm::{ use sea_orm::{
ActiveModelTrait, ColumnTrait, ConnectionTrait, EntityTrait, PaginatorTrait, QueryFilter, QueryOrder, Set, ActiveModelTrait, ColumnTrait, ConnectionTrait, EntityTrait, PaginatorTrait, QueryFilter, QueryOrder, Set,
@@ -55,6 +55,8 @@ async fn create_missing(orm: &sea_orm::DatabaseConnection) -> Result<()> {
schema.create_table_from_entity(subscriptions::Entity), schema.create_table_from_entity(subscriptions::Entity),
schema.create_table_from_entity(entry_state::Entity), schema.create_table_from_entity(entry_state::Entity),
schema.create_table_from_entity(sessions::Entity), schema.create_table_from_entity(sessions::Entity),
schema.create_table_from_entity(catalogue::Entity),
schema.create_table_from_entity(settings::Entity),
] { ] {
orm.execute(table.if_not_exists()).await.context("creating the schema")?; orm.execute(table.if_not_exists()).await.context("creating the schema")?;
} }
@@ -100,6 +102,9 @@ pub fn location() -> String {
.unwrap_or_else(|| crate::config::data_dir().join("state.db").display().to_string()) .unwrap_or_else(|| crate::config::data_dir().join("state.db").display().to_string())
} }
/// The Postgres connection option that keeps notices from being sent at all; see `connect`.
const QUIET: &str = "options=-c%20client_min_messages%3Dwarning";
/// A URL fit for a log or an error: the password taken out. /// A URL fit for a log or an error: the password taken out.
fn redact(url: &str) -> String { fn redact(url: &str) -> String {
match (url.find("://"), url.rfind('@')) { match (url.find("://"), url.rfind('@')) {
@@ -118,15 +123,30 @@ fn is_postgres(location: &str) -> bool {
/// sqlx's defaults for SQLite are what ipx wants: foreign keys on, and a five-second wait for a /// sqlx's defaults for SQLite are what ipx wants: foreign keys on, and a five-second wait for a
/// lock, which is what rusqlite was set to. /// lock, which is what rusqlite was set to.
async fn connect(location: &str) -> Result<sea_orm::DatabaseConnection> { async fn connect(location: &str) -> Result<sea_orm::DatabaseConnection> {
let url = let mut opts = sea_orm::ConnectOptions::new(url_for(location));
if is_postgres(location) { location.to_owned() } else { format!("sqlite://{location}?mode=rwc") };
let mut opts = sea_orm::ConnectOptions::new(url);
opts.sqlx_logging(false); opts.sqlx_logging(false);
sea_orm::Database::connect(opts) sea_orm::Database::connect(opts)
.await .await
.with_context(|| format!("opening {}", redact(location))) .with_context(|| format!("opening {}", redact(location)))
} }
/// The URL `connect` hands sqlx for a location: a SQLite file, created if missing, or a Postgres
/// URL with notices turned off.
fn url_for(location: &str) -> String {
if !is_postgres(location) {
format!("sqlite://{location}?mode=rwc")
} else if location.contains("options=") {
location.to_owned() // someone chose their own; theirs stands
} else {
// Warnings and up only. Postgres answers every CREATE ... IF NOT EXISTS on an existing
// table with a notice, eleven on each open; sqlx logs each, no filter here wants them,
// and tracing-subscriber's per-layer filters then swallowed the next line ipx logged
// (0.3.23: "moved the feeds ... into the database" went missing that way).
let sep = if location.contains('?') { '&' } else { '?' };
format!("{location}{sep}{QUIET}")
}
}
/// One person's wants for one feed. `None` in a field means the feed's own setting stands. /// One person's wants for one feed. `None` in a field means the feed's own setting stands.
#[derive(Debug, Clone, Default)] #[derive(Debug, Clone, Default)]
@@ -280,7 +300,9 @@ impl Db {
admin.execute_unprepared(&format!("CREATE SCHEMA {schema}")).await?; admin.execute_unprepared(&format!("CREATE SCHEMA {schema}")).await?;
// Every connection in the pool starts in it, so each test sees only its own tables. // Every connection in the pool starts in it, so each test sees only its own tables.
let sep = if url.contains('?') { '&' } else { '?' }; let sep = if url.contains('?') { '&' } else { '?' };
let orm = connect(&format!("{url}{sep}options=-c%20search_path%3D{schema}")).await?; let orm =
connect(&format!("{url}{sep}options=-c%20search_path%3D{schema}%20-c%20client_min_messages%3Dwarning"))
.await?;
create_missing(&orm).await?; create_missing(&orm).await?;
return Ok(Self { orm, tmp: None }); return Ok(Self { orm, tmp: None });
} }
@@ -765,10 +787,10 @@ fn entries_from(a: &mut Args, user_id: i64, feed_id: Option<&str>, filter: Filte
/// ///
/// ponytail: file type and size look at the item's first and largest file. The row shows the file /// ponytail: file type and size look at the item's first and largest file. The row shows the file
/// it summarises, which is almost always that one; sort by that one if they ever disagree. /// it summarises, which is almost always that one; sort by that one if they ever disagree.
/// `pinned_first` puts your pinned items above the rest, each part in the order asked for, so a ///
/// pin keeps something at the top of its list (issue #35). Not when sorting by the pin itself, /// Pinned items are not lifted above the rest: sorting by the pin column, or the Pinned tab, does
/// where the direction chosen is the point. /// that when it is wanted, and lifting them always was undone (issues #35 and #36).
pub fn order_sql(col: &str, dir: &str, pinned_first: bool) -> String { pub fn order_sql(col: &str, dir: &str) -> String {
let expr = match col { let expr = match col {
"kept" => "coalesce(s.flagged, false)", "kept" => "coalesce(s.flagged, false)",
"title" => "lower(coalesce(e.title, ''))", "title" => "lower(coalesce(e.title, ''))",
@@ -781,9 +803,8 @@ pub fn order_sql(col: &str, dir: &str, pinned_first: bool) -> String {
// and Postgres as the largest, so "largest first" on Postgres opened with every item that has // and Postgres as the largest, so "largest first" on Postgres opened with every item that has
// no file. NULLS FIRST going up and LAST going down keeps what SQLite did. // no file. NULLS FIRST going up and LAST going down keeps what SQLite did.
let dir = if dir == "asc" { "ASC NULLS FIRST" } else { "DESC NULLS LAST" }; let dir = if dir == "asc" { "ASC NULLS FIRST" } else { "DESC NULLS LAST" };
let pins = if pinned_first && col != "kept" { "coalesce(s.flagged, false) DESC, " } else { "" };
// The guid breaks what ties remain: SQLite's rowid did, and Postgres has none. // The guid breaks what ties remain: SQLite's rowid did, and Postgres has none.
format!("{pins}{expr} {dir}, coalesce(e.published, e.first_seen) DESC, e.guid DESC") format!("{expr} {dir}, coalesce(e.published, e.first_seen) DESC, e.guid DESC")
} }
/// Which slice of a feed the UI is asking for. /// Which slice of a feed the UI is asking for.
@@ -964,6 +985,76 @@ impl Db {
Ok(()) Ok(())
} }
// ---- the configuration: the catalogue and the server settings ----
/// The feeds and server settings, once the database holds them; None before, when they are
/// still config.toml's.
pub async fn stored_config(
&self,
) -> Result<Option<(crate::config::Stored, std::collections::BTreeMap<String, crate::config::Feed>)>> {
let Some(general) = settings::Entity::find_by_id("general".to_owned()).one(&self.orm).await? else {
return Ok(None);
};
let stored = serde_json::from_str(&general.value).context("reading the stored server settings")?;
let mut feeds = std::collections::BTreeMap::new();
for row in catalogue::Entity::find().all(&self.orm).await? {
let feed = serde_json::from_str(&row.spec).with_context(|| format!("reading feed {}", row.id))?;
feeds.insert(row.id, feed);
}
Ok(Some((stored, feeds)))
}
/// Takes the feeds and server settings from `cfg` (config.toml, as read) into a database that
/// has none. False when it already had them: another process got there first, and its copy
/// stands. One transaction, and the settings row goes in first, so two processes starting at
/// once cannot both import.
pub async fn import_config(&self, cfg: &crate::config::Config) -> Result<bool> {
use sea_orm::TransactionTrait;
let backend = self.orm.get_database_backend();
let tx = self.orm.begin().await?;
let claimed = tx
.execute_raw(Statement::from_sql_and_values(
backend,
"INSERT INTO settings (name, value) VALUES ('general', $1) ON CONFLICT DO NOTHING",
vec![serde_json::to_string(&crate::config::Stored::of(cfg))?.into()],
))
.await?
.rows_affected();
if claimed == 0 {
return Ok(false); // dropped, so rolled back
}
for (id, feed) in &cfg.feeds {
catalogue::ActiveModel { id: Set(id.clone()), spec: Set(serde_json::to_string(feed)?) }
.insert(&tx)
.await?;
}
tx.commit().await?;
Ok(true)
}
/// Writes the feeds and server settings as they now stand: what config.toml's save did.
/// The whole catalogue at once, in one transaction, so a feed removed goes too.
pub async fn store_config(&self, cfg: &crate::config::Config) -> Result<()> {
use sea_orm::TransactionTrait;
let backend = self.orm.get_database_backend();
let tx = self.orm.begin().await?;
tx.execute_raw(Statement::from_sql_and_values(
backend,
"INSERT INTO settings (name, value) VALUES ('general', $1)
ON CONFLICT (name) DO UPDATE SET value = excluded.value",
vec![serde_json::to_string(&crate::config::Stored::of(cfg))?.into()],
))
.await?;
catalogue::Entity::delete_many().exec(&tx).await?;
for (id, feed) in &cfg.feeds {
catalogue::ActiveModel { id: Set(id.clone()), spec: Set(serde_json::to_string(feed)?) }
.insert(&tx)
.await?;
}
tx.commit().await?;
Ok(())
}
// ---- moving to another database ---- // ---- moving to another database ----
/// Copies every row of `from` into this database, which must be empty: the move from the /// Copies every row of `from` into this database, which must be empty: the move from the
@@ -986,6 +1077,8 @@ impl Db {
("subscriptions", copy_table::<subscriptions::Entity>(&from.orm, &tx).await?), ("subscriptions", copy_table::<subscriptions::Entity>(&from.orm, &tx).await?),
("entry_state", copy_table::<entry_state::Entity>(&from.orm, &tx).await?), ("entry_state", copy_table::<entry_state::Entity>(&from.orm, &tx).await?),
("sessions", copy_table::<sessions::Entity>(&from.orm, &tx).await?), ("sessions", copy_table::<sessions::Entity>(&from.orm, &tx).await?),
("catalogue", copy_table::<catalogue::Entity>(&from.orm, &tx).await?),
("settings", copy_table::<settings::Entity>(&from.orm, &tx).await?),
]; ];
if self.orm.get_database_backend() == sea_orm::DbBackend::Postgres { if self.orm.get_database_backend() == sea_orm::DbBackend::Postgres {
// The copied ids came with the rows; the counters that hand out new ones start past // The copied ids came with the rows; the counters that hand out new ones start past
@@ -1729,7 +1822,7 @@ mod tests {
).await ).await
.unwrap(); .unwrap();
let order = async |col: &str, dir: &str| -> Vec<String> { let order = async |col: &str, dir: &str| -> Vec<String> {
db.entries_in(1, None, Filter::All, None, 0, 50, &order_sql(col, dir, false)).await db.entries_in(1, None, Filter::All, None, 0, 50, &order_sql(col, dir)).await
.unwrap() .unwrap()
.into_iter() .into_iter()
.map(|e| e.guid) .map(|e| e.guid)
@@ -1751,20 +1844,11 @@ mod tests {
assert_eq!(order("published", "desc").await, ["c", "b", "a"]); assert_eq!(order("published", "desc").await, ["c", "b", "a"]);
db.set_entry_flag(1, "f", "a", EntryFlag::Flagged, true).await.unwrap(); db.set_entry_flag(1, "f", "a", EntryFlag::Flagged, true).await.unwrap();
assert_eq!(order("kept", "desc").await[0], "a"); assert_eq!(order("kept", "desc").await[0], "a");
// Pinned first: the pinned banana tops every sort, the rest in the order asked for. // A pin does not lift an item above the others in any other sort (#36).
let pinned = async |col: &str, dir: &str| -> Vec<String> { assert_eq!(order("published", "desc").await, ["c", "b", "a"]);
db.entries_in(1, None, Filter::All, None, 0, 50, &order_sql(col, dir, true)).await
.unwrap()
.into_iter()
.map(|e| e.guid)
.collect()
};
assert_eq!(pinned("published", "desc").await, ["a", "c", "b"]);
assert_eq!(pinned("title", "desc").await, ["a", "c", "b"]);
assert_eq!(pinned("kept", "asc").await[2], "a", "sorting by the pin itself keeps its direction");
// An unknown column or direction is newest first; the name itself never reaches the SQL. // An unknown column or direction is newest first; the name itself never reaches the SQL.
assert_eq!(order("title; DROP TABLE entries", "sideways").await, ["c", "b", "a"]); assert_eq!(order("title; DROP TABLE entries", "sideways").await, ["c", "b", "a"]);
assert!(!order_sql("x'; --", "asc", false).contains("x'")); assert!(!order_sql("x'; --", "asc").contains("x'"));
} }
#[tokio::test] #[tokio::test]
@@ -1813,7 +1897,7 @@ mod tests {
// Starring and position are just as private. // Starring and position are just as private.
db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).await.unwrap(); db.set_entry_flag(1, "f", "b", EntryFlag::Flagged, true).await.unwrap();
db.set_position(2, "f", "b", 42, Some(600)).await.unwrap(); db.set_position(2, "f", "b", 42, Some(600)).await.unwrap();
let order = order_sql("published", "desc", false); let order = order_sql("published", "desc");
let page = async |user| db.entries_in(user, Some("f"), Filter::All, None, 0, 50, &order).await.unwrap(); let page = async |user| db.entries_in(user, Some("f"), Filter::All, None, 0, 50, &order).await.unwrap();
let (ray, sam) = (page(1).await, page(2).await); let (ray, sam) = (page(1).await, page(2).await);
let ray_b = ray.iter().find(|e| e.guid == "b").unwrap(); let ray_b = ray.iter().find(|e| e.guid == "b").unwrap();
@@ -1927,7 +2011,7 @@ mod tests {
for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged, Filter::InProgress] { for f in [Filter::All, Filter::Unread, Filter::Downloaded, Filter::Flagged, Filter::InProgress] {
// Both paths must run without erroring, and agree with each other. // Both paths must run without erroring, and agree with each other.
let order = order_sql("published", "desc", false); let order = order_sql("published", "desc");
let rows = db.entries_in(7, Some("f"), f, None, 0, 50, &order).await.unwrap(); let rows = db.entries_in(7, Some("f"), f, None, 0, 50, &order).await.unwrap();
let n = db.count_in(7, Some("f"), f, None).await.unwrap(); let n = db.count_in(7, Some("f"), f, None).await.unwrap();
assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page"); assert_eq!(rows.len() as i64, n, "{f:?} count disagrees with the page");
@@ -1946,7 +2030,7 @@ mod tests {
"search is case-insensitive and covers the description"); "search is case-insensitive and covers the description");
// Currently Listening: started, not finished, and not just an accidental tap. // Currently Listening: started, not finished, and not just an accidental tap.
let order = order_sql("published", "desc", false); let order = order_sql("published", "desc");
let listening = async || { let listening = async || {
let rows = db.entries_in(7, Some("f"), Filter::InProgress, None, 0, 50, &order).await.unwrap(); let rows = db.entries_in(7, Some("f"), Filter::InProgress, None, 0, 50, &order).await.unwrap();
rows.into_iter().map(|e| e.guid).collect::<Vec<_>>() rows.into_iter().map(|e| e.guid).collect::<Vec<_>>()
@@ -2087,6 +2171,51 @@ mod tests {
assert!(db.pinned_feeds(me).await.unwrap().is_empty()); assert!(db.pinned_feeds(me).await.unwrap().is_empty());
} }
#[test]
fn postgres_is_asked_for_no_notices_and_a_password_never_reaches_a_log() {
assert_eq!(url_for("postgres://u:p@h/d"), format!("postgres://u:p@h/d?{QUIET}"));
assert_eq!(url_for("postgres://u:p@h/d?sslmode=disable"), format!("postgres://u:p@h/d?sslmode=disable&{QUIET}"));
assert_eq!(url_for("postgres://u:p@h/d?options=-c%20x%3Dy"), "postgres://u:p@h/d?options=-c%20x%3Dy", "theirs stands");
assert_eq!(url_for("/data/state.db"), "sqlite:///data/state.db?mode=rwc");
assert_eq!(redact("postgres://ipodderx:s3cret@h:5433/ipodderx"), "postgres://ipodderx:***@h:5433/ipodderx");
}
#[tokio::test]
async fn the_configuration_goes_in_once_and_comes_back_as_it_went() {
let db = Db::memory().await.unwrap();
assert!(db.stored_config().await.unwrap().is_none(), "nothing until it is taken in");
let mut cfg: crate::config::Config = toml::from_str(
r#"
[general]
schedule = "every 2h"
max_total_gb = 1.5
[feeds.a]
url = "http://x/a.xml"
keywords = ["one", "two"]
[feeds.b]
url = "http://x/b.xml"
password = "secret"
"#,
)
.unwrap();
assert!(db.import_config(&cfg).await.unwrap());
assert!(!db.import_config(&cfg).await.unwrap(), "a second import finds the first and stands back");
let (stored, feeds) = db.stored_config().await.unwrap().unwrap();
assert_eq!(stored, crate::config::Stored::of(&cfg));
assert_eq!(feeds.keys().collect::<Vec<_>>(), ["a", "b"]);
assert_eq!(feeds["a"].keywords, ["one", "two"]);
assert_eq!(feeds["b"].password.as_deref(), Some("secret"));
// Storing is the whole catalogue: a feed taken out goes, one put in arrives.
cfg.feeds.remove("a");
cfg.feeds.insert("c".into(), cfg.feeds["b"].clone());
cfg.general.schedule = "every 30m".into();
db.store_config(&cfg).await.unwrap();
let (stored, feeds) = db.stored_config().await.unwrap().unwrap();
assert_eq!(stored.schedule, "every 30m");
assert_eq!(feeds.keys().collect::<Vec<_>>(), ["b", "c"]);
}
#[tokio::test] #[tokio::test]
async fn enclosure_url_is_the_dedupe_key() { async fn enclosure_url_is_the_dedupe_key() {
let db = Db::memory().await.unwrap(); let db = Db::memory().await.unwrap();

View File

@@ -242,3 +242,44 @@ pub mod sessions {
owned_by_user!(); owned_by_user!();
} }
/// The catalogue: every feed configured, with its shared settings as `config::Feed` in JSON, so a
/// new setting on a feed needs no new column. It was config.toml's `[feeds]` (issue #18).
pub mod catalogue {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "catalogue")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub id: String,
#[sea_orm(column_type = "Text")]
pub spec: String,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}
/// The server's settings, by name, each a JSON value. `general` is `config::Stored`: what was in
/// config.toml's `[general]` and the admin page edits. Its row being there is what says the
/// configuration has moved in (issue #18).
pub mod settings {
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "settings")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false, column_type = "Text")]
pub name: String,
#[sea_orm(column_type = "Text")]
pub value: String,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
}

View File

@@ -144,13 +144,19 @@ impl Ctx {
} }
/// Re-reads config.toml into the live snapshot. /// Re-reads config.toml into the live snapshot.
pub fn reload_cfg(&self, path: &std::path::Path) -> Result<()> { /// Keeps a changed catalogue or server settings: in the database, and for everything running
let fresh = config::Config::load(path)?; /// here from now on. What config.toml's save and reload did, before the database held them.
*self.cfg.write().unwrap() = std::sync::Arc::new(fresh); pub async fn store_cfg(&self, cfg: config::Config) -> Result<()> {
tracing::info!("config reloaded"); self.db.store_config(&cfg).await?;
self.set_cfg(cfg);
Ok(()) Ok(())
} }
fn set_cfg(&self, cfg: config::Config) {
*self.cfg.write().unwrap() = std::sync::Arc::new(cfg);
tracing::info!("config reloaded");
}
async fn torrents(&self) -> Result<&torrent::Torrents> { async fn torrents(&self) -> Result<&torrent::Torrents> {
let cfg = self.cfg(); let cfg = self.cfg();
self.torrents self.torrents
@@ -211,6 +217,14 @@ async fn main() -> Result<()> {
return ipc::proxy(&cfg.general.socket, cmd).await; return ipc::proxy(&cfg.general.socket, cmd).await;
} }
// copy-db fills an empty database from another, configuration included; taking config.toml
// into it first would have the copy collide with it.
let cfg = if matches!(cli.command, Command::CopyDb { .. }) {
cfg
} else {
assemble_config(&db, cfg, &config_path).await?
};
let is_daemon = matches!(cli.command, Command::Daemon { .. }); let is_daemon = matches!(cli.command, Command::Daemon { .. });
let (events, _) = broadcast::channel(1024); let (events, _) = broadcast::channel(1024);
let ctx = Arc::new(Ctx { let ctx = Arc::new(Ctx {
@@ -227,14 +241,14 @@ async fn main() -> Result<()> {
}); });
match cli.command { match cli.command {
Command::List => list(&ctx, &config_path).await, Command::List => list(&ctx).await,
Command::Daemon { web } => daemon(ctx, config_path, web, events).await, Command::Daemon { web } => daemon(ctx, config_path, web, events).await,
Command::Add { url, folder, keywords } => { Command::Add { url, folder, keywords } => {
add(&ctx, &config_path, &url, folder, keywords).await add(&ctx, &url, folder, keywords).await
} }
Command::Rm { feed } => rm(&ctx, &config_path, &feed).await, Command::Rm { feed } => rm(&ctx, &feed).await,
Command::User { cmd } => user_cmd(&ctx, cmd).await, Command::User { cmd } => user_cmd(&ctx, cmd).await,
Command::Import { file } => import(&ctx, &config_path, &file).await, Command::Import { file } => import(&ctx, &file).await,
Command::Export { file } => export(&ctx, &file).await, Command::Export { file } => export(&ctx, &file).await,
Command::CopyDb { from } => copy_db(&ctx, &from).await, Command::CopyDb { from } => copy_db(&ctx, &from).await,
_ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await, _ => run(&ctx, wire_cmd.expect("only List and Daemon have no wire form")).await,
@@ -533,8 +547,9 @@ async fn start_web(
fresh.web.enabled = true; fresh.web.enabled = true;
fresh.web.bind = bind.clone(); fresh.web.bind = bind.clone();
fresh.web.token = crate::auth::new_session_token(); fresh.web.token = crate::auth::new_session_token();
fresh.save(config_path)?; // The token is config.toml's, not the database's: it decides who gets in.
ctx.reload_cfg(config_path)?; fresh.save_bootstrap(config_path)?;
ctx.set_cfg(fresh.clone());
println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token); println!("web ui token generated. Open:\n http://{bind}/?token={}", fresh.web.token);
} else { } else {
println!( println!(
@@ -549,7 +564,6 @@ async fn start_web(
let state = web::WebState { let state = web::WebState {
ctx: ctx.clone(), ctx: ctx.clone(),
config_path: config_path.to_path_buf(),
cmds: cmds.clone(), cmds: cmds.clone(),
events: events.clone(), events: events.clone(),
}; };
@@ -575,7 +589,6 @@ async fn shutdown() {
/// Subscribes to one feed, naming it from its own title. /// Subscribes to one feed, naming it from its own title.
async fn add( async fn add(
ctx: &Ctx, ctx: &Ctx,
config_path: &std::path::Path,
url: &str, url: &str,
folder: Option<String>, folder: Option<String>,
keywords: Vec<String>, keywords: Vec<String>,
@@ -587,7 +600,7 @@ async fn add(
anyhow::bail!("already subscribed as {:?}", existing.id); anyhow::bail!("already subscribed as {:?}", existing.id);
} }
let id = add_one(ctx, &mut cfg, url, folder, keywords).await?; let id = add_one(ctx, &mut cfg, url, folder, keywords).await?;
cfg.save(config_path)?; ctx.store_cfg(cfg).await?;
println!("added {id}"); println!("added {id}");
Ok(()) Ok(())
} }
@@ -659,7 +672,7 @@ fn url_stem(url: &str) -> String {
.unwrap_or_else(|| url.to_owned()) .unwrap_or_else(|| url.to_owned())
} }
async fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()> { async fn rm(ctx: &Ctx, feed: &str) -> Result<()> {
let mut cfg = (*ctx.cfg()).clone(); let mut cfg = (*ctx.cfg()).clone();
if cfg.feeds.remove(feed).is_none() { if cfg.feeds.remove(feed).is_none() {
// Derived from an OPML: drop it here, though the subscription will list it again // Derived from an OPML: drop it here, though the subscription will list it again
@@ -668,14 +681,14 @@ async fn rm(ctx: &Ctx, config_path: &std::path::Path, feed: &str) -> Result<()>
println!("removed {feed}; it came from an OPML subscription and may return on the next read"); println!("removed {feed}; it came from an OPML subscription and may return on the next read");
return Ok(()); return Ok(());
} }
cfg.save(config_path)?; ctx.store_cfg(cfg).await?;
// State and files stay: re-adding the feed should not re-download its back catalogue. // State and files stay: re-adding the feed should not re-download its back catalogue.
println!("removed {feed}; downloads and history kept"); println!("removed {feed}; downloads and history kept");
retire_group(ctx, feed).await?; retire_group(ctx, feed).await?;
Ok(()) Ok(())
} }
async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path) -> Result<()> { async fn import(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
let text = std::fs::read_to_string(file) let text = std::fs::read_to_string(file)
.with_context(|| format!("reading {}", file.display()))?; .with_context(|| format!("reading {}", file.display()))?;
// The CLI speaks for the operator, as the shared web token does. // The CLI speaks for the operator, as the shared web token does.
@@ -687,7 +700,7 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
.ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?; .ok_or_else(|| anyhow::anyhow!("no admin account to subscribe: ipx user add <name> --admin"))?;
let doc = opml::OPML::from_str(&text) let doc = opml::OPML::from_str(&text)
.map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?; .map_err(|e| anyhow::anyhow!("{} is not OPML: {e}", file.display()))?;
let (added, had) = subscribe_opml(ctx, config_path, &doc, admin.id).await?; let (added, had) = subscribe_opml(ctx, &doc, admin.id).await?;
println!("subscribed {} to {added} feed(s); {had} already there", admin.name); println!("subscribed {} to {added} feed(s); {had} already there", admin.name);
Ok(()) Ok(())
} }
@@ -704,7 +717,6 @@ async fn import(ctx: &Ctx, config_path: &std::path::Path, file: &std::path::Path
/// before anything is touched: a 400 from the web, a message from the CLI. /// before anything is touched: a 400 from the web, a message from the CLI.
pub async fn subscribe_opml( pub async fn subscribe_opml(
ctx: &Ctx, ctx: &Ctx,
config_path: &std::path::Path,
doc: &opml::OPML, doc: &opml::OPML,
user_id: i64, user_id: i64,
) -> Result<(usize, usize)> { ) -> Result<(usize, usize)> {
@@ -751,8 +763,7 @@ pub async fn subscribe_opml(
ids.push(id); ids.push(id);
} }
if grew { if grew {
cfg.save(config_path)?; ctx.store_cfg(cfg).await?;
ctx.reload_cfg(config_path)?;
} }
let (mut added, mut had) = (0, 0); let (mut added, mut had) = (0, 0);
@@ -778,6 +789,40 @@ pub fn collect_outlines(outlines: &[opml::Outline], out: &mut Vec<(String, Strin
} }
} }
/// The configuration ipx runs with: config.toml for where things are and who may sign in, the
/// database for the feeds and the server settings (issue #18). The first time a database holds
/// neither, it takes them from config.toml, which is then cut down to the rest, the original kept
/// beside it as config.toml.pre-database.
async fn assemble_config(db: &db::Db, mut cfg: config::Config, path: &std::path::Path) -> Result<config::Config> {
for _ in 0..2 {
if let Some((stored, feeds)) = db.stored_config().await? {
if config::Config::file_holds_stored(path) {
tracing::warn!(
"config.toml still lists feeds or server settings; they are ignored, since the \
database holds them now. Change them in the web UI, or with ipx add and rm."
);
}
stored.apply(&mut cfg);
cfg.feeds = feeds;
return Ok(cfg);
}
if db.import_config(&cfg).await? {
if path.exists() {
let original = path.with_extension("toml.pre-database");
if !original.exists() {
std::fs::copy(path, &original)
.with_context(|| format!("keeping the original as {}", original.display()))?;
}
cfg.save_bootstrap(path)?;
}
tracing::info!(feeds = cfg.feeds.len(), "moved the feeds and server settings from config.toml into the database");
return Ok(cfg);
}
// Another ipx imported between our look and our insert; the next pass reads its copy.
}
anyhow::bail!("the database says it holds the configuration and then that it does not")
}
async fn copy_db(ctx: &Ctx, from: &std::path::Path) -> Result<()> { async fn copy_db(ctx: &Ctx, from: &std::path::Path) -> Result<()> {
anyhow::ensure!(from.exists(), "{} does not exist", from.display()); anyhow::ensure!(from.exists(), "{} does not exist", from.display());
let source = db::Db::open(&from.display().to_string()).await?; let source = db::Db::open(&from.display().to_string()).await?;
@@ -808,10 +853,10 @@ async fn export(ctx: &Ctx, file: &std::path::Path) -> Result<()> {
Ok(()) Ok(())
} }
async fn list(ctx: &Ctx, config_path: &std::path::Path) -> Result<()> { async fn list(ctx: &Ctx) -> Result<()> {
let cfg = ctx.cfg(); let cfg = ctx.cfg();
if cfg.feeds.is_empty() { if cfg.feeds.is_empty() {
println!("No feeds configured in {}", config_path.display()); println!("No feeds configured. Add one with `ipx add <url>`, or in the web UI.");
return Ok(()); return Ok(());
} }
for (id, feed) in &cfg.feeds { for (id, feed) in &cfg.feeds {
@@ -1670,6 +1715,32 @@ fn duration(secs: u64) -> String {
mod tests { mod tests {
use super::*; use super::*;
#[tokio::test]
async fn the_first_start_moves_the_configuration_in_and_trims_the_file() {
let dir = std::env::temp_dir().join(format!("ipx-assemble-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("config.toml");
std::fs::write(
&path,
"[general]\nschedule = \"every 2h\"\n[web]\ntoken = \"t\"\n[feeds.show]\nurl = \"http://x/show.xml\"\n",
)
.unwrap();
let db = db::Db::memory().await.unwrap();
let first = assemble_config(&db, config::Config::load(&path).unwrap(), &path).await.unwrap();
assert_eq!(first.feeds.keys().collect::<Vec<_>>(), ["show"]);
assert_eq!(first.general.schedule, "every 2h");
assert!(dir.join("config.toml.pre-database").exists(), "the original is kept");
assert!(!config::Config::file_holds_stored(&path), "and the file no longer lists them");
// The next start reads them from the database, the trimmed file notwithstanding.
let next = assemble_config(&db, config::Config::load(&path).unwrap(), &path).await.unwrap();
assert_eq!(next.feeds.keys().collect::<Vec<_>>(), ["show"]);
assert_eq!(next.general.schedule, "every 2h");
assert_eq!(next.web.token, "t");
std::fs::remove_dir_all(&dir).unwrap();
}
fn feed() -> config::Feed { fn feed() -> config::Feed {
// Whatever `ipx add` would write, which is the shape every code path sees. // Whatever `ipx add` would write, which is the shape every code path sees.
let mut cfg = config::Config::default(); let mut cfg = config::Config::default();

View File

@@ -16,7 +16,6 @@ use serde::Deserialize;
use tower::ServiceExt; use tower::ServiceExt;
use tower_http::services::ServeFile; use tower_http::services::ServeFile;
use serde::Serialize; use serde::Serialize;
use std::path::PathBuf;
use std::sync::Arc; use std::sync::Arc;
use tokio::sync::{broadcast, mpsc}; use tokio::sync::{broadcast, mpsc};
@@ -28,7 +27,6 @@ const COOKIE: &str = "ipx_token";
#[derive(Clone)] #[derive(Clone)]
pub struct WebState { pub struct WebState {
pub ctx: Arc<Ctx>, pub ctx: Arc<Ctx>,
pub config_path: PathBuf,
pub cmds: mpsc::Sender<Command>, pub cmds: mpsc::Sender<Command>,
pub events: broadcast::Sender<Event>, pub events: broadcast::Sender<Event>,
} }
@@ -1146,12 +1144,7 @@ async fn entry_page(
let filter = crate::db::Filter::parse(page.filter.as_deref().unwrap_or("all")); let filter = crate::db::Filter::parse(page.filter.as_deref().unwrap_or("all"));
let search = page.q.as_deref().map(str::trim).filter(|q| !q.is_empty()); let search = page.q.as_deref().map(str::trim).filter(|q| !q.is_empty());
let db = &state.ctx.db; let db = &state.ctx.db;
// Currently Listening keeps its own order, pinned or not: it is what you are part-way through. let order = crate::db::order_sql(page.sort.as_deref().unwrap_or("published"), page.dir.as_deref().unwrap_or("desc"));
let order = crate::db::order_sql(
page.sort.as_deref().unwrap_or("published"),
page.dir.as_deref().unwrap_or("desc"),
filter != crate::db::Filter::InProgress,
);
let mut rows = let mut rows =
db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order).await?; db.entries_in(user_id, feed, filter, search, page.offset, page.limit.clamp(1, 200), &order).await?;
let mut sanitizer = feed_sanitizer(); let mut sanitizer = feed_sanitizer();
@@ -1234,8 +1227,7 @@ async fn add_feed(
)); ));
} }
let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?; let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?;
cfg.save(&state.config_path)?; state.ctx.store_cfg(cfg).await?;
state.ctx.reload_cfg(&state.config_path)?;
state.ctx.db.subscribe(user.id, &id).await?; state.ctx.db.subscribe(user.id, &id).await?;
explicit_on_add(&state, user.id, &id, body.allow_explicit).await?; explicit_on_add(&state, user.id, &id, body.allow_explicit).await?;
scan_soon(&state, Some(id.clone())).await; scan_soon(&state, Some(id.clone())).await;
@@ -1383,8 +1375,7 @@ async fn patch_feed(
if let Some(v) = body.category { if let Some(v) = body.category {
feed.category = v.map(|s| s.trim().to_owned()).filter(|s| !s.is_empty()); feed.category = v.map(|s| s.trim().to_owned()).filter(|s| !s.is_empty());
} }
cfg.save(&state.config_path)?; state.ctx.store_cfg(cfg).await?;
state.ctx.reload_cfg(&state.config_path)?;
if url_changed { if url_changed {
// Refreshing a rotated auth token is the common case; entries and download history // Refreshing a rotated auth token is the common case; entries and download history
// are keyed by feed id, so they survive the change. // are keyed by feed id, so they survive the change.
@@ -1420,8 +1411,7 @@ async fn remove_feed(
state.ctx.db.drop_managed(&id).await?; state.ctx.db.drop_managed(&id).await?;
return Ok(StatusCode::NO_CONTENT); return Ok(StatusCode::NO_CONTENT);
} }
cfg.save(&state.config_path)?; state.ctx.store_cfg(cfg).await?;
state.ctx.reload_cfg(&state.config_path)?;
crate::retire_group(&state.ctx, &id).await?; crate::retire_group(&state.ctx, &id).await?;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }
@@ -1718,7 +1708,7 @@ async fn import_opml(
// file arrives as text, is read here, and is gone when the request ends. // file arrives as text, is read here, and is gone when the request ends.
let doc = opml::OPML::from_str(&body.xml) let doc = opml::OPML::from_str(&body.xml)
.map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?; .map_err(|e| ApiError::bad_request(format!("that is not an OPML file: {e}")))?;
let (added, already) = crate::subscribe_opml(&state.ctx, &state.config_path, &doc, user.id).await?; let (added, already) = crate::subscribe_opml(&state.ctx, &doc, user.id).await?;
if added > 0 { if added > 0 {
scan_soon(&state, None).await; scan_soon(&state, None).await;
} }
@@ -1792,8 +1782,7 @@ async fn patch_settings(
if let Some(v) = body.max_age_days { if let Some(v) = body.max_age_days {
cfg.general.max_age_days = v; cfg.general.max_age_days = v;
} }
cfg.save(&state.config_path)?; state.ctx.store_cfg(cfg).await?;
state.ctx.reload_cfg(&state.config_path)?;
Ok(StatusCode::NO_CONTENT) Ok(StatusCode::NO_CONTENT)
} }

View File

@@ -1299,20 +1299,3 @@ test('a pinned feed, even one from inside a folder, goes to the top of the list'
await expect(page.locator('.feed.pinned')).toHaveCount(0); await expect(page.locator('.feed.pinned')).toHaveCount(0);
await expect(page.locator('.feed.child', { hasText: name })).toHaveCount(1); await expect(page.locator('.feed.child', { hasText: name })).toHaveCount(1);
}); });
test('a pinned item goes to the top of its list, and back when unpinned', async ({ page }) => {
await page.locator('.feed', { hasText: 'Test Show' }).click();
await page.locator('.tabs button', { hasText: 'All' }).first().click();
await expect(page.locator('.ep').nth(1)).toBeVisible({ timeout: 20_000 });
const guids = () => page.locator('.ep').evaluateAll(rows => rows.map(r => r.dataset.guid));
const before = await guids();
const last = before[before.length - 1];
const row = page.locator(`.ep[data-guid="${last}"]`);
await row.locator('[data-a="flag"]').click();
await expect.poll(async () => (await guids())[0]).toBe(last);
// It is the server's order, so it holds on a reload.
await page.reload();
await expect.poll(async () => (await guids())[0]).toBe(last);
await page.locator(`.ep[data-guid="${last}"] [data-a="flag"]`).click();
await expect.poll(guids).toEqual(before);
});

View File

@@ -314,9 +314,7 @@ async function epAction(a: string, e, el, encId?: number){
const path=`/api/entries/${encodeURIComponent(e.feed_id)}/${encodeURIComponent(e.guid)}`; const path=`/api/entries/${encodeURIComponent(e.feed_id)}/${encodeURIComponent(e.guid)}`;
try{ try{
if(a==='play') play(e, encId!=null ? enc : undefined); if(a==='play') play(e, encId!=null ? enc : undefined);
// A pinned item sits at the top of its list (the server sorts it there), so the list is if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); }
// asked for again rather than the row redrawn where it stands.
if(a==='flag'){ e.flagged=!e.flagged; await api(path+'/flags',{method:'POST',body:JSON.stringify({flagged:e.flagged})}); redraw(); loadEntries(); }
if(a==='read'){ await setRead(e,!e.read); redraw(); } if(a==='read'){ await setRead(e,!e.read); redraw(); }
if(a==='get'){ if(a==='get'){
if(!enc) return; if(!enc) return;