Sync SQLite at checkpoints, not after every write (#135)
The load tests (#133) found a daemon on SQLite scanning 1.3s a feed of 20 items served from the same machine: 1,500 feeds would have taken about 50 minutes. On Tower's /tmp, a loop device, one synchronous 4 KB write takes 52ms, and SQLite was in WAL mode with its default synchronous=FULL, which waits for the disk on every commit; nearly every statement is a commit of its own, a feed's entries and enclosures being written a row at a time. Subscribing the first admin to an imported catalogue of 1,500 feeds took 39s, before the web server listened. Every SQLite connection now sets synchronous=NORMAL, SQLite's own advice for WAL: it syncs at checkpoints. A crash of ipx loses nothing; a power cut can lose the last transactions, and never corrupts the file. The same scan takes 27s and the same subscribing 0.75s. Postgres, which production runs on, is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- A database on SQLite, the default, no longer waits for the disk after nearly every write. On a slow disk a scan of 1,500 feeds took most of an hour and now takes half a minute. Postgres is unaffected.
|
||||
- The search box finds a feed in the Directory by name. It did nothing there.
|
||||
- Feeds that file themselves under Apple's Football or Soccer show under Sports, and those under the old Software How-To under Technology, instead of as categories of their own.
|
||||
- The download queue, in `ipx status`, `/api/status` and the dashboard, counts only what iPX will download on its own: older episodes beyond a show's limit, and files of feeds that do not download automatically, are listed but no longer counted as waiting.
|
||||
|
||||
Reference in New Issue
Block a user