diff --git a/docs/architecture.md b/docs/architecture.md index e96ead1..ffdfea9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -101,7 +101,7 @@ signals id, user_id, name, keywords signal_verdicts signal_id, feed_id, guid, hit PK (signal_id, feed_id, guid) ``` -A Signal (#254) is a feed someone makes from their own subscriptions. `signals::run`, a task of its +A Signal (#254, [signals.md](signals.md)) is a feed someone makes from their own subscriptions. `signals::run`, a task of its own beside DeadAir's, woken by a scan with something new or a Signal made or changed, asks Jev about each item of the last 30 days that has no verdict yet for one of its owner's Signals: one call an item, a noul question a Signal, on the owner's `users.typesafe_key`. Every answer is kept, diff --git a/docs/signals.md b/docs/signals.md new file mode 100644 index 0000000..e9a6dbd --- /dev/null +++ b/docs/signals.md @@ -0,0 +1,301 @@ +# Signals + +A Signal is a feed someone makes from their own subscriptions (#254). They give it a name and the +words to listen for (`formula 1`, `f1`). TypeSafe's Jev reads each item of the feeds they subscribe +to and picks out the ones about those words. Those items show in the Signal as well as in their +own feed. The rest stays noise, where it was. + +Each person brings their own TypeSafe key, entered in Settings. Nothing is judged for someone +without one. Jev is asked once for each item, on that person's key, with one question for each of +their Signals that has not heard about the item yet. + +## What someone sees + +- **The sidebar.** A Signals folder comes first in the feed list, under All Subscriptions (#255, + #256). It is closed or open as an OPML's folder is, remembered in each browser. A + at the right + of the folder's row opens New signal. Inside the folder are the person's Signals, each with its + words under its name and its unread count. +- **The folder's page** lists the Signals, as an OPML's page lists its feeds, with a + of its own. +- **A Signal's page** is an ordinary item list: All, Unread, Downloaded and Pinned, search and sort. + Its header says what it listens for, and that nothing is picked out without a key. Edit changes + the name or the words; Delete removes the Signal and leaves the items in their feeds. +- **Read state is the item's.** Reading an item in a Signal reads it in its feed, and the other way + round. The counts follow without a reload (#258). +- **Settings, TypeSafe key.** A password field. A key saves on leaving the field, and the bin + forgets it. The key is never sent back to the page, which only learns whether one is set. + +## How an item gets into a Signal + +```mermaid +sequenceDiagram + autonumber + participant W as Scan worker + participant J as Signals judge
(signals::run) + participant DB as Database + participant T as TypeSafe
api.typesafe.ai + participant P as The page + + W->>DB: new entries stored + W->>J: ctx.signals.notify_one() + Note over J: also woken by a new Signal,
new keywords, a key saved,
or every 5 minutes + J->>DB: unjudged(skip refused keys, 50) + DB-->>J: items, each with the Signals
that have not heard of it + loop each item + J->>T: POST /v1/systemone
one noul question per Signal + T-->>J: answers: noul 0..1 per question + J->>DB: record_verdicts (hit = noul >= 0.5) + end + J-->>P: signal_hits event (SSE), once a pass per person + P->>DB: GET /api/signals (counts),
and the open Signal's list +``` + +The judge is a task of its own beside DeadAir's (`crates/ipx/src/signals.rs`), not a job on the +scan worker, so a slow TypeSafe never holds up a scan. + +### Which items are asked about + +`Db::unjudged` finds, newest first, up to 50 items that are: + +- in a feed the Signal's owner subscribes to, +- published (or, with no date, first seen) in the last 30 days (`SIGNAL_DAYS`), +- not hidden from them by their block lists, +- not yet in `signal_verdicts` for that Signal, +- for someone with a key that TypeSafe has not refused since the daemon started. + +So a new Signal looks back a month, and after that sees each new item. A show subscribed to today +has its back catalogue skipped, apart from its last month. + +Rows for one item and one person are grouped, so an item is one call whatever the number of +Signals. A full batch means there may be more, and the next pass starts at once. Otherwise the +judge sleeps until woken or for 5 minutes. + +### Every answer is kept + +`signal_verdicts` records each answer, fitting or not. An item with no row has not been asked yet; +an item with a row is never asked again. So the cost is one call per item per person, once. +Changing a Signal's words deletes its verdicts, and everything in the window is asked again. + +## TypeSafe + +Jev is TypeSafe's System One model, already used to pick a feed's Directory category +(`feed::classify`). The address is `feed::JEV_URL`, `https://api.typesafe.ai/v1/systemone`. +Signals use the **noul** question type, which answers how likely a statement is true, from 0 to 1. +Questions in one call are judged independently and in parallel. + +### A request + +Sent with `Authorization: Bearer `, for an item and two of their Signals: + +```json +{ + "model": "jev-latest", + "state": "Feed: Pit Lane\nTitle: Monaco qualifying\n\nQualifying report from the Monaco Grand Prix.", + "questions": { + "signal_3": { + "type": "noul", + "instructions": "Is this item about any of these: formula 1, f1?", + "criteria": { + "true": "The item's subject is one of them, or plainly part of one.", + "false": "None of them is its subject: one is only mentioned in passing, or none is." + } + }, + "signal_4": { + "type": "noul", + "instructions": "Is this item about any of these: baking?", + "criteria": { + "true": "The item's subject is one of them, or plainly part of one.", + "false": "None of them is its subject: one is only mentioned in passing, or none is." + } + } + } +} +``` + +- `state` is text: the feed's title, the item's title, and its show notes with the HTML taken out, + cut to 2,000 characters (`TEXT_CHARS`). +- Each question is keyed `signal_`, and its answer comes back under the same key. +- `criteria` is an object saying what true and false mean. A sentence there is refused with a 422 + (#257). + +### Its answer + +What TypeSafe answered to the request above, on 2026-10-16: + +```json +{ + "model": "jev-1.13.0", + "answers": { + "signal_3": { "type": "noul", "noul": 0.96 }, + "signal_4": { "type": "noul", "noul": 0.02 } + }, + "usage": { "input_tokens": 389, "output_tokens": 40 } +} +``` + +0.5 or more is a fit (`THRESHOLD`): the item goes into Signal 3 and not Signal 4. A question +missing from the answer is left unjudged and asked again next pass. + +### When TypeSafe says no + +| TypeSafe answers | What the judge does | +|---|---| +| 401, 402, 403 | The key is refused. That person is passed over until they save a key or the daemon restarts, so a bad key is not sent the whole queue. Logged once: "TypeSafe refused a Signal key". | +| 400, 422 | The request itself is wrong, so every item's would be. The pass ends; the items wait for the next one. Logged: "TypeSafe refused a Signals request as malformed", with TypeSafe's reason. | +| anything else, or no answer | That item is left unjudged and asked again next pass. Logged: "Jev could not judge an item". | + +The error keeps TypeSafe's reply, cut to 500 characters, for example: + +``` +TypeSafe answered 422: {"detail":[{"type":"model_attributes_type","loc":["body","questions","3","noul","criteria"],"msg":"Input should be a valid dictionary or object to extract fields from","input":"Yes if the subject is one of them."}]} +``` + +## The API + +Every route is the signed-in person's own. A Signal of someone else's is not found, as one that +does not exist. + +### `GET /api/signals` + +```json +[ + { "id": 1, "name": "AI Development", "keywords": ["ai", "development", "programming"], "unread": 5 }, + { "id": 2, "name": "Apple", "keywords": ["apple", "mac", "iphone"], "unread": 0 } +] +``` + +`unread` counts the items Jev picked out that are still in a subscribed feed, not hidden, and +not read. + +### `POST /api/signals` + +```json +{ "name": "Formula 1", "keywords": ["formula 1", "f1"] } +``` + +Answers `{ "id": 3 }` and wakes the judge. A name is 1 to 100 characters. Between 1 and 50 words +are allowed, each up to 200 characters, kept once whatever their case. Anything else is a `400` +saying so. + +### `PATCH /api/signals/{id}` + +Either field or both: `{ "name": "F1" }`, `{ "keywords": ["f1", "grand prix"] }`. New keywords drop +the Signal's verdicts and wake the judge. Answers `204`, or `404` for a Signal that is not yours. + +### `DELETE /api/signals/{id}` + +Answers `204`. The Signal and its verdicts go; the items stay in their feeds. + +### `GET /api/signals/{id}/entries` + +The same query as a feed's list, `offset`, `limit`, `filter`, `q`, `sort`, `dir`: + +``` +GET /api/signals/3/entries?filter=unread&sort=published&dir=desc&limit=50 +``` + +```json +{ + "total": 1, + "entries": [ + { + "guid": "sig-1", "feed_id": "pit-lane", "title": "Monaco qualifying", + "link": "https://pitlane.example/monaco", "published": 1792170000, + "description": "

Qualifying report…

", "read": false, "flagged": false, + "image": null, "duration": null, "episode": null, "season": null, "position": 0, + "enclosures": [] + } + ] +} +``` + +Reading or pinning one is the item's own route, `POST /api/entries/{feed_id}/{guid}/flags`, as +anywhere else. + +### `PATCH /api/me`, the key + +```json +{ "typesafe_key": "ts_…" } +``` + +A string saves the key and lets the judge try it again. `null` or an empty string forgets it. A +key is up to 500 printable characters. `GET /api/me` says `"typesafe_key_set": true` and nothing +more. + +### The event + +Once a pass, for each person it found items for, the judge sends this on the event stream +(`/api/events`): + +```json +{ "ev": "signal_hits", "user": 2, "hits": 3 } +``` + +A page with Signals fetches `/api/signals` again, and the open Signal's list. + +## Data + +```mermaid +erDiagram + users ||--o{ signals : owns + signals ||--o{ signal_verdicts : "has heard of" + entries ||--o{ signal_verdicts : "judged in" + users { + bigint id + text typesafe_key "NULL: no key" + } + signals { + bigint id + bigint user_id + text name + text keywords "JSON array" + bigint created + } + signal_verdicts { + bigint signal_id "PK" + text feed_id "PK" + text guid "PK" + bool hit + } +``` + +- `signals` and `signal_verdicts` are created from their entities like every table + (`crates/ipx/src/entity.rs`). `users.typesafe_key` was added to an existing table, so + `create_missing` adds the column when it is missing. +- A Signal's list is `entries_where` with + `EXISTS (SELECT 1 FROM signal_verdicts v WHERE v.signal_id = $n AND v.feed_id = e.feed_id AND v.guid = e.guid AND v.hit)` + on top of the subscribed-feeds scope. So filters, search, sort, block lists and read state all + work as for All Subscriptions. +- A verdict for a Signal deleted while Jev was being asked is not stored. Postgres's foreign key + would refuse it, and SQLite would keep it for nothing. + +## Watching it + +| | | +|---|---| +| Traces | A `signals` span for each pass (`entries`), and a `jev` span for each call (`user`, `signal_count`, `http.response.status_code`, and TypeSafe's reply as `error` when it refuses). | +| Log, counted | `ev="signal"`, "an item fits a Signal", for each item with a fit: `user`, `feed`, `new` (how many of their Signals it fits). | +| Log, warnings | The three in the table above. | +| Events | `signal_hits` on the socket and `/api/events`. | + +## Limits + +- **The key is kept in plain text** in `users.typesafe_key`, because every call needs it whole. It + should be encrypted with a server key if the database is ever shared with anything else. +- **Refused keys are remembered in memory**, so a restart tries each once again. +- **Keywords changed while an item is being judged** can leave that one item's verdict from the old + words. +- **There is no meter** of how many calls a key has made. It is one call per item in the person's + feeds in the window: a person with 1,500 feeds busy enough to publish 3,000 items a month spends + about 3,000 calls the first month and then as many as come in. +- **Only the last 30 days** are ever judged, by publish date. + +## Testing + +- `cargo test -p ipx`: the request's shape and the threshold (`signals::tests`), and the queue, + verdicts, ownership, unread counts and new keywords (`db::tests::a_signal_shows_what_jev_said_fits_and_only_to_its_owner`), + on SQLite and Postgres both. +- `node --test tests/page-unit.js`: a Signal's place id and its list's address. +- `npx playwright test -g Signal`: makes a Signal from the folder's +, and checks that the matching + item shows and that reading it reads it in its feed and clears the badge. TypeSafe is stood in for + by the fixture server's `/jev` (`IPX_TYPESAFE_URL`), which refuses the shape TypeSafe refuses, and + `/signal.xml` is dated today, since only the last 30 days are judged.