docs/signals.md: how Signals work, with TypeSafe's requests and answers and the API

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:57:05 +00:00
parent 1ae91170fd
commit f8371f255d
2 changed files with 302 additions and 1 deletions
--- a/docs/signals.md
+++ b/docs/signals.md
@@ -0,0 +1,301 @@
+# Signals
+
+A Signal is a feed someone makes from their own subscriptions (#254). They give it a name and the
+words to listen for (`formula 1`, `f1`). TypeSafe's Jev reads each item of the feeds they subscribe
+to and picks out the ones about those words. Those items show in the Signal as well as in their
+own feed. The rest stays noise, where it was.
+
+Each person brings their own TypeSafe key, entered in Settings. Nothing is judged for someone
+without one. Jev is asked once for each item, on that person's key, with one question for each of
+their Signals that has not heard about the item yet.
+
+## What someone sees
+
+- **The sidebar.** A Signals folder comes first in the feed list, under All Subscriptions (#255,
+  #256). It is closed or open as an OPML's folder is, remembered in each browser. A + at the right
+  of the folder's row opens New signal. Inside the folder are the person's Signals, each with its
+  words under its name and its unread count.
+- **The folder's page** lists the Signals, as an OPML's page lists its feeds, with a + of its own.
+- **A Signal's page** is an ordinary item list: All, Unread, Downloaded and Pinned, search and sort.
+  Its header says what it listens for, and that nothing is picked out without a key. Edit changes
+  the name or the words; Delete removes the Signal and leaves the items in their feeds.
+- **Read state is the item's.** Reading an item in a Signal reads it in its feed, and the other way
+  round. The counts follow without a reload (#258).
+- **Settings, TypeSafe key.** A password field. A key saves on leaving the field, and the bin
+  forgets it. The key is never sent back to the page, which only learns whether one is set.
+
+## How an item gets into a Signal
+
+```mermaid
+sequenceDiagram
+    autonumber
+    participant W as Scan worker
+    participant J as Signals judge<br/>(signals::run)
+    participant DB as Database
+    participant T as TypeSafe<br/>api.typesafe.ai
+    participant P as The page
+
+    W->>DB: new entries stored
+    W->>J: ctx.signals.notify_one()
+    Note over J: also woken by a new Signal,<br/>new keywords, a key saved,<br/>or every 5 minutes
+    J->>DB: unjudged(skip refused keys, 50)
+    DB-->>J: items, each with the Signals<br/>that have not heard of it
+    loop each item
+        J->>T: POST /v1/systemone<br/>one noul question per Signal
+        T-->>J: answers: noul 0..1 per question
+        J->>DB: record_verdicts (hit = noul >= 0.5)
+    end
+    J-->>P: signal_hits event (SSE), once a pass per person
+    P->>DB: GET /api/signals (counts),<br/>and the open Signal's list
+```
+
+The judge is a task of its own beside DeadAir's (`crates/ipx/src/signals.rs`), not a job on the
+scan worker, so a slow TypeSafe never holds up a scan.
+
+### Which items are asked about
+
+`Db::unjudged` finds, newest first, up to 50 items that are:
+
+- in a feed the Signal's owner subscribes to,
+- published (or, with no date, first seen) in the last 30 days (`SIGNAL_DAYS`),
+- not hidden from them by their block lists,
+- not yet in `signal_verdicts` for that Signal,
+- for someone with a key that TypeSafe has not refused since the daemon started.
+
+So a new Signal looks back a month, and after that sees each new item. A show subscribed to today
+has its back catalogue skipped, apart from its last month.
+
+Rows for one item and one person are grouped, so an item is one call whatever the number of
+Signals. A full batch means there may be more, and the next pass starts at once. Otherwise the
+judge sleeps until woken or for 5 minutes.
+
+### Every answer is kept
+
+`signal_verdicts` records each answer, fitting or not. An item with no row has not been asked yet;
+an item with a row is never asked again. So the cost is one call per item per person, once.
+Changing a Signal's words deletes its verdicts, and everything in the window is asked again.
+
+## TypeSafe
+
+Jev is TypeSafe's System One model, already used to pick a feed's Directory category
+(`feed::classify`). The address is `feed::JEV_URL`, `https://api.typesafe.ai/v1/systemone`.
+Signals use the **noul** question type, which answers how likely a statement is true, from 0 to 1.
+Questions in one call are judged independently and in parallel.
+
+### A request
+
+Sent with `Authorization: Bearer <the person's key>`, for an item and two of their Signals:
+
+```json
+{
+  "model": "jev-latest",
+  "state": "Feed: Pit Lane\nTitle: Monaco qualifying\n\nQualifying report from the Monaco Grand Prix.",
+  "questions": {
+    "signal_3": {
+      "type": "noul",
+      "instructions": "Is this item about any of these: formula 1, f1?",
+      "criteria": {
+        "true": "The item's subject is one of them, or plainly part of one.",
+        "false": "None of them is its subject: one is only mentioned in passing, or none is."
+      }
+    },
+    "signal_4": {
+      "type": "noul",
+      "instructions": "Is this item about any of these: baking?",
+      "criteria": {
+        "true": "The item's subject is one of them, or plainly part of one.",
+        "false": "None of them is its subject: one is only mentioned in passing, or none is."
+      }
+    }
+  }
+}
+```
+
+- `state` is text: the feed's title, the item's title, and its show notes with the HTML taken out,
+  cut to 2,000 characters (`TEXT_CHARS`).
+- Each question is keyed `signal_<id>`, and its answer comes back under the same key.
+- `criteria` is an object saying what true and false mean. A sentence there is refused with a 422
+  (#257).
+
+### Its answer
+
+What TypeSafe answered to the request above, on 2026-10-16:
+
+```json
+{
+  "model": "jev-1.13.0",
+  "answers": {
+    "signal_3": { "type": "noul", "noul": 0.96 },
+    "signal_4": { "type": "noul", "noul": 0.02 }
+  },
+  "usage": { "input_tokens": 389, "output_tokens": 40 }
+}
+```
+
+0.5 or more is a fit (`THRESHOLD`): the item goes into Signal 3 and not Signal 4. A question
+missing from the answer is left unjudged and asked again next pass.
+
+### When TypeSafe says no
+
+| TypeSafe answers | What the judge does |
+|---|---|
+| 401, 402, 403 | The key is refused. That person is passed over until they save a key or the daemon restarts, so a bad key is not sent the whole queue. Logged once: "TypeSafe refused a Signal key". |
+| 400, 422 | The request itself is wrong, so every item's would be. The pass ends; the items wait for the next one. Logged: "TypeSafe refused a Signals request as malformed", with TypeSafe's reason. |
+| anything else, or no answer | That item is left unjudged and asked again next pass. Logged: "Jev could not judge an item". |
+
+The error keeps TypeSafe's reply, cut to 500 characters, for example:
+
+```
+TypeSafe answered 422: {"detail":[{"type":"model_attributes_type","loc":["body","questions","3","noul","criteria"],"msg":"Input should be a valid dictionary or object to extract fields from","input":"Yes if the subject is one of them."}]}
+```
+
+## The API
+
+Every route is the signed-in person's own. A Signal of someone else's is not found, as one that
+does not exist.
+
+### `GET /api/signals`
+
+```json
+[
+  { "id": 1, "name": "AI Development", "keywords": ["ai", "development", "programming"], "unread": 5 },
+  { "id": 2, "name": "Apple", "keywords": ["apple", "mac", "iphone"], "unread": 0 }
+]
+```
+
+`unread` counts the items Jev picked out that are still in a subscribed feed, not hidden, and
+not read.
+
+### `POST /api/signals`
+
+```json
+{ "name": "Formula 1", "keywords": ["formula 1", "f1"] }
+```
+
+Answers `{ "id": 3 }` and wakes the judge. A name is 1 to 100 characters. Between 1 and 50 words
+are allowed, each up to 200 characters, kept once whatever their case. Anything else is a `400`
+saying so.
+
+### `PATCH /api/signals/{id}`
+
+Either field or both: `{ "name": "F1" }`, `{ "keywords": ["f1", "grand prix"] }`. New keywords drop
+the Signal's verdicts and wake the judge. Answers `204`, or `404` for a Signal that is not yours.
+
+### `DELETE /api/signals/{id}`
+
+Answers `204`. The Signal and its verdicts go; the items stay in their feeds.
+
+### `GET /api/signals/{id}/entries`
+
+The same query as a feed's list, `offset`, `limit`, `filter`, `q`, `sort`, `dir`:
+
+```
+GET /api/signals/3/entries?filter=unread&sort=published&dir=desc&limit=50
+```
+
+```json
+{
+  "total": 1,
+  "entries": [
+    {
+      "guid": "sig-1", "feed_id": "pit-lane", "title": "Monaco qualifying",
+      "link": "https://pitlane.example/monaco", "published": 1792170000,
+      "description": "<p>Qualifying report…</p>", "read": false, "flagged": false,
+      "image": null, "duration": null, "episode": null, "season": null, "position": 0,
+      "enclosures": []
+    }
+  ]
+}
+```
+
+Reading or pinning one is the item's own route, `POST /api/entries/{feed_id}/{guid}/flags`, as
+anywhere else.
+
+### `PATCH /api/me`, the key
+
+```json
+{ "typesafe_key": "ts_…" }
+```
+
+A string saves the key and lets the judge try it again. `null` or an empty string forgets it. A
+key is up to 500 printable characters. `GET /api/me` says `"typesafe_key_set": true` and nothing
+more.
+
+### The event
+
+Once a pass, for each person it found items for, the judge sends this on the event stream
+(`/api/events`):
+
+```json
+{ "ev": "signal_hits", "user": 2, "hits": 3 }
+```
+
+A page with Signals fetches `/api/signals` again, and the open Signal's list.
+
+## Data
+
+```mermaid
+erDiagram
+    users ||--o{ signals : owns
+    signals ||--o{ signal_verdicts : "has heard of"
+    entries ||--o{ signal_verdicts : "judged in"
+    users {
+        bigint id
+        text typesafe_key "NULL: no key"
+    }
+    signals {
+        bigint id
+        bigint user_id
+        text name
+        text keywords "JSON array"
+        bigint created
+    }
+    signal_verdicts {
+        bigint signal_id "PK"
+        text feed_id "PK"
+        text guid "PK"
+        bool hit
+    }
+```
+
+- `signals` and `signal_verdicts` are created from their entities like every table
+  (`crates/ipx/src/entity.rs`). `users.typesafe_key` was added to an existing table, so
+  `create_missing` adds the column when it is missing.
+- A Signal's list is `entries_where` with
+  `EXISTS (SELECT 1 FROM signal_verdicts v WHERE v.signal_id = $n AND v.feed_id = e.feed_id AND v.guid = e.guid AND v.hit)`
+  on top of the subscribed-feeds scope. So filters, search, sort, block lists and read state all
+  work as for All Subscriptions.
+- A verdict for a Signal deleted while Jev was being asked is not stored. Postgres's foreign key
+  would refuse it, and SQLite would keep it for nothing.
+
+## Watching it
+
+| | |
+|---|---|
+| Traces | A `signals` span for each pass (`entries`), and a `jev` span for each call (`user`, `signal_count`, `http.response.status_code`, and TypeSafe's reply as `error` when it refuses). |
+| Log, counted | `ev="signal"`, "an item fits a Signal", for each item with a fit: `user`, `feed`, `new` (how many of their Signals it fits). |
+| Log, warnings | The three in the table above. |
+| Events | `signal_hits` on the socket and `/api/events`. |
+
+## Limits
+
+- **The key is kept in plain text** in `users.typesafe_key`, because every call needs it whole. It
+  should be encrypted with a server key if the database is ever shared with anything else.
+- **Refused keys are remembered in memory**, so a restart tries each once again.
+- **Keywords changed while an item is being judged** can leave that one item's verdict from the old
+  words.
+- **There is no meter** of how many calls a key has made. It is one call per item in the person's
+  feeds in the window: a person with 1,500 feeds busy enough to publish 3,000 items a month spends
+  about 3,000 calls the first month and then as many as come in.
+- **Only the last 30 days** are ever judged, by publish date.
+
+## Testing
+
+- `cargo test -p ipx`: the request's shape and the threshold (`signals::tests`), and the queue,
+  verdicts, ownership, unread counts and new keywords (`db::tests::a_signal_shows_what_jev_said_fits_and_only_to_its_owner`),
+  on SQLite and Postgres both.
+- `node --test tests/page-unit.js`: a Signal's place id and its list's address.
+- `npx playwright test -g Signal`: makes a Signal from the folder's +, and checks that the matching
+  item shows and that reading it reads it in its feed and clears the badge. TypeSafe is stood in for
+  by the fixture server's `/jev` (`IPX_TYPESAFE_URL`), which refuses the shape TypeSafe refuses, and
+  `/signal.xml` is dated today, since only the last 30 days are judged.