302 lines
12 KiB
Markdown
302 lines
12 KiB
Markdown
# Signals
|
|
|
|
A Signal is a feed someone makes from their own subscriptions (#254). They give it a name and the
|
|
words to listen for (`formula 1`, `f1`). TypeSafe's Jev reads each item of the feeds they subscribe
|
|
to and picks out the ones about those words. Those items show in the Signal as well as in their
|
|
own feed. The rest stays noise, where it was.
|
|
|
|
Each person brings their own TypeSafe key, entered in Settings. Nothing is judged for someone
|
|
without one. Jev is asked once for each item, on that person's key, with one question for each of
|
|
their Signals that has not heard about the item yet.
|
|
|
|
## What someone sees
|
|
|
|
- **The sidebar.** A Signals folder comes first in the feed list, under All Subscriptions (#255,
|
|
#256). It is closed or open as an OPML's folder is, remembered in each browser. A + at the right
|
|
of the folder's row opens New signal. Inside the folder are the person's Signals, each with its
|
|
words under its name and its unread count.
|
|
- **The folder's page** lists the Signals, as an OPML's page lists its feeds, with a + of its own.
|
|
- **A Signal's page** is an ordinary item list: All, Unread, Downloaded and Pinned, search and sort.
|
|
Its header says what it listens for, and that nothing is picked out without a key. Edit changes
|
|
the name or the words; Delete removes the Signal and leaves the items in their feeds.
|
|
- **Read state is the item's.** Reading an item in a Signal reads it in its feed, and the other way
|
|
round. The counts follow without a reload (#258).
|
|
- **Settings, TypeSafe key.** A password field. A key saves on leaving the field, and the bin
|
|
forgets it. The key is never sent back to the page, which only learns whether one is set.
|
|
|
|
## How an item gets into a Signal
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
autonumber
|
|
participant W as Scan worker
|
|
participant J as Signals judge<br/>(signals::run)
|
|
participant DB as Database
|
|
participant T as TypeSafe<br/>api.typesafe.ai
|
|
participant P as The page
|
|
|
|
W->>DB: new entries stored
|
|
W->>J: ctx.signals.notify_one()
|
|
Note over J: also woken by a new Signal,<br/>new keywords, a key saved,<br/>or every 5 minutes
|
|
J->>DB: unjudged(skip refused keys, 50)
|
|
DB-->>J: items, each with the Signals<br/>that have not heard of it
|
|
loop each item
|
|
J->>T: POST /v1/systemone<br/>one noul question per Signal
|
|
T-->>J: answers: noul 0..1 per question
|
|
J->>DB: record_verdicts (hit = noul >= 0.5)
|
|
end
|
|
J-->>P: signal_hits event (SSE), once a pass per person
|
|
P->>DB: GET /api/signals (counts),<br/>and the open Signal's list
|
|
```
|
|
|
|
The judge is a task of its own beside DeadAir's (`crates/ipx/src/signals.rs`), not a job on the
|
|
scan worker, so a slow TypeSafe never holds up a scan.
|
|
|
|
### Which items are asked about
|
|
|
|
`Db::unjudged` finds, newest first, up to 50 items that are:
|
|
|
|
- in a feed the Signal's owner subscribes to,
|
|
- published (or, with no date, first seen) in the last 30 days (`SIGNAL_DAYS`),
|
|
- not hidden from them by their block lists,
|
|
- not yet in `signal_verdicts` for that Signal,
|
|
- for someone with a key that TypeSafe has not refused since the daemon started.
|
|
|
|
So a new Signal looks back a month, and after that sees each new item. A show subscribed to today
|
|
has its back catalogue skipped, apart from its last month.
|
|
|
|
Rows for one item and one person are grouped, so an item is one call whatever the number of
|
|
Signals. A full batch means there may be more, and the next pass starts at once. Otherwise the
|
|
judge sleeps until woken or for 5 minutes.
|
|
|
|
### Every answer is kept
|
|
|
|
`signal_verdicts` records each answer, fitting or not. An item with no row has not been asked yet;
|
|
an item with a row is never asked again. So the cost is one call per item per person, once.
|
|
Changing a Signal's words deletes its verdicts, and everything in the window is asked again.
|
|
|
|
## TypeSafe
|
|
|
|
Jev is TypeSafe's System One model, already used to pick a feed's Directory category
|
|
(`feed::classify`). The address is `feed::JEV_URL`, `https://api.typesafe.ai/v1/systemone`.
|
|
Signals use the **noul** question type, which answers how likely a statement is true, from 0 to 1.
|
|
Questions in one call are judged independently and in parallel.
|
|
|
|
### A request
|
|
|
|
Sent with `Authorization: Bearer <the person's key>`, for an item and two of their Signals:
|
|
|
|
```json
|
|
{
|
|
"model": "jev-latest",
|
|
"state": "Feed: Pit Lane\nTitle: Monaco qualifying\n\nQualifying report from the Monaco Grand Prix.",
|
|
"questions": {
|
|
"signal_3": {
|
|
"type": "noul",
|
|
"instructions": "Is this item about any of these: formula 1, f1?",
|
|
"criteria": {
|
|
"true": "The item's subject is one of them, or plainly part of one.",
|
|
"false": "None of them is its subject: one is only mentioned in passing, or none is."
|
|
}
|
|
},
|
|
"signal_4": {
|
|
"type": "noul",
|
|
"instructions": "Is this item about any of these: baking?",
|
|
"criteria": {
|
|
"true": "The item's subject is one of them, or plainly part of one.",
|
|
"false": "None of them is its subject: one is only mentioned in passing, or none is."
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
- `state` is text: the feed's title, the item's title, and its show notes with the HTML taken out,
|
|
cut to 2,000 characters (`TEXT_CHARS`).
|
|
- Each question is keyed `signal_<id>`, and its answer comes back under the same key.
|
|
- `criteria` is an object saying what true and false mean. A sentence there is refused with a 422
|
|
(#257).
|
|
|
|
### Its answer
|
|
|
|
What TypeSafe answered to the request above, on 2026-10-16:
|
|
|
|
```json
|
|
{
|
|
"model": "jev-1.13.0",
|
|
"answers": {
|
|
"signal_3": { "type": "noul", "noul": 0.96 },
|
|
"signal_4": { "type": "noul", "noul": 0.02 }
|
|
},
|
|
"usage": { "input_tokens": 389, "output_tokens": 40 }
|
|
}
|
|
```
|
|
|
|
0.5 or more is a fit (`THRESHOLD`): the item goes into Signal 3 and not Signal 4. A question
|
|
missing from the answer is left unjudged and asked again next pass.
|
|
|
|
### When TypeSafe says no
|
|
|
|
| TypeSafe answers | What the judge does |
|
|
|---|---|
|
|
| 401, 402, 403 | The key is refused. That person is passed over until they save a key or the daemon restarts, so a bad key is not sent the whole queue. Logged once: "TypeSafe refused a Signal key". |
|
|
| 400, 422 | The request itself is wrong, so every item's would be. The pass ends; the items wait for the next one. Logged: "TypeSafe refused a Signals request as malformed", with TypeSafe's reason. |
|
|
| anything else, or no answer | That item is left unjudged and asked again next pass. Logged: "Jev could not judge an item". |
|
|
|
|
The error keeps TypeSafe's reply, cut to 500 characters, for example:
|
|
|
|
```
|
|
TypeSafe answered 422: {"detail":[{"type":"model_attributes_type","loc":["body","questions","3","noul","criteria"],"msg":"Input should be a valid dictionary or object to extract fields from","input":"Yes if the subject is one of them."}]}
|
|
```
|
|
|
|
## The API
|
|
|
|
Every route is the signed-in person's own. A Signal of someone else's is not found, as one that
|
|
does not exist.
|
|
|
|
### `GET /api/signals`
|
|
|
|
```json
|
|
[
|
|
{ "id": 1, "name": "AI Development", "keywords": ["ai", "development", "programming"], "unread": 5 },
|
|
{ "id": 2, "name": "Apple", "keywords": ["apple", "mac", "iphone"], "unread": 0 }
|
|
]
|
|
```
|
|
|
|
`unread` counts the items Jev picked out that are still in a subscribed feed, not hidden, and
|
|
not read.
|
|
|
|
### `POST /api/signals`
|
|
|
|
```json
|
|
{ "name": "Formula 1", "keywords": ["formula 1", "f1"] }
|
|
```
|
|
|
|
Answers `{ "id": 3 }` and wakes the judge. A name is 1 to 100 characters. Between 1 and 50 words
|
|
are allowed, each up to 200 characters, kept once whatever their case. Anything else is a `400`
|
|
saying so.
|
|
|
|
### `PATCH /api/signals/{id}`
|
|
|
|
Either field or both: `{ "name": "F1" }`, `{ "keywords": ["f1", "grand prix"] }`. New keywords drop
|
|
the Signal's verdicts and wake the judge. Answers `204`, or `404` for a Signal that is not yours.
|
|
|
|
### `DELETE /api/signals/{id}`
|
|
|
|
Answers `204`. The Signal and its verdicts go; the items stay in their feeds.
|
|
|
|
### `GET /api/signals/{id}/entries`
|
|
|
|
The same query as a feed's list, `offset`, `limit`, `filter`, `q`, `sort`, `dir`:
|
|
|
|
```
|
|
GET /api/signals/3/entries?filter=unread&sort=published&dir=desc&limit=50
|
|
```
|
|
|
|
```json
|
|
{
|
|
"total": 1,
|
|
"entries": [
|
|
{
|
|
"guid": "sig-1", "feed_id": "pit-lane", "title": "Monaco qualifying",
|
|
"link": "https://pitlane.example/monaco", "published": 1792170000,
|
|
"description": "<p>Qualifying report…</p>", "read": false, "flagged": false,
|
|
"image": null, "duration": null, "episode": null, "season": null, "position": 0,
|
|
"enclosures": []
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Reading or pinning one is the item's own route, `POST /api/entries/{feed_id}/{guid}/flags`, as
|
|
anywhere else.
|
|
|
|
### `PATCH /api/me`, the key
|
|
|
|
```json
|
|
{ "typesafe_key": "ts_…" }
|
|
```
|
|
|
|
A string saves the key and lets the judge try it again. `null` or an empty string forgets it. A
|
|
key is up to 500 printable characters. `GET /api/me` says `"typesafe_key_set": true` and nothing
|
|
more.
|
|
|
|
### The event
|
|
|
|
Once a pass, for each person it found items for, the judge sends this on the event stream
|
|
(`/api/events`):
|
|
|
|
```json
|
|
{ "ev": "signal_hits", "user": 2, "hits": 3 }
|
|
```
|
|
|
|
A page with Signals fetches `/api/signals` again, and the open Signal's list.
|
|
|
|
## Data
|
|
|
|
```mermaid
|
|
erDiagram
|
|
users ||--o{ signals : owns
|
|
signals ||--o{ signal_verdicts : "has heard of"
|
|
entries ||--o{ signal_verdicts : "judged in"
|
|
users {
|
|
bigint id
|
|
text typesafe_key "NULL: no key"
|
|
}
|
|
signals {
|
|
bigint id
|
|
bigint user_id
|
|
text name
|
|
text keywords "JSON array"
|
|
bigint created
|
|
}
|
|
signal_verdicts {
|
|
bigint signal_id "PK"
|
|
text feed_id "PK"
|
|
text guid "PK"
|
|
bool hit
|
|
}
|
|
```
|
|
|
|
- `signals` and `signal_verdicts` are created from their entities like every table
|
|
(`crates/ipx/src/entity.rs`). `users.typesafe_key` was added to an existing table, so
|
|
`create_missing` adds the column when it is missing.
|
|
- A Signal's list is `entries_where` with
|
|
`EXISTS (SELECT 1 FROM signal_verdicts v WHERE v.signal_id = $n AND v.feed_id = e.feed_id AND v.guid = e.guid AND v.hit)`
|
|
on top of the subscribed-feeds scope. So filters, search, sort, block lists and read state all
|
|
work as for All Subscriptions.
|
|
- A verdict for a Signal deleted while Jev was being asked is not stored. Postgres's foreign key
|
|
would refuse it, and SQLite would keep it for nothing.
|
|
|
|
## Watching it
|
|
|
|
| | |
|
|
|---|---|
|
|
| Traces | A `signals` span for each pass (`entries`), and a `jev` span for each call (`user`, `signal_count`, `http.response.status_code`, and TypeSafe's reply as `error` when it refuses). |
|
|
| Log, counted | `ev="signal"`, "an item fits a Signal", for each item with a fit: `user`, `feed`, `new` (how many of their Signals it fits). |
|
|
| Log, warnings | The three in the table above. |
|
|
| Events | `signal_hits` on the socket and `/api/events`. |
|
|
|
|
## Limits
|
|
|
|
- **The key is kept in plain text** in `users.typesafe_key`, because every call needs it whole. It
|
|
should be encrypted with a server key if the database is ever shared with anything else.
|
|
- **Refused keys are remembered in memory**, so a restart tries each once again.
|
|
- **Keywords changed while an item is being judged** can leave that one item's verdict from the old
|
|
words.
|
|
- **There is no meter** of how many calls a key has made. It is one call per item in the person's
|
|
feeds in the window: a person with 1,500 feeds busy enough to publish 3,000 items a month spends
|
|
about 3,000 calls the first month and then as many as come in.
|
|
- **Only the last 30 days** are ever judged, by publish date.
|
|
|
|
## Testing
|
|
|
|
- `cargo test -p ipx`: the request's shape and the threshold (`signals::tests`), and the queue,
|
|
verdicts, ownership, unread counts and new keywords (`db::tests::a_signal_shows_what_jev_said_fits_and_only_to_its_owner`),
|
|
on SQLite and Postgres both.
|
|
- `node --test tests/page-unit.js`: a Signal's place id and its list's address.
|
|
- `npx playwright test -g Signal`: makes a Signal from the folder's +, and checks that the matching
|
|
item shows and that reading it reads it in its feed and clears the badge. TypeSafe is stood in for
|
|
by the fixture server's `/jev` (`IPX_TYPESAFE_URL`), which refuses the shape TypeSafe refuses, and
|
|
`/signal.xml` is dated today, since only the last 30 days are judged.
|