ipx/docs/signals.md

# Signals

A Signal is a feed someone makes from their own subscriptions (#254). They give it a name, the
words to listen for (`formula 1`, `f1`) and, when those are not enough, a sentence saying what they
mean (#259). TypeSafe's Jev reads each item of the feeds they subscribe to and picks out the ones
about that subject: the name, the words and the sentence together, not any one word. Those items show in the Signal as well as in their
own feed. The rest stays noise, where it was.

Whose TypeSafe key pays for Jev is the admin's choice (#271), under Admin, Server, **Signals for
everyone, on this server's TypeSafe key** (`signals_on_server`):

- **On**, with the server's key in `TYPESAFE_KEY`: everyone has Signals, with nothing to set up.
  Every item is asked about on the server's key, and keys people saved are kept but not used.
- **Off**, the default, or on with no `TYPESAFE_KEY`: each person turns Signals on in Settings
  with a key of their own. Nothing is judged for someone without one, and their Signals folder is
  not shown.

Jev is asked once for each item, on that key, with one question for each of the person's Signals
that has not heard about the item yet.

## What someone sees

- **The sidebar.** Once Signals are on for someone, a Signals folder comes first in the feed list, under All Subscriptions (#255,
  #256). It is closed or open as an OPML's folder is, remembered in each browser. Its row is like
  any folder's (#265); New signal is the + on the folder's page. Inside the folder are the
  person's Signals, each with its items and unread counted under its name.
- **An icon of its own** (#261), chosen in the Signal's dialog from all of Font Awesome Free 7.3.1,
  the set the page's own icons come from: 2,883 icons, solid, regular and brands, searched by name
  and Font Awesome's search terms ("ai" finds the robot). Without one, a Signal shows its initials.
  **Auto** asks Jev to choose (#263), and a Signal made without an icon gets Jev's pick by
  itself, a moment after it is made. Every Signal's tile is in the accent tint the folder has.
- **Each Signal's row** says "12 items · 3 unread", as a feed's row counts its own.
- **The folder's page** lists the Signals, with the + for a new one.
- **A Signal's page** is an ordinary item list: All, Unread, Downloaded and Pinned, search and sort.
  Its header says what it listens for (the sentence, or the words), and, on people's own keys,
  that nothing is picked out without one. Edit changes the name, the words or the sentence; Delete removes the Signal
  and leaves the items in their feeds.
- **Read state is the item's.** Reading an item in a Signal reads it in its feed, and the other way
  round. The counts follow without a reload (#258).
- **Settings, Signals.** On the server's key, a line saying the server pays. On people's own, a
  Signals switch: turning it on shows a password field for the key, which saves on leaving the
  field once TypeSafe has taken it (one short question on that key); a key TypeSafe refuses is
  not saved, and the page says so. Turning the switch off, or the bin, forgets the key and hides
  the folder; the Signals are kept. The key is never sent back to the page, which only learns
  whether one is set.

## How an item gets into a Signal

```mermaid
sequenceDiagram
    autonumber
    participant W as Scan worker
    participant J as Signals judge<br/>(signals::run)
    participant DB as Database
    participant T as TypeSafe<br/>api.typesafe.ai
    participant P as The page

    W->>DB: new entries stored
    W->>J: ctx.signals.notify_one()
    Note over J: also woken by a new Signal,<br/>new keywords, a key saved,<br/>or every 5 minutes
    J->>DB: unjudged(skip refused keys, 50)
    DB-->>J: items, each with the Signals<br/>that have not heard of it
    loop each item
        J->>T: POST /v1/systemone<br/>one noul question per Signal
        T-->>J: answers: noul 0..1 per question
        J->>DB: record_verdicts (hit = noul >= 0.5)
    end
    J-->>P: signal_hits event (SSE), once a pass per person
    P->>DB: GET /api/signals (counts),<br/>and the open Signal's list
```

The judge is a task of its own beside DeadAir's (`crates/ipx/src/signals.rs`), not a job on the
scan worker, so a slow TypeSafe never holds up a scan.

### Which items are asked about

`Db::unjudged` finds, newest first, up to 50 items that are:

- in a feed the Signal's owner subscribes to,
- not hidden from them by their block lists,
- for someone with a key that TypeSafe has not refused since the daemon started,
- and either published (or, with no date, first seen) in the look-back window and not yet in
  `signal_verdicts` for that Signal, or picked out by it before it was edited (`recheck`), however
  old.

The window is the admin's, Admin, Server, **Signals look back (days)** (`signal_days`, 7 by
default, 1 to 365; #276): each item in it is a call, and a month of production's feeds was about
8,000. So a new Signal looks back that far, and after that sees each new item. A show subscribed
to today has its back catalogue skipped, apart from the window.

The recent half reads the index `entries_recent`, on each feed's items by date, the expression it
is asked by; without it each wake read all of `entries`, 605 MB, 566 ms to find nothing (#274).

Rows for one item and one person are grouped, so an item is one call whatever the number of
Signals. On the server's key, everyone's Signals on the same item go in one call too, its text
sent once (#275). Up to ten calls are in flight at once (`IN_FLIGHT`): one at a time, about 150 ms
each, a month of items took twenty minutes. Every call to Jev, Signals', icons' and key checks',
takes a turn (`pace`, #277): TypeSafe's default limits are 80 requests and 100,000 input tokens a
second, and ipx keeps to 60 and 80,000, each call holding the line for a 60th of a second or its
guessed tokens' share, whichever is longer. A 429 holds every call back for its `Retry-After`, or
a second, and ends the pass. Each `jev` span says how long it waited (`paced_ms`), and
`ipx_jev_paced_seconds_total` adds them up; the dashboard draws calls and tokens a second against
the limits. A full batch means there may be more, and the next pass
starts at once. Otherwise the judge sleeps until woken or for 5 minutes.

### Every answer is kept

`signal_verdicts` records each answer, fitting or not. An item with no row has not been asked yet;
an item with a row is never asked again. So the cost is one call per item per person, once.
Changing a Signal's name, words or sentence marks what it picked out to be asked again
(`recheck`), and those stay in its list until answered; what it passed over stays passed over, and
new items are asked the new question as they come (#276). The first start with sentences (#259) deleted every verdict, since they answered the
old question.

## TypeSafe

Jev is TypeSafe's System One model, already used to pick a feed's Directory category
(`feed::classify`). The address is `feed::JEV_URL`, `https://api.typesafe.ai/v1/systemone`.
Signals use the **noul** question type, which answers how likely a statement is true, from 0 to 1.
Questions in one call are judged independently and in parallel.

### A request

Sent with `Authorization: Bearer <the key>`, the server's or the person's, for an item and two of their Signals:

```json
{
  "model": "jev-latest",
  "state": "Feed: Pit Lane\nTitle: Monaco qualifying\n\nQualifying report from the Monaco Grand Prix.",
  "questions": {
    "signal_3": {
      "type": "noul",
      "instructions": "The reader follows a subject they call \"F1\", described by these words: formula 1, f1. Is this item about that subject?",
      "criteria": {
        "true": "The item's main subject is that subject, as the name, the words and the reader's description together describe it.",
        "false": "It is about something else, or touches the subject only in passing."
      }
    },
    "signal_4": {
      "type": "noul",
      "instructions": "The reader follows a subject they call \"Food\", described by these words: baking. In their own words: Bread and cakes, not restaurants. Is this item about that subject?",
      "criteria": {
        "true": "The item's main subject is that subject, as the name, the words and the reader's description together describe it.",
        "false": "It is about something else, or touches the subject only in passing."
      }
    }
  }
}
```

- `state` is text: the feed's title, the item's title, and its show notes with the HTML taken out,
  cut to 1,000 characters (`TEXT_CHARS`). It was 2,000 until #275: on 55 real items and six
  Signals, 1,000 changed none of 330 verdicts and saved 12% of the input tokens (1,266 to 1,112 a
  call); 500 changed one. Most of the input is the questions themselves, not the item.
- Each question is keyed `signal_<id>`, and its answer comes back under the same key.
- `criteria` is an object saying what true and false mean. A sentence there is refused with a 422
  (#257).
- **One subject, not any word.** The first question was "Is this item about any of these: ai,
  development, programming?", so one broad word was enough: AI Development took Sega pulling a game
  remaster (0.90), a Surface Laptop review (0.95) and a C++ tutorial (0.99), 54 items in all. Asked
  about the subject, with the sentence "AI tools for writing software: coding agents, AI dev tools,
  vibe coding, AI in code review.", those scored 0.06, 0.16 and 0.02, and 5 of the 54 stayed in,
  AI code review in the Linux kernel highest at 0.94 (#259).

### Its answer

What TypeSafe answered to a request of this shape, on 2026-10-16:

```json
{
  "model": "jev-1.13.0",
  "answers": {
    "signal_3": { "type": "noul", "noul": 0.96 },
    "signal_4": { "type": "noul", "noul": 0.02 }
  },
  "usage": { "input_tokens": 389, "output_tokens": 40 }
}
```

0.5 or more is a fit (`THRESHOLD`): the item goes into Signal 3 and not Signal 4. A question
missing from the answer is left unjudged and asked again next pass.

### When TypeSafe says no

| TypeSafe answers | What the judge does |
|---|---|
| 401, 402, 403, on the server's key | Everyone's key is refused. The pass ends and the queue waits for the next one, five minutes on. Logged: "TypeSafe refused the server's key for Signals". |
| 401, 402, 403, on a person's key | The key is refused. That person is passed over until they save a key or the daemon restarts, so a bad key is not sent the whole queue. Logged once: "TypeSafe refused a Signal key". |
| 400, 422 | The request itself is wrong, so every item's would be. The pass ends; the items wait for the next one. Logged: "TypeSafe refused a Signals request as malformed", with TypeSafe's reason. |
| 429 | Over TypeSafe's limits. Every call waits for `Retry-After` or a second; the pass ends, what was in flight kept, and the rest wait for the next. Logged once: "TypeSafe asked Signals to slow down". |
| anything else, or no answer | That item is left unjudged and asked again next pass. Logged: "Jev could not judge an item". |

The error keeps TypeSafe's reply, cut to 500 characters, for example:

```
TypeSafe answered 422: {"detail":[{"type":"model_attributes_type","loc":["body","questions","3","noul","criteria"],"msg":"Input should be a valid dictionary or object to extract fields from","input":"Yes if the subject is one of them."}]}
```

## The API

Every route is the signed-in person's own. A Signal of someone else's is not found, as one that
does not exist.

### `GET /api/signals`

```json
[
  { "id": 1, "name": "AI Development", "keywords": ["ai", "development", "programming"],
    "description": "AI tools for writing software: coding agents, AI dev tools, vibe coding, AI in code review.",
    "icon": "solid/laptop-code", "items": 48, "unread": 5 },
  { "id": 2, "name": "Apple", "keywords": ["apple", "mac", "iphone"], "description": null, "icon": null, "items": 272, "unread": 0 }
]
```

`items` counts the items Jev picked out that are still in a subscribed feed and not hidden, what
its list shows; `unread` those of them not read.

### `POST /api/signals`

```json
{ "name": "Formula 1", "keywords": ["formula 1", "f1"], "description": "Grand prix racing, not road cars." }
```

Answers `{ "id": 3 }` and wakes the judge. A name is 1 to 100 characters. Between 1 and 50 words
are allowed, each up to 200 characters, kept once whatever their case. `description` is optional,
up to 500 characters. Anything else is a `400` saying so.

### `PATCH /api/signals/{id}`

Any of the three: `{ "name": "F1" }`, `{ "keywords": ["f1", "grand prix"] }`,
`{ "description": null }` (null or empty clears it), `{ "icon": "solid/robot" }`. A change to the
name, words or sentence drops the Signal's verdicts and wakes the judge, since each is part of the
question; a new icon does not. The page sends only what changed. An icon is `solid/`, `regular/` or
`brands/` and a plain name, since the page puts it into an address: anything else is a `400`. Answers `204`, or `404` for a Signal that is not yours.

### `DELETE /api/signals/{id}`

Answers `204`. The Signal and its verdicts go; the items stay in their feeds.

### `GET /api/signals/{id}/entries`

The same query as a feed's list, `offset`, `limit`, `filter`, `q`, `sort`, `dir`:

```
GET /api/signals/3/entries?filter=unread&sort=published&dir=desc&limit=50
```

```json
{
  "total": 1,
  "entries": [
    {
      "guid": "sig-1", "feed_id": "pit-lane", "title": "Monaco qualifying",
      "link": "https://pitlane.example/monaco", "published": 1792170000,
      "description": "<p>Qualifying report…</p>", "read": false, "flagged": false,
      "image": null, "duration": null, "episode": null, "season": null, "position": 0,
      "enclosures": []
    }
  ]
}
```

Reading or pinning one is the item's own route, `POST /api/entries/{feed_id}/{guid}/flags`, as
anywhere else.

### `PATCH /api/me`, the key

```json
{ "typesafe_key": "ts_…" }
```

A string is checked with TypeSafe, by one short question on that key, then saved, and the judge
tries it again. `null` or an empty string forgets it. A key is up to 500 printable characters.
A key TypeSafe refuses is a `400` ("TypeSafe did not accept that key") and not saved; TypeSafe not
answering is a `502`, and the key is not saved either. `GET /api/me` says `"typesafe_key_set":
true` and nothing more about the key, and `"signals": "server"` or `"own"`, whose key is used.

### `PATCH /api/settings`, who pays

An admin's `{ "signals_on_server": true }` judges everyone's Signals on `TYPESAFE_KEY`, and
`false` on each person's own. `GET /api/settings` has `signals_on_server` and
`typesafe_server_key`, whether the server has a key to use; without one the setting changes
nothing. A change wakes the judge and lets keys refused before be tried again.

### `POST /api/signals/icon`

Jev's pick of an icon, on the **server's** `TYPESAFE_KEY`, not the person's: it is the page's own
doing, once a Signal. Jev chooses among options and cannot name one, so the page searches Font
Awesome's index for each word of the name, words and sentence, the best six each, and sends up to 40:

```json
{
  "name": "AI Development",
  "keywords": ["ai", "development", "programming"],
  "description": "AI tools for writing software: coding agents, AI dev tools, vibe coding, AI in code review.",
  "candidates": [["solid/robot", "Robot: ai, android, bot, machine"], ["solid/laptop-code", "Laptop Code: computer, cpu, dell, demo"], ["solid/code", "Code: brackets, code, development, html"]]
}
```

ipx asks a **choice** question, each candidate a choice and its label and terms what it means:

```json
{
  "model": "jev-latest",
  "state": "Signal: AI Development\nListens for: ai, development, programming\nIn their words: AI tools for writing software: coding agents, AI dev tools, vibe coding, AI in code review.",
  "questions": { "icon": {
    "type": "choice",
    "instructions": "Which icon best stands for this subject in a list of the reader's topics?",
    "criteria": { "solid/robot": "Robot: ai, android, bot, machine", "solid/laptop-code": "Laptop Code: computer, cpu, dell, demo" }
  } }
}
```

TypeSafe answered, on 2026-10-16, for six candidates:

```json
{ "model": "jev-1.13.0", "answers": { "icon": { "type": "choice", "choice": "solid/laptop-code", "confidence": 0.34,
  "probabilities": { "solid/robot": 0.4, "solid/code": 0.14, "solid/laptop-code": 0.45, "solid/brain": 0.01, "solid/microchip": 0.0, "solid/terminal": 0.0 } } } }
```

The answer is `{ "icon": "solid/laptop-code" }`, only ever one it was offered. Each candidate has to
be an icon name the API takes, 1 to 60 of them. `404` when the server has no key, `502` when Jev
fails (logged: "Jev could not pick an icon"). The call has a `jev_icon` span.

### The event

Once a pass, for each person it found items for, the judge sends this on the event stream
(`/api/events`):

```json
{ "ev": "signal_hits", "user": 2, "hits": 3 }
```

A page with Signals fetches `/api/signals` again, and the open Signal's list.

## Data

```mermaid
erDiagram
    users ||--o{ signals : owns
    signals ||--o{ signal_verdicts : "has heard of"
    entries ||--o{ signal_verdicts : "judged in"
    users {
        bigint id
        text typesafe_key "NULL: no key"
    }
    signals {
        bigint id
        bigint user_id
        text name
        text keywords "JSON array"
        text description "NULL: none"
        text icon "solid/robot, NULL: initials"
        bigint created
    }
    signal_verdicts {
        bigint signal_id "PK"
        text feed_id "PK"
        text guid "PK"
        bool hit
    }
```

- `signals` and `signal_verdicts` are created from their entities like every table
  (`crates/ipx/src/entity.rs`). `users.typesafe_key`, `signals.description` and `signals.icon`
  were added to existing tables, so `create_missing` adds each column when it is missing.
- The icons are files of ipx-web's, not the database's: `web/build.mjs` copies Font Awesome Free's
  SVGs to `/icons/<style>/<name>.svg`, kept a day, and writes `/icons/index.json`, names, labels
  and search terms, about 90 KB compressed, named by its hash and fetched when the picker first
  opens. A Signal's tile draws its one file as a CSS mask, so it takes the theme's colour.
- A Signal's list is `entries_where` with
  `EXISTS (SELECT 1 FROM signal_verdicts v WHERE v.signal_id = $n AND v.feed_id = e.feed_id AND v.guid = e.guid AND v.hit)`
  on top of the subscribed-feeds scope. So filters, search, sort, block lists and read state all
  work as for All Subscriptions.
- A verdict for a Signal deleted while Jev was being asked is not stored. Postgres's foreign key
  would refuse it, and SQLite would keep it for nothing.

## Watching it

| | |
|---|---|
| Traces | A `signals` span for each pass (`entries`), and a `jev` span for each call (`user`, `signal_count`, `http.response.status_code`, and TypeSafe's reply as `error` when it refuses). |
| Log, counted | `ev="signal"`, "an item fits a Signal", for each item with a fit: `user`, `feed`, `new` (how many of their Signals it fits). |
| Log, warnings | The three in the table above. |
| Events | `signal_hits` on the socket and `/api/events`. |

## Limits

- **The key is kept in plain text** in `users.typesafe_key`, because every call needs it whole. It
  should be encrypted with a server key if the database is ever shared with anything else.
- **Refused keys are remembered in memory**, so a restart tries each once again.
- **Keywords changed while an item is being judged** can leave that one item's verdict from the old
  words.
- **The meter is the server's**, `ipx_jev_tokens_total` and `ipx_jev_requests_total` on
  `/metrics`, and the Signals row of the dashboard: tokens as TypeSafe's answers count them, by
  person (`user`, the account's name) and whose key paid (`key`, `server` or `own`; #278), with
  `ipx_signals_judged_total` and `ipx_signals_hits_total` the same way. A call shared by several
  people on the server's key is split by each one's share of its questions. Icon picks bill the
  person on the server's key, key checks the person on their own. The dashboard's **Who used what**
  sums them over its time range. It is
  one call per item in the person's feeds as they come, and per item in the window for a new Signal.
- **Only the window** is judged by publish date, apart from what an edit asks again.

## Testing

- `cargo test -p ipx`: the request's shape and the threshold (`signals::tests`), and the queue,
  verdicts, ownership, unread counts and new keywords (`db::tests::a_signal_shows_what_jev_said_fits_and_only_to_its_owner`),
  on SQLite and Postgres both.
- `node --test tests/page-unit.js`: a Signal's place id and its list's address.
- `npx playwright test -g Signal`: makes a Signal from the folder's +, and checks that the matching
  item shows and that reading it reads it in its feed and clears the badge. TypeSafe is stood in for
  by the fixture server's `/jev` (`IPX_TYPESAFE_URL`), which refuses the shape TypeSafe refuses, and
  `/signal.xml` is dated today, since only the window is judged. They spend the TypeSafe key, so
  each runs only when the Signals code or the test has changed since it last passed (`signalsTest`;
  `IPX_SIGNALS_TESTS=1` runs them anyway).