ipx/docs/directory.md

The directory

ipx-directory is a catalogue of public podcasts and blogs that ipx servers share (#204). An ipx server sends it the public feeds its people subscribe to, and how many of them do. The directory fetches each feed's title, artwork, category, description and latest twenty items every six hours, and any server holding a token can browse it. It never downloads a file, and it never stores one's address: every ipx server still fetches its own feeds.

This server's is at https://directory.ipxreader.app (moved from ipx-dir.sdf1.net, which answers as well until every server using it has moved, #214), the ipx-directory service of the Arcane project ipx, with its database ipx_directory on the databases project's Postgres.

In ipx

An admin sets it up in Admin, Directory: the directory's address, the token its operator gave this server, and whether to share. ipx asks the directory on its people's behalf, so the page never talks to it and never sees the token or a feed's address. With no directory set up, the Directory says so and feeds are added by their address.

ipx keeps the address and token with its other server settings, in its database (settings, directory_url and directory_token), not in config.toml. The token is written, never shown again: the tab says it is set and offers to clear it. To replace one, make a new token on the directory, paste it over the old, then delete the old token's entry from the directory's config.

With sharing on, ipx sends its snapshot when what it would send has changed (checked after every scan, and whenever the catalogue or these settings change), and every half hour regardless (#211): each feed checked at least once, with how many people here subscribe, and the category an admin gave a feed that names none. It never sends an OPML or a Patreon creator itself (their feeds go instead), or a private feed. A feed's settings have a Private switch for a paid feed iPX does not recognise by its address; anyone subscribed may turn it on, and only an admin off.

When this server's token is rwa, its admins administer the directory from iPX (#206). A feed's page in the Directory has Check now and Remove from the directory, which blocks it so no server's snapshot brings it back. Admin, Directory lists the Servers sharing, by token name, with how many feeds and subscribers each gave, and Blocked, by title, with Unblock. Dropping a server's snapshot takes its feeds and counts out until it next shares; to keep a server out, revoke its token. An admin's own subscriptions open as their feeds, not as Directory pages, so removing one of those is done by marking it private, or from another admin's account.

In a web browser

Anyone may browse it at its address, no token and no sign-in (#215): the most subscribed, the categories, every feed A to Z, a search by name, and a page for each feed with its description, its latest items and its address, with a Copy button, to subscribe to in any podcast app or feed reader. The pages only read: nothing in them changes the directory. They are rendered by the directory itself from what the API serves, with the publisher's HTML shown as text, an item's link only when it is http(s), and a Content-Security-Policy that allows the site's own script and stylesheet and the covers, which load from their publishers. /v1 is the API and still needs a token for everything.

Tokens

Everything but /healthz needs a token, sent as Authorization: Bearer <token>. Each one has a name and one of three levels of access:

	browse	send snapshots	remove, block, refresh feeds; drop a server's snapshot
ro	yes
rw	yes	yes
rwa	yes	yes	yes

Tokens are in the config file and nowhere else, and no level of access can make or revoke one over the API. To make one:

docker exec ipx-directory ipx-directory token new <name> --access ro|rw|rwa

It prints the token, to give to whoever it is for, and the lines to add to the config file. The file holds the token's SHA-256, never the token. The directory reads the file again within a minute of a change, so adding a token and revoking one (deleting its entry) need no restart. A revoked token also takes the counts its server sent. A file that does not load (a misspelled access, two tokens with one name) leaves the tokens as they were, and the log says why.

The config file

/config/directory.toml in the container (DIRECTORY_CONFIG, or --config), on Tower /mnt/fast/appdata/ipx-directory/directory.toml:

bind = "0.0.0.0:8100"
# The environment variable holding the database URL: postgres://... or sqlite://...
database_url_env = "DIRECTORY_DATABASE_URL"

[[tokens]]
name = "tower"
access = "rwa"
token_sha256 = "…"

DIRECTORY_LOG_FORMAT=json logs one JSON object a line, for Loki, as ipx's IPX_LOG_FORMAT does. Each request is a line with its method, path, status, duration_ms and the token's name.

With OTEL_EXPORTER_OTLP_ENDPOINT set (http://host:4318, OTLP over HTTP), it sends traces: a span a request, a child of the caller's when the request carries a traceparent, as ipx's do, and a trace a refresher round with a span a feed (#213). OTEL_RESOURCE_ATTRIBUTES tags them, as production does with deployment.environment.name=production. Its version is in crates/directory/Cargo.toml, logged at start and answered by ipx-directory --version; its changelog is crates/directory/CHANGELOG.md.

Running one

services:
  ipx-directory:
    image: ipx-directory          # docker build -f Dockerfile.directory -t ipx-directory .
    restart: unless-stopped
    environment:
      DIRECTORY_DATABASE_URL: "postgres://ipx_directory:password@db-host:5432/ipx_directory"
    ports:
      - "8100:8100"
    volumes:
      - ./directory:/config:ro    # directory.toml
    healthcheck:
      test: ["CMD", "ipx-directory", "health"]

What it will not hold

A private feed is never meant to reach it. ipx leaves out a feed marked private, one with a username or password, and one whose address ipx_feed::url_looks_private catches: a paid-feed host, or a key in the query. The directory checks every address it is sent with the same function and refuses what it catches, so one server's miss does not publish someone's paid feed to every other. What both miss, an rwa token can remove: DELETE /v1/feeds/{id} removes the feed and blocks it, so no snapshot brings it back.

API

Route	Access
GET /healthz	none	ok when it can read its database
GET /v1/me	ro	the token's name and access
GET /v1/feeds	ro	every feed A to Z: id, url, title, image, category, subcategory, podcast, subscribers
GET /v1/popular?limit=10	ro	the most subscribed, at most 100
GET /v1/feeds/{id}	ro	one feed, with description, failing (why its last check failed, in words) and its latest twenty items (guid, title, link, published, duration, description), never their files
POST /v1/snapshot	rw	[{"url", "subscribers"}], at most 20,000: replaces everything this token sent before; answers what it accepted and how many it refused as private, blocked or invalid
DELETE /v1/feeds/{id}	rwa	remove and block
GET /v1/blocked, DELETE /v1/blocked/{id}	rwa	what is blocked; unblock
GET /v1/instances	rwa	each token holding a snapshot: its name, feeds and subscribers
DELETE /v1/instances/{name}	rwa	drop one token's snapshot, its counts with it
POST /v1/feeds/{id}/refresh	rwa	fetch the feed now, answering with its page

A feed's id is the first 64 bits of the SHA-256 of its address, normalised (scheme and host in lower case, no default port or fragment), so every server arrives at the same id. subscribers is the sum over every token's latest snapshot. A feed is listed once the directory has read it as a feed, so an OPML or an address that never worked is not, and goes when no snapshot has it any more. Descriptions are the feed's own HTML, unsanitized: whoever shows them sanitizes them, as ipx does.