rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

The directory has a robots.txt and a sitemap (#380)

Browse Source 
Crawlers asked for both about 70 times a day and got 404. robots.txt
keeps them off /search, which is any query string at all, and the token-only
API, and names the sitemap; the sitemap lists /, /all, each category and each
feed. Both need whole addresses, so they are written for the host asked for:
directory.ipxreader.app through the tunnel, whose cloudflared says https, or
the LAN address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-26 02:15:05 +00:00
parent 08b28f0598
commit 3f27dfdfdb
3 changed files with 51 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
crates/directory/CHANGELOG.md
View File

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- A `robots.txt` and a sitemap listing the home page, A to Z, every category and every feed, so search engines find the pages, not only the links between them (#380).
- `/metrics` counts its requests and their times by route and status, the feeds it checks and those that failed, and the snapshots it takes (#370).
- Its log lines inside a trace carry the trace's and span's ids, as ipx's do, so a failed feed check leads to its trace (#375).