rays/ipx
1
0
Fork 0
Code Issues 11 Pull Requests Actions Packages Projects Releases 20 Wiki Activity

The directory has a robots.txt and a sitemap (#380)

Browse Source 
Crawlers asked for both about 70 times a day and got 404. robots.txt
keeps them off /search, which is any query string at all, and the token-only
API, and names the sitemap; the sitemap lists /, /all, each category and each
feed. Both need whole addresses, so they are written for the host asked for:
directory.ipxreader.app through the tunnel, whose cloudflared says https, or
the LAN address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-26 02:15:05 +00:00
parent 08b28f0598
commit 3f27dfdfdb
3 changed files with 51 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
crates/directory/src/api.rs
View File

@@ -381,6 +381,12 @@ mod tests {
assert_eq!(get_page(&app, "/category/Nothing").await.0, 404);
assert_eq!(get_page(&app, "/search?q=").await.0, 303);
assert_eq!(get_page(&app, "/site.js").await.0, 200);
// Crawlers are pointed at every page (#380).
let (s, robots, _) = get_page(&app, "/robots.txt").await;
assert!(s == 200 && robots.contains("Disallow: /v1/") && robots.contains("/sitemap.xml"), "{robots}");
let (s, map, _) = get_page(&app, "/sitemap.xml").await;
assert_eq!(s, 200);
assert!(map.contains(&format!("/feed/{id}</loc>")) && map.contains("/category/Podcasting</loc>"), "{map}");
// Reading the pages takes no token; the API still does.
assert_eq!(call(&app, "GET", "/v1/feeds", None, None).await.0, 401);
assert_eq!(call(&app, "DELETE", &format!("/v1/feeds/{id}"), None, None).await.0, 401);