rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

The directory has no robots.txt or sitemap #380

New Issue
Closed
opened 2026-10-25 19:12:13 -07:00 by rays · 1 comment
rays commented 2026-10-25 19:12:13 -07:00
Owner
Copy Link

Crawlers ask directory.ipxreader.app for /robots.txt (46 a day) and /sitemap.xml (24) and get 404, so they find its public pages (#215) only by following links. Seen in the prod check of 2026-10-26: sum by (path) of the directory's 404s.

Crawlers ask directory.ipxreader.app for /robots.txt (46 a day) and /sitemap.xml (24) and get 404, so they find its public pages (#215) only by following links. Seen in the prod check of 2026-10-26: sum by (path) of the directory's 404s.
rays added the enhancement label 2026-10-25 19:12:13 -07:00
rays commented 2026-10-25 19:18:08 -07:00
Author
Owner
Copy Link

Fixed in 3f27dfd: /robots.txt lets crawlers read every page but /search and /v1/ and names the sitemap; /sitemap.xml lists /, /all, each category and each feed, 1,926 addresses now, with the host asked for (https through the tunnel). Deployed: both answer at directory.ipxreader.app.

Fixed in 3f27dfd: /robots.txt lets crawlers read every page but /search and /v1/ and names the sitemap; /sitemap.xml lists /, /all, each category and each feed, 1,926 addresses now, with the host asked for (https through the tunnel). Deployed: both answer at directory.ipxreader.app.
rays closed this issue 2026-10-25 19:18:08 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
rays
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rays/ipx#380
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?