mirror of
https://github.com/pockethost/pockethost.git
synced 2025-05-22 23:06:55 +00:00
11 lines
229 B
Plaintext
11 lines
229 B
Plaintext
# www.robotstxt.org/
|
|
|
|
# Allow crawling of all content
|
|
User-agent: *
|
|
Allow: /
|
|
|
|
# Disallow crawling of specific paths (examples)
|
|
Disallow: /instances/
|
|
|
|
# Sitemap location (if you have one)
|
|
Sitemap: https://pockethost.io/sitemap.xml |