I was going through Stack Overflow's robots.txt (for the fun of it) and noticed:
Sitemap: https://stackoverflow.com/sitemap.xml
at the last line. The link gives a 404.
Am I not allowed to access it or is the link broken?
I was going through Stack Overflow's robots.txt (for the fun of it) and noticed:
Sitemap: https://stackoverflow.com/sitemap.xml
at the last line. The link gives a 404.
Am I not allowed to access it or is the link broken?
Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google crawler network, that 404 is very much by design.
I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.
As stated by Jeff Atwood here.
So to access the sitemap.xml file, your originating IP address needs to be whitelisted by Stack Exchange staff.
The reasons for this is, as in most cases, performance. And probably because we can't have nice things.