23

I was going through Stack Overflow's robots.txt (for the fun of it) and noticed:

Sitemap: https://stackoverflow.com/sitemap.xml

at the last line. The link gives a 404.

Am I not allowed to access it or is the link broken?

1 Answer 1

34

Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google crawler network, that 404 is very much by design.

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

As stated by Jeff Atwood here.

So to access the sitemap.xml file, your originating IP address needs to be whitelisted by Stack Exchange staff.

The reasons for this is, as in most cases, performance. And probably because we can't have nice things.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .