Skip to main content
(While we are at it.) - yes, I did not bump it (that was by a question edit)!
Source Link
Peter Mortensen
  • 31.6k
  • 4
  • 22
  • 14

Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google Crawlercrawler network, that 404 is very much by design.

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

As stated by Jeff Atwood here.

So to access the sitemap.xml file, your originating IP-address address needs to be whitelisted by Stack Exchange staff.

The reasons for this is, as in most cases, performance. And probably because we can't have nice things.

Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google Crawler network, that 404 very much by design.

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

As stated by Jeff Atwood here.

So to access the sitemap.xml, your originating IP-address needs to be whitelisted by Stack Exchange staff.

The reasons for this is, as in most cases, performance. And probably because we can't have nice things.

Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google crawler network, that 404 is very much by design.

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

As stated by Jeff Atwood here.

So to access the sitemap.xml file, your originating IP address needs to be whitelisted by Stack Exchange staff.

The reasons for this is, as in most cases, performance. And probably because we can't have nice things.

Source Link
rene
  • 42.3k
  • 55
  • 610
  • 777

Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google Crawler network, that 404 very much by design.

I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.

As stated by Jeff Atwood here.

So to access the sitemap.xml, your originating IP-address needs to be whitelisted by Stack Exchange staff.

The reasons for this is, as in most cases, performance. And probably because we can't have nice things.