21

I noticed earlier that the sitemap.xml linked in the robots.txt returns HTTP 404.
On stackexchange.com no sitemap is linked and on other Stack Exchange sites it's 404 again.

9
  • 3
    Not like that really matters, Google has a custom indexing policy for Stack Overflow anyway, but I guess they should at least remove it from their robots.txt.
    – Erik A
    Commented Jan 28, 2019 at 16:00
  • 18
    It returns 404 becuase the sitemap.xml is whitelisted for the googlebot (and a few others) and blacklisted, aka 404, for everyone else.
    – rene
    Commented Jan 28, 2019 at 16:15
  • 18
    reference: meta.stackexchange.com/questions/33965/… so not a bug but status-by-design
    – rene
    Commented Jan 28, 2019 at 16:17
  • @rene Interesting. Thanks for the heads up!
    – NullDev
    Commented Jan 28, 2019 at 16:40
  • @ErikvonAsmuth Well, googlebot always is always in the same IP Range. So its probably restricted to specific IP's.
    – NullDev
    Commented Jan 28, 2019 at 16:44
  • @NullDev Yeah, removed the comment because I thought that might be true (+ I forgot to remove cookies so that could also interfere)
    – Erik A
    Commented Jan 28, 2019 at 17:14
  • 3
    @NullDev Not quite true about the IP range, Google doesn't publish the IP addresses, instead it suggests you do a reverse DNS lookup to get the name, then a forward lookup to confirm the IP.
    – DavidG
    Commented Jan 29, 2019 at 11:45
  • @DavidG Ah good to know. Thanks for the info!
    – NullDev
    Commented Jan 29, 2019 at 16:52
  • 403 would be the appropriate error code here...
    – idmean
    Commented Jan 29, 2019 at 19:51

1 Answer 1

6

As rene said in a comment, the sitemap.xml only actually returns a sitemap if the thing accessing it is on a list of things allowed to access it. Search engines like Googlebot and BingBot are allowed, but all other users aren't.

The reason for only allowing some things through is because the sitemap is very large and uses up quite a lot of bandwidth. It was implemented in response to too many users requesting it, probably due to badly-written Firefox extensions:

We have to protect the sitemap because it's enormous; we only publish the last 50k questions, but that's a huge XML file. It's not like it contains any secrets or anything -- it's just an XML file with a list of the last 50k questions to be updated on any of the trilogy sites.

Before we did this, it used up many gigabytes of bandwidth through incorrect retrievals. We think badly written Firefox plugins were mostly to blame, but it's hard to tell.

Anyway, we use a whitelist type approach. If you're on the whitelist, you get to retrieve the sitemap. If you are not, you don't.

As far as how the site detects whether someone's allowed, it appears to be based on user agent.

3
  • It does a reverse DNS check: meta.stackexchange.com/a/180660/158100
    – rene
    Commented Jan 29, 2019 at 20:01
  • @rene So then why did Shog say it's by user agent?
    – gparyani
    Commented Jan 29, 2019 at 20:03
  • Because he is wrong? Spoofing the user agent doesn't give me a sitemap.xml and it never has. I'm pretty sure I got that ip check confirmed in the past but I can't find where. I expect it might work for Shog9 assuming he reaches SO over an internal route that is allowed to access that resource.
    – rene
    Commented Jan 29, 2019 at 20:10

Not the answer you're looking for? Browse other questions tagged .