I noticed earlier that the sitemap.xml linked in the robots.txt returns HTTP 404.
On stackexchange.com no sitemap is linked and on other Stack Exchange sites it's 404 again.
-
3Not like that really matters, Google has a custom indexing policy for Stack Overflow anyway, but I guess they should at least remove it from their robots.txt.– Erik ACommented Jan 28, 2019 at 16:00
-
18It returns 404 becuase the sitemap.xml is whitelisted for the googlebot (and a few others) and blacklisted, aka 404, for everyone else.– reneCommented Jan 28, 2019 at 16:15
-
18reference: meta.stackexchange.com/questions/33965/… so not a bug but status-by-design– reneCommented Jan 28, 2019 at 16:17
-
@rene Interesting. Thanks for the heads up!– NullDevCommented Jan 28, 2019 at 16:40
-
@ErikvonAsmuth Well, googlebot always is always in the same IP Range. So its probably restricted to specific IP's.– NullDevCommented Jan 28, 2019 at 16:44
-
@NullDev Yeah, removed the comment because I thought that might be true (+ I forgot to remove cookies so that could also interfere)– Erik ACommented Jan 28, 2019 at 17:14
-
3@NullDev Not quite true about the IP range, Google doesn't publish the IP addresses, instead it suggests you do a reverse DNS lookup to get the name, then a forward lookup to confirm the IP.– DavidGCommented Jan 29, 2019 at 11:45
-
@DavidG Ah good to know. Thanks for the info!– NullDevCommented Jan 29, 2019 at 16:52
-
403 would be the appropriate error code here...– idmeanCommented Jan 29, 2019 at 19:51
1 Answer
As rene said in a comment, the sitemap.xml
only actually returns a sitemap if the thing accessing it is on a list of things allowed to access it. Search engines like Googlebot and BingBot are allowed, but all other users aren't.
The reason for only allowing some things through is because the sitemap is very large and uses up quite a lot of bandwidth. It was implemented in response to too many users requesting it, probably due to badly-written Firefox extensions:
We have to protect the sitemap because it's enormous; we only publish the last 50k questions, but that's a huge XML file. It's not like it contains any secrets or anything -- it's just an XML file with a list of the last 50k questions to be updated on any of the trilogy sites.
Before we did this, it used up many gigabytes of bandwidth through incorrect retrievals. We think badly written Firefox plugins were mostly to blame, but it's hard to tell.
Anyway, we use a whitelist type approach. If you're on the whitelist, you get to retrieve the sitemap. If you are not, you don't.
As far as how the site detects whether someone's allowed, it appears to be based on user agent.
-
It does a reverse DNS check: meta.stackexchange.com/a/180660/158100– reneCommented Jan 29, 2019 at 20:01
-
-
Because he is wrong? Spoofing the user agent doesn't give me a sitemap.xml and it never has. I'm pretty sure I got that ip check confirmed in the past but I can't find where. I expect it might work for Shog9 assuming he reaches SO over an internal route that is allowed to access that resource.– reneCommented Jan 29, 2019 at 20:10