Assuming you're not accessing https://stackoverflow.com/sitemap.xml from within the Google Crawlercrawler network, that 404 is very much by design.
I agree that checking the user-agent wasn't enough. I added a reverse DNS check as well.
As stated by Jeff Atwood here.
So to access the sitemap.xml file, your originating IP-address address needs to be whitelisted by Stack Exchange staff.
The reasons for this is, as in most cases, performance. And probably because we can't have nice things.