133

It's unusual for me.

  1. I go to Performance questions about Generating Html Files from database.
  2. I tap on harmen's username.
  3. I'm directed to https://stackoverflow.com/error?aspxerrorpath=/users/176603/harmen with:

    We apologize for any inconvenience, but an unexpected error occurred while you were browsing our site. It’s not you, it’s us. This is our fault. Detailed information about this error has automatically been recorded and we have been notified.


Actually, now most links on Stack Overflow are returning me an error: I can hardly browse at all.

enter image description here

32
  • 5
    @ivarni Now I can't use Stack Overflow at all. Only Meta.
    – Cœur
    Commented Feb 23, 2018 at 8:07
  • 1
    It's down for me also not able to use it twitter.com/arasuvel001/status/966950632400871424
    – Arasuvel
    Commented Feb 23, 2018 at 8:22
  • 3
    Just for the record: I have been seeing odd search results (tag search on c# showing no records for example) the last few days. It happens intermittently. I mentioned it in the Tavern but did not get confirmation if they found a root cause. Last night Marc Gravell mentioned cache issues and loves to hear about it. The site and the specific pages are working atm for me (I'm in western-europe, in case it matters).
    – rene
    Commented Feb 23, 2018 at 8:22
  • And I just repro-ed it (8:51 UTC) on this link
    – rene
    Commented Feb 23, 2018 at 8:51
  • 4
    I can't reproduce with the specific links, but it seems random. At one point I got an error message telling me there was maintenance, then the "this is our fault". It doesn't seem to be related to specific links, but I can reproduce it from time to time. Comments don't work properly either, the inbox (and all of this is random, it doesn't always happen). Commented Feb 23, 2018 at 9:19
  • 2
    Singapore is having sporadic errors. The push notifications for comments and rep to my phone seem unaffected, but page downloads are 50-50. I'm losing points here, please fix this! Commented Feb 23, 2018 at 9:52
  • 11
    Apparently a world-wide problem. Guess we'll have to wait til the US-based staff wakes up and fixes it :)
    – Lundin
    Commented Feb 23, 2018 at 9:55
  • 1
    I'm pretty sure there are a couple of servers gone haywire which are put in and out of rotation by the load-balancer.
    – rene
    Commented Feb 23, 2018 at 10:00
  • 1
    Happy note :) Member for 46 days, I've never seen the site fall. Works very very well most of the time. Thanks for this great site and great community. Commented Feb 23, 2018 at 10:20
  • 6
    Maybe they just want us to see the new updated error page. The new image looks cool btw Commented Feb 23, 2018 at 10:21
  • 1
    I just had this on 4 out of 5 questions opened. I cannot even vote to close Commented Feb 23, 2018 at 10:25
  • 67
    The problem is clearly the new page with the dead robot. Previously you got a picture of a sysadmin cat who was "workin on ur problemz". The cat has been fired and thus nobody is working to fix this.
    – Lundin
    Commented Feb 23, 2018 at 10:26
  • 10
    status:twitter.com/Nick_Craver/status/966987353301639169
    – Suraj Rao
    Commented Feb 23, 2018 at 10:46
  • 1
    I'm looking for a dev now that can shine some light on this. Sorry for the crickets from us on this one folks.
    – user50049
    Commented Feb 23, 2018 at 15:59
  • 34
    Live footage of the investigation and work being done to fix this: i.sstatic.net/jJOlp.gif
    – Adam Lear StaffMod
    Commented Feb 23, 2018 at 17:09

2 Answers 2

95

I unfortunately don't have an awesome answer here. We don't have an exhaustive list of contributing factors yet. I woke up to a server farm on fire as well. I'll share what we know, it's all I've got.

There were 2 web servers (of 9) affected. ny-web01 and ny-web04 exhausted connection pool capacity to Stack Overflow's database by many orders of magnitude. When this happens at our scale, it's impossible (mathematically) to recover from on a particular server (without halting traffic).

Some context for below:

  • Stack Overflow runs on 9 web servers
  • Each one handles 200-500 requests/sec at the time of day this occured
  • All requests for the Q&A network are handled by a single application pool

What kept the application down

What happens in this situation is requests in the IIS queue (from HAProxy) need a thread from the pool. These then go to waiting behind the connection pool acquisition queue upon the first database call (we re-use 1 DB context for the lifetime of a request). At a per-server volume of 200-500 requests/sec average this time of day, you eventually get to thread pool starvation in IIS and async (or any other) continuations stop happening.

The reason this happens is let's say we have a request doing work. It has done a database query and is waiting to handle the result. This task requires a completion thread, of which we have a finite amount. It looks like this:

Request -> DB Query -> SQL Server -> (data in buffer) -> (please sir may I have a thread?)

But, we don't have a thread. Because all of those requests from IIS have now exhausted the thread pool at our traffic per server volume. Almost every new thread available as the pool grows (which is very slowly) is being eaten by the next request in queue. All of the database calls (or any awaiter) in flight are waiting a very long time on a thread. During this time, they are occupying one of the connections in the pool. New requests are waiting on the pool and old requests are blocked by the new request (stolen threads) from finishing.

We now have a deadlock situation. In theory, less traffic or faster thread pool growth would resolve this, but less traffic didn't happen (I'll get to that) and faster thread pool growth isn't very controllable (and doing so is very dicey).

Most of the time efficiency (lots of requests on few servers) is awesome. In this situation it isn't. It exacerbates the problem.

What went wrong with our load balancer

So you saw a robot, why the hell didn't HAProxy take those 2 servers out of rotation before I did? Well, because ASP.NET errors redirect to /error in the world we're in (we're not on ASP.NET Core yet...we're working towards it). This means HAProxy gets a 302 back, it interprets this as "success". It only removes the server when things get really bad and all those queues I talked about back up to the network layer and HAProxy <-> IIS timeouts occur.

There is a fix for this, we can tell HAProxy to only expect certain status codes (it's a regex setting), and it should know that our home page should never redirect. This is something I added the ability for in puppet months ago but never found the time to deploy. The priority just went up and I'll make sure this goes live next week.

What triggered this?

This is the part we're still digging on. You can see from this SQL connection graph per server something is causing a connection explosion (per app pool) intermittently:

SQL Connections

While it may seem obvious that "something is opening up a lot of connections" it only appears to be a root cause. When we get a lot of SQL connection pool timeouts in the log this is a developer's natural first assumption. But that's misleading.

Remember above when we covered all the queues ultimately in play? Now imagine a stall across requests at any point in the pipeline. Ultimately you'd end up with a bunch of requests at the IIS/app pool front door waiting to charge in like an angry mob. There are various reasons this could happen. It could be a network hiccup, or a dependency failover, or a stall in a dependent service, or time spent in a locked section of code. Or more likely for us: time spent in garbage collection. We don't see evidence of that directly here but it's also very hard to diagnose due to GC being a pay-the-piper after the fact order. Correlating requests and causation isn't trivial and requires a lot of digging.

So we still have unknowns. We're working on it, I just wanted to give an update here since this was a very visible outage I strive to keep you away from.

I'm sorry we had a bad day and hope it didn't inconvenience anyone too much. We'll do better.

Update: I'm marking this complete as the HAProxy fix was deployed that week - going forward web servers will be summarily terminated without due process if they step out of line. We're harsh, but fair.

12
  • 19
    Is it actually on fire or is that a metaphor? Commented Feb 23, 2018 at 22:28
  • 26
    I think it is awesome to be given this level of insight into these issues. As a user it is comforting to see the level of response and as a developer it is interesting to get a glimpse of the process.
    – takendarkk
    Commented Feb 23, 2018 at 22:45
  • @csmckelvey I remember seeing a write up or two from amazon about major aws issues. An even bigger scale than SO. Always interesting to see stuff at huge scales like this.
    – mbrig
    Commented Feb 23, 2018 at 22:51
  • Reminds me of: en.wikipedia.org/wiki/Lp0_on_fire Commented Feb 23, 2018 at 22:54
  • 23
    @Azor-Ahai we don't have a webcam in the cage these servers are in, but an actual fire might explain at least 2 things.
    – Nick Craver Mod
    Commented Feb 24, 2018 at 0:35
  • Why isn't that ASP.NET error redirect disabled?
    – jpmc26
    Commented Feb 25, 2018 at 1:54
  • 12
    Why are the servers in a cage? Did they do something wrong? Did they have fair trial? Commented Feb 25, 2018 at 6:56
  • 4
    @ShadowWizard: that would be to protect them from the mentioned angry mobs.
    – Jongware
    Commented Feb 25, 2018 at 9:28
  • 1
    @ShadowWizard cause they get bitey sometimes. Commented Feb 25, 2018 at 11:54
  • @JourneymanGeek can't blame them, with such overload they're really hungry. Commented Feb 25, 2018 at 12:10
  • @JourneymanGeek That's an image I don't ordinarily associate with the denizens of our datacenters. I should create a photoshopped image with teeth.... thanks for the morning chuckle! Commented Feb 25, 2018 at 12:30
  • 2
    Err. Sorry, but the first sentence is wrong. You do have an awesome answer. Commented Feb 25, 2018 at 21:39
6

Note that once you receive the error, you're redirected to those weirdo URLs:

https://stackoverflow.com/error?aspxerrorpath=/users/176603/harmen

When this happens, you can't just refresh the page. You have to either go back and click on the link again, or you need to delete out the error?aspxerrorpath=/ part to get back the real URL:

https://stackoverflow.com/users/176603/harmen

Even trying again immediately will often allow it to load. If you're not doing this, then you'll get a lot more "errors."

6
  • 3
    I've tried refreshing many opened tabs after an error recovery, and I've kept getting errors. (I use Safari and it doesn't display the path after the domain by default). It took me long before I've realized that I was actually refreshing the error page. I think it should be on SO's todo list to keep the URL the same instead of redirecting to an error page, for a better UX. Commented Feb 25, 2018 at 10:11
  • 1
    @CanPoyrazoğlu 6 to 8 weeks* ;) Although personally, I think the meme should be changed to 6 to 8 years! (* See this meta)
    – robinCTS
    Commented Feb 25, 2018 at 11:33
  • Yeah this is annoying. IMO it should be a 500 response without a Location header, with the response content as now Commented Feb 25, 2018 at 13:11
  • 1
    @LightnessRacesinOrbit blame ASP.NET.
    – Braiam
    Commented Feb 25, 2018 at 16:38
  • @Braiam: Done, as requested. Commented Feb 25, 2018 at 16:41
  • @Braiam or IIS. I really want to blame Chrome and/or SAP, but I guess that's a stretch.. Commented Feb 25, 2018 at 18:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .