I unfortunately don't have an awesome answer here. We don't have an exhaustive list of contributing factors yet. I woke up to a server farm on fire as well. I'll share what we know, it's all I've got.
There were 2 web servers (of 9) affected. ny-web01
and ny-web04
exhausted connection pool capacity to Stack Overflow's database by many orders of magnitude. When this happens at our scale, it's impossible (mathematically) to recover from on a particular server (without halting traffic).
Some context for below:
- Stack Overflow runs on 9 web servers
- Each one handles 200-500 requests/sec at the time of day this occured
- All requests for the Q&A network are handled by a single application pool
What kept the application down
What happens in this situation is requests in the IIS queue (from HAProxy) need a thread from the pool. These then go to waiting behind the connection pool acquisition queue upon the first database call (we re-use 1 DB context for the lifetime of a request). At a per-server volume of 200-500 requests/sec average this time of day, you eventually get to thread pool starvation in IIS and async
(or any other) continuations stop happening.
The reason this happens is let's say we have a request doing work. It has done a database query and is waiting to handle the result. This task requires a completion thread, of which we have a finite amount. It looks like this:
Request -> DB Query -> SQL Server -> (data in buffer) -> (please sir may I have a thread?)
But, we don't have a thread. Because all of those requests from IIS have now exhausted the thread pool at our traffic per server volume. Almost every new thread available as the pool grows (which is very slowly) is being eaten by the next request in queue. All of the database calls (or any awaiter) in flight are waiting a very long time on a thread. During this time, they are occupying one of the connections in the pool. New requests are waiting on the pool and old requests are blocked by the new request (stolen threads) from finishing.
We now have a deadlock situation. In theory, less traffic or faster thread pool growth would resolve this, but less traffic didn't happen (I'll get to that) and faster thread pool growth isn't very controllable (and doing so is very dicey).
Most of the time efficiency (lots of requests on few servers) is awesome. In this situation it isn't. It exacerbates the problem.
What went wrong with our load balancer
So you saw a robot, why the hell didn't HAProxy take those 2 servers out of rotation before I did? Well, because ASP.NET errors redirect to /error
in the world we're in (we're not on ASP.NET Core yet...we're working towards it). This means HAProxy gets a 302 back, it interprets this as "success". It only removes the server when things get really bad and all those queues I talked about back up to the network layer and HAProxy <-> IIS timeouts occur.
There is a fix for this, we can tell HAProxy to only expect certain status codes (it's a regex setting), and it should know that our home page should never redirect. This is something I added the ability for in puppet months ago but never found the time to deploy. The priority just went up and I'll make sure this goes live next week.
What triggered this?
This is the part we're still digging on. You can see from this SQL connection graph per server something is causing a connection explosion (per app pool) intermittently:
While it may seem obvious that "something is opening up a lot of connections" it only appears to be a root cause. When we get a lot of SQL connection pool timeouts in the log this is a developer's natural first assumption. But that's misleading.
Remember above when we covered all the queues ultimately in play? Now imagine a stall across requests at any point in the pipeline. Ultimately you'd end up with a bunch of requests at the IIS/app pool front door waiting to charge in like an angry mob. There are various reasons this could happen. It could be a network hiccup, or a dependency failover, or a stall in a dependent service, or time spent in a locked section of code. Or more likely for us: time spent in garbage collection. We don't see evidence of that directly here but it's also very hard to diagnose due to GC being a pay-the-piper after the fact order. Correlating requests and causation isn't trivial and requires a lot of digging.
So we still have unknowns. We're working on it, I just wanted to give an update here since this was a very visible outage I strive to keep you away from.
I'm sorry we had a bad day and hope it didn't inconvenience anyone too much. We'll do better.
Update: I'm marking this complete as the HAProxy fix was deployed that week - going forward web servers will be summarily terminated without due process if they step out of line. We're harsh, but fair.