53

A while ago we had Stack Status announced. The point of the site was to let keep us better apprised of problems on the site. Why then, is it almost always reporting "All systems operational" when they aren't operational?

Take the downtime that is occurring over on Main at the moment, right now, Stack Status is reporting "All systems operational" yet the user base knows this isn't true.

More odd, when the site did initially go down, an incident was raised and the sites showed as unavailable, however, that incident was seemingly deleted, and the status of the sites changed to "operational" while they were still down.

What I don't understand is, why implement a tool to keep the userbase better apprised if:

  1. It isn't correct.
  2. When it is correct, delete the incident(s) and make it incorrect

Stack Status currently says that there have been 2 incidents in the last 2 weeks, but based on my personal use, there has been an incident on most days (there was a probably with users getting blocked access via rate limiting yesterday for example, and I'm sure I got the "offline for maintenance" message earlier in the week).

During the time I have written this, the site has remained "offline" for the majority of the time, and there was a downtime before I started typing this.

Stack Status was a good idea, and it's a good tool if it is used correctly; why therefore isn't it?

28
  • 2
    Yeah, it's deader than 'Dead' Jock McDead, again:( 'The service is unavailable.' Commented Jul 29, 2022 at 14:01
  • 2
    Well, there's at least there new incident (though I dispute the time of said incident, as it was not there are 14:42 (which I assume is my local time)).
    – Thom A
    Commented Jul 29, 2022 at 14:02
  • 1
    Now it shows "Outage". Maybe it just doesn't update as fast as you would like. I can also confirm that SO worked about ~10mins ago for me, so it can't be wrong for that long (although not reliable).
    – MegaIng
    Commented Jul 29, 2022 at 14:05
  • 3
    Stack isn't exactly super fast at responding to these things, but that might be down to the dev team already being minimal. I agree, though; as it currently stands, stackstatus is completely useless. There are better platforms for tracking statuses. One thing that would make stackstatus useful is some automated systems. Not necessarily automatic incident declaration, but at least a ping graph, and consistent use of stackstatus when there's real maintenance would go a long way towards making it a bit more useful Commented Jul 29, 2022 at 14:11
  • 10
    Though personally, I just assume that if SO is down, it's being DDoSed, because for the past few months, that has been 100% correct. SE has never been good at transparency either, at least not during ongoing incidents. I'm not even asking for a lot, just an acknowledgement that the site is being DDoSed and measures are being taken to mitigate it is better than radio silence, either with no details but an acknowledgement that a thing happened that has been fixed, or by not reporting anything about the downtime at all. Commented Jul 29, 2022 at 14:15
  • 1
    ... though for the sake of balance, they did at one point write a blog post about their DDoS mitigation, as well as parts of the adaption the attackers did, and their countermeasures, but that doesn't help when the transparency is needed right as The Thing:tm: happens. Don't get me wrong, the blog post (or MSE question, I forget) was still nice, but doesn't help much when new stuff happens Commented Jul 29, 2022 at 14:16
  • 1
    I assume you mean this annoucement, Update on the ongoing DDoS attacks and blocking Tor exit nodes, @ZoestandswithUkraine . I agree, it was nice to get an update a the time, and gave us a reason for why the sites has been so flaky of late. Just frustrating that it's been radio silence since, as you note. It's a real motivation killer to be trying to answer a question to be greeted with errors.
    – Thom A
    Commented Jul 29, 2022 at 14:21
  • 1
    @Larnu that's the one, yeah. Hard to tell if the attackers this time are the same as the previous attacks, but I guess that's fully irrelevant. I fully agree it's a motivation killer, especially when (seemingly random) instability is part of it. I hope they find a way to nail the attackers to the wall at some point Commented Jul 29, 2022 at 14:25
  • 18
    FWIW it now shows it's down. I should also note that, within the industry, very few status pages reflect problems in real-time. AWS, for instance, sometimes takes hours to show there's a problem, if at all.
    – Machavity Mod
    Commented Jul 29, 2022 at 14:29
  • 2
    Doesn't the error page already do a great job of telling you there's an error anyway?
    – Kevin B
    Commented Jul 29, 2022 at 15:11
  • 3
    Worth adding that the incident that's been added since the creation of this post does list "high load" as the culprit, which definitely leans towards DDOS, if not saying it explicitly.
    – zcoop98
    Commented Jul 29, 2022 at 15:16
  • 2
    @Machavity FWIW, it showed that the site is down within a few minutes of this post being made, which itself was within a few minutes of the incident happening. So the status page might not be real time, but it's really damn close to that.
    – MegaIng
    Commented Jul 29, 2022 at 18:44
  • 2
    I'm sick of this DDOS <expletive deleted>. Why can't people just use MSDOS like we used to? Commented Jul 29, 2022 at 22:21
  • 4
    I can't speak officially on behalf of the SRE team, but they are making efforts to make the automation between internal incident management and external visibility a lot more, well, automatic. In multiple recent incidents the automation did not work as defined/expected and so some manual intervention still has to take place. Until we get that automation completely sorted out, I think it's expected that our priority is going to be (a) get the site up (b) anything else.
    – Aaron Bertrand Staff
    Commented Jul 30, 2022 at 18:21
  • 1
    @AaronBertrand I can't remember the last time I checked stackstatus, I assumed it was hooked up because of the test message. Re: DDoS, a lot of us assume the downtime is malicious, because it's been happening a lot, and as a post linked earlier here backs up, there's a statement from SE that acknowledges that there have been proper, malicious attacks, and that they're highly adaptive. That doesn't guarantee downtime = DDoS, I'll give you that, but you lot give us absolutely no indication. The consistency and emerging nature of repeated downtime matches that of an attack Commented Jul 31, 2022 at 14:19

1 Answer 1

8

Apologies for the late response, lifted from the larger Meta post:

Now that we have an automated status page, we are examining how we can improve the process so that it is updated sooner. Other improvements revolve around additions to the status page itself and how information is displayed. To be clear, the status page reflects that a human is working on the issue, not whether our monitoring system has detected issues. We are working to improve internal processes related to communication, standardizing processes to be more consistent, and clarifying what events trigger communication.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .