16

The following URLs all yield 502 errors for me:

I was alerted to this by another user.

Update: seems to be back up.

14
  • 10
    I think it's actually cloudflare that's experiencing problems.
    – ChrisF Mod
    Jul 2, 2019 at 14:02
  • 1
    The images work for me and you're using HTTP instead of HTTPS in your last link. But I wonder what SE should do about a third-party provider having a downtime?
    – Tom
    Jul 2, 2019 at 14:03
  • I just noticed this, some images were working, others weren't (maybe some working was due to caching) Jul 2, 2019 at 14:05
  • They all load for me currently, so either a transient issue or a localised CDN issue, where a specific (set of) access point(s) is down.
    – Martijn Pieters Mod
    Jul 2, 2019 at 14:09
  • @Tom Seems fickle - my face briefly reappeared for me, then was a 502 again when I started writing this comment, then yielded the custom "hair on fire" imgur error message, and now is back. As for what SE should do - in the short term, probably nothing. But I figure it's worth having these Meta questions to document the outages and let people know why they're seeing broken images.
    – Mark Amery
    Jul 2, 2019 at 14:10
  • 3
    @Tom: nothing, if the downtime is transient, or start looking for an alternative provider if the problem is systemic. Just like any engineering issue. The web is distributed, so sometimes something breaks for a bit. Nothing major here.
    – Martijn Pieters Mod
    Jul 2, 2019 at 14:11
  • I have the same issue and I sometimes get a nice "Imgur is over capacity!" error page.
    – Glorfindel
    Jul 2, 2019 at 14:11
  • 4
    It's hit-or-miss right now. Cloudflare's status page says they're looking into issues
    – Machavity Mod
    Jul 2, 2019 at 14:12
  • I thought i.stack.imgur.com is a special instance for Stack Exchange? Jul 2, 2019 at 14:13
  • @Chr that’s correct.
    – Tim
    Jul 2, 2019 at 14:16
  • @ChristianGollhardt it can still run on Cloudflare. Imgur itself didn't appear to be affected though.
    – Zoe is on strike Mod
    Jul 2, 2019 at 14:23
  • 2
    @Zoe most of imgur runs via fastly these days. I suspect that the SO domain just hasn’t transitioned yet. (No actual evidence other than that imgur explicitly lists their fastly CDN status on their status page).
    – Martijn Pieters Mod
    Jul 2, 2019 at 14:33
  • @Tom What they should do is cash in on that sweet SLA money, they probably have one for their instance of imgur. The downtime was probably too short, though.
    – Erik A
    Jul 2, 2019 at 14:34
  • The service is running right now.
    – Gourav
    Jul 2, 2019 at 14:35

1 Answer 1

16

According to Cloudflare Status:

Cloudflare has implemented a fix for this issue and is currently monitoring the results.

We will update the status once the issue is resolved.

So hopefully this is working again.


Update 2019-07-03 00:30 UTC:

An email was sent to Cloudflare customers with the following information:

Dear Cloudflare Customer,

Today at approximately 13:42 UTC we experienced a global service disruption that affected most Cloudflare traffic for 27 minutes.

The issue was triggered by a bug in a software deploy of the Cloudflare Web Application Firewall (WAF) which resulted in a CPU usage spike globally, and 502 errors for our customers. To restore global traffic we temporarily disabled certain WAF capabilities, removed the underlying software bug, then verified and re-enabled all WAF services.

We’re deeply sorry about how this disruption has impacted your services. Our engineering teams continue to investigate this issue and we will be sharing detailed incident report(s) on the Cloudflare blog.

~The Cloudflare Team


There is also a blog post with more details regarding the specific cause of this outage:

The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. These rules were being deployed in a simulated mode...

Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw...

We were seeing an unprecedented CPU exhaustion event, which was novel for us as we had not experienced global CPU exhaustion before...

At 1402 UTC we understood what was happening and decided to issue a ‘global kill’ on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic. That occurred at 1409 UTC.

We then went on to review the offending pull request, roll back the specific rules, test the change to ensure that we were 100% certain that we had the correct fix, and re-enabled the WAF Managed Rulesets at 1452 UTC.

2

Not the answer you're looking for? Browse other questions tagged .