27

I was lucky enough to catch this:

enter image description here

A sudden notification of a massive amount of rep gain, and when you click on the tab, absolutely nothing. The screenshot was taken literally seconds after I caught and opened it.

This happens to be the 3rd time I've noticed this today. The first time, it was a +50 gain, and the second time it was +92. The first two times, I assumed it was a user serial upvoting (and undoing it) as a joke, and dropped it. But this is a bit much.

If you view my profile, you'll see I have not answered any bounty questions, nor am I aware of any bounty being offered on any questions I have answered.

This clearly seems like a bug. What could be the cause of it?


Edit: Occurrence #4

enter image description here

Edit: Occurrence #8

enter image description here

Edit: Occurrence #12

enter image description here

Edit:

enter image description here

Edit:

enter image description here

Edit:

enter image description here

11
  • 3
    @HansPassant Well, it's still on the fritz (see edit), so I'll leave it up for a while yet.
    – cs95
    Commented Sep 27, 2017 at 16:40
  • 1
    ..and still bugging out now. 2017-09-28 : 16.06 GMT
    – Paulie_D
    Commented Sep 28, 2017 at 15:04
  • 2
    I asked the same thing on Meta Stack Exchange a while ago, but my question was tagged [status-completed]. It seems like this bug is back. Commented Sep 29, 2017 at 11:04
  • I think it has something to do with me downvoting an answer (hence resulting in a -1 entry in the rep history), and then that answer getting deleted, which results in removal of that entry - which causes an update on the client (browser). But for some reason, instead of silently removing the entry, the application is giving me a notification of all rep I have gained during the day. [My-two-cents] Commented Sep 29, 2017 at 11:28
  • 4
    You are the new reputation king of SO, don't be surprised if the rep system is bugging out.
    – ٴٴٴ
    Commented Sep 29, 2017 at 11:37
  • For what it’s worth, I have an inverse effect that I get a notification for some reply on my phone, and when I look at it from my desktop, I don’t see the icon highlighted. But when I then press the notification icon, I can see that reply appear there as unread.
    – poke
    Commented Sep 30, 2017 at 12:05
  • @poke that’s happened to me a few times as well.
    – cs95
    Commented Sep 30, 2017 at 16:06
  • 1
    Still occurring for me too, my dupe question with details Commented Oct 9, 2017 at 7:50
  • To help me find a cause: is this happening only in realtime (websockets) or persists when you reload the page? We're a bit slammed with Channels at the moment but obviously don't want to leave this annoying crap in. Any info would be helpful in isolating the area here. Commented Oct 10, 2017 at 21:49
  • It looks recalc-to-network related, though I'm not sure what the cause is quite yet (no obvious changes/regressions there). I'll be taking a deep look in the AM. Commented Oct 10, 2017 at 22:15
  • @NickCraver It happens periodically, and isn’t triggered by anything in particular. I’ll keep a sharp lookout for any clues the next time it happens.
    – cs95
    Commented Oct 11, 2017 at 2:59

1 Answer 1

21

This was a very subtle bug that manifested in all sorts of specific ways. It has been resolved with a deploy just a moment ago.

The root of the issue is that some work we did for Channels incorrectly handled string parsing in a specific situation of global keys (database 0 in Redis). We are architecting Channels to scale to a hundreds-of-thousands order of magnitude from the onset. Up until this point, all sites have their own database in Redis and global keys share database 0. Global keys are network-wide things like your top bar.

Now up until this point, every Site has an Id (a PK in a table). This Id is 1:1 with the Redis database number. That's how sharding has worked since the creation of the network. However, there are 2 reasons to no longer do this:

  1. Having a high database count is fundamentally expensive in Redis. The key purge that runs every 100ms loops through the databases to do such. Currently we have about 600 databases (capped at 1000), and this cost is fixed. If we up the cap to say 1,000,000, performance would instantly suffer.
  2. Redis cluster (which we'll move to when multi-datacenter replication works for our setup) no longer has databases. Each key you sync would have to carry it and it's a cost few needed...so it was dropped.

So we solve this by prefixing keys with the site ID now. This prepares us for cluster which we wanted to do anyway...so we prepped this in the work for Channels.

The specific issue here is that database 0 has always (for safety) appended the tier we're on (to ensure dev and prod keys never collide even if they somehow pointed at the same servers and ports). The code that added the key prefix as needed incorrectly thought this prefix was a site prefix and tried to strip it when necessary as well.

So, "topbar" as we call this in code. What happens with the achievements dropdown is we keep track of when you "last read" it and update that when you click it. This is stored for 30 days since we need it to both recalc how much rep to show when a rep recalc happens and know what to highlight as new when it expands. Since this is 30 days, it takes forever to expire in both L1 (local application domain) and L2 cache (Redis).

When one server (the one your loaded the dropdown from) updates the last-read, it should set it on all servers, here:

SO Code

The problem is that the message sent to the other servers in that broadcastRefresh: true was being sent, but the global prefix was being stripped before the L1 purge...so it wasn't clearing the key,

Now, those relcalcs. The network-wide recalc for NetworkRepHistory happens after a local per-site recalc as part of our network aggregator. This happens, effectively, on a random web server. One with (ding ding) a bad cache. Here's why that matters:

Network Recalc code Thankfully I thought this might be a problem and added logging back when we did it, yay!

So when the recalc happen with a bad "last read" cache on that server, we calculated the range of rep to add up incorrectly...showing you a too-big value.

Note that this also manifested in a slightly different near-realtime way when switching between Stack Overflow and Meta because Stack Overflow runs on the web01-09 servers and Meta Stack Overflow/Exchange run on web10-11, due to the same cache mismatch.

I know that's a bit long but this was a bit complicated to track down, so I wanted to share. Bonus image: Redis MONITOR screenshot from spinup debugging:

redis spinup

2
  • 1
    Do you have other fixes for Jon Skeet? Commented Oct 11, 2017 at 17:26
  • 1
    Absolutely. He causes all sorts of scaling and abnormal assumption issues, for example in SQL query plans. Commented Oct 11, 2017 at 17:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .