This was a very subtle bug that manifested in all sorts of specific ways. It has been resolved with a deploy just a moment ago.
The root of the issue is that some work we did for Channels incorrectly handled string parsing in a specific situation of global keys (database 0 in Redis). We are architecting Channels to scale to a hundreds-of-thousands order of magnitude from the onset. Up until this point, all sites have their own database in Redis and global keys share database 0. Global keys are network-wide things like your top bar.
Now up until this point, every Site has an Id (a PK in a table). This Id is 1:1 with the Redis database number. That's how sharding has worked since the creation of the network. However, there are 2 reasons to no longer do this:
- Having a high database count is fundamentally expensive in Redis. The key purge that runs every 100ms loops through the databases to do such. Currently we have about 600 databases (capped at 1000), and this cost is fixed. If we up the cap to say 1,000,000, performance would instantly suffer.
- Redis cluster (which we'll move to when multi-datacenter replication works for our setup) no longer has databases. Each key you sync would have to carry it and it's a cost few needed...so it was dropped.
So we solve this by prefixing keys with the site ID now. This prepares us for cluster which we wanted to do anyway...so we prepped this in the work for Channels.
The specific issue here is that database 0 has always (for safety) appended the tier we're on (to ensure dev and prod keys never collide even if they somehow pointed at the same servers and ports). The code that added the key prefix as needed incorrectly thought this prefix was a site prefix and tried to strip it when necessary as well.
So, "topbar" as we call this in code. What happens with the achievements dropdown is we keep track of when you "last read" it and update that when you click it. This is stored for 30 days since we need it to both recalc how much rep to show when a rep recalc happens and know what to highlight as new when it expands. Since this is 30 days, it takes forever to expire in both L1 (local application domain) and L2 cache (Redis).
When one server (the one your loaded the dropdown from) updates the last-read, it should set it on all servers, here:
The problem is that the message sent to the other servers in that broadcastRefresh: true
was being sent, but the global prefix was being stripped before the L1 purge...so it wasn't clearing the key,
Now, those relcalcs. The network-wide recalc for NetworkRepHistory
happens after a local per-site recalc as part of our network aggregator. This happens, effectively, on a random web server. One with (ding ding) a bad cache. Here's why that matters:
Thankfully I thought this might be a problem and added logging back when we did it, yay!
So when the recalc happen with a bad "last read" cache on that server, we calculated the range of rep to add up incorrectly...showing you a too-big value.
Note that this also manifested in a slightly different near-realtime way when switching between Stack Overflow and Meta because Stack Overflow runs on the web01-09
servers and Meta Stack Overflow/Exchange run on web10-11
, due to the same cache mismatch.
I know that's a bit long but this was a bit complicated to track down, so I wanted to share. Bonus image: Redis MONITOR
screenshot from spinup debugging: