Update: This has been completed. A fix for the backend tooling and a cleanup pass to issue missing grants has been run. 1,009 accounts have just received the bonus.
The bug here is an interesting consequence of our database setup. In short: we query some data from the primary, and some data from the replica. Association bonuses are one of the mechanics (along with some things like badges) that utilize the replica.
Now the way this works is, when you trip the association bonus threshold, we grant it on the other sites - this involves a query. Unfortunately, the circumstances surrounding the reputation recalc basically broke the assumption that the replica server is reasonably in sync. It was nowhere close.
This is my fault. When we planned out how the recalc would work technically, I failed to factor in that the recalc itself doesn't just update the RepHistory
table (a known quantity), but also updates a field called TargetRepChange
on the Posts2Votes
table (the source of truth). This is a good thing, as everything is in sync, but since the rules changed, the data resulting from those rule calculations was updated as well. This was dozens of millions of updates I didn't factor into the costing.
The amount of transactional data we were replicating between servers was more than double what I had anticipated - this led to growth in the translation logs and a delay in replays (called "redos") on our SQL Server replicas. When the functionality to grant a bonus kicked in...it didn't think you deserved one, based on the stale data it had. That sucks, and I apologize! Luckily, this is totally fixable for all users affected (including any affected for any reason at any point in the past).
Unfortunately, the mechanism we have to grant the bonus for anyone missing it has been outgrown (e.g., it's timing out - I just tried it). It has 2 modes: “simulation” and "do the grants" mode. I need to fix the simulation for streaming output so I can sanity check it before smashing go (although I'm ~99% confident it'd work fine...let's not make a bigger mess).
I plan to fix this up Monday and will get all missing association bonuses granted. I'll update this post once it's done :)