To get this out of the way: nope, not leap second related.
The yearly reputation reset failed at midnight, so I had to issue a recalc (which is simply the easiest fix) for the 1,334,741 affected users. This has just completed and the yearly totals are now correct. Bonus: the global recalc code got some love and we can now do this across all web servers with n
threads to do thousands of recalcs per second. So that's fun.
It failed because this SQL update failed:
Update Users Set ReputationYear = 0 Where ReputationYear <> 0
How does it work? We have a column for each Reputation interval, Reputation____
for Today
, Week
, Month
, Quarter
, and Year
. At the end of the interval, we simply reset them to 0
. Why? Because keeping them updated is trivial and extremely lightweight. It's one of the very few places we use triggers in the database. That trigger looks like this:
IF UPDATE(Reputation)
AND NOT UPDATE(ReputationToday)
AND NOT UPDATE(ReputationWeek)
AND NOT UPDATE(ReputationMonth)
AND NOT UPDATE(ReputationQuarter)
AND NOT UPDATE(ReputationYear)
AND NOT UPDATE(ReputationSinceLastCheck)
BEGIN
UPDATE u
SET u.ReputationToday = u.ReputationToday + (i.Reputation - d.Reputation),
u.ReputationWeek = u.ReputationWeek + (i.Reputation - d.Reputation),
u.ReputationMonth = u.ReputationMonth + (i.Reputation - d.Reputation),
u.ReputationQuarter = u.ReputationQuarter + (i.Reputation - d.Reputation),
u.ReputationYear = u.ReputationYear + (i.Reputation - d.Reputation),
u.ReputationSinceLastCheck = u.ReputationSinceLastCheck + (i.Reputation - d.Reputation)
FROM dbo.Users AS u
INNER JOIN INSERTED AS i ON u.Id = i.Id
INNER JOIN DELETED AS d ON i.Id = d.Id;
END
That trigger updates the other interval columns for any Reputation
(your total) column change. It's just echoing it to the current interval. This is very cheap, and something we don't have to remember in code at all. The exclusions (AND NOT
s up top) are for when we set any intervals directly (like in a recalc!), we wouldn't want the trigger to take action in those cases.
Anyway, that SQL update timed out and here we are. I've built a way to quickly fix this in the future, it's simply a risk tradeoff for extreme performance in play. Upping the timeout isn't an option, as that would (in the 30+ second case) block user requests over their retry interval at midnight and cause 503s. That's not really cool, so we have this fix we can now run (and run fast) if it happens again.
I hope that's either enlightening or gives you something to laugh at. Happy New Years!