11

The user listings defaults to showing users by reputation for the past 'period', where 'period' can be set to the current week, month, quarter or year.

Since January 1st, however, the view for this year is broken, in that reputation for many users has not been set back to 0 after New Year.

This is what the first page looks like right now:

year-to-date ranking

Note that 100k in 3 days is not something even Jon Skeet can manage, presumably Günter hasn't either. For the other views, the totals are correct (Günter Zöchbauer gained 700 since the new year so far).

The stackexchange.com top user rankings for the year (updated once a day) are correct still, as are the year rankings for users on other sites.

Can a dev please pull loose whatever got stuck here?

Note that last time this happened only a month view was affected and remained b0rken for the whole month. Please don't leave the year view in this state for the rest of the year. :-(

3
  • 2
    Nick found it earlier today and has been working to fix it.
    – Taryn
    Commented Jan 3, 2017 at 17:09
  • 1
    @bluefeet: \o/! Commented Jan 3, 2017 at 17:09
  • 1
    The new year has not been kind to poor ol' Nick Craver.
    – Cody Gray Mod
    Commented Jan 3, 2017 at 18:02

1 Answer 1

26

To get this out of the way: nope, not leap second related.

The yearly reputation reset failed at midnight, so I had to issue a recalc (which is simply the easiest fix) for the 1,334,741 affected users. This has just completed and the yearly totals are now correct. Bonus: the global recalc code got some love and we can now do this across all web servers with n threads to do thousands of recalcs per second. So that's fun.

It failed because this SQL update failed:

Update Users Set ReputationYear = 0 Where ReputationYear <> 0

How does it work? We have a column for each Reputation interval, Reputation____ for Today, Week, Month, Quarter, and Year. At the end of the interval, we simply reset them to 0. Why? Because keeping them updated is trivial and extremely lightweight. It's one of the very few places we use triggers in the database. That trigger looks like this:

IF UPDATE(Reputation) 
   AND NOT UPDATE(ReputationToday) 
   AND NOT UPDATE(ReputationWeek) 
   AND NOT UPDATE(ReputationMonth) 
   AND NOT UPDATE(ReputationQuarter) 
   AND NOT UPDATE(ReputationYear) 
   AND NOT UPDATE(ReputationSinceLastCheck)
BEGIN
UPDATE u
SET u.ReputationToday = u.ReputationToday + (i.Reputation - d.Reputation),
    u.ReputationWeek = u.ReputationWeek + (i.Reputation - d.Reputation),
    u.ReputationMonth = u.ReputationMonth + (i.Reputation - d.Reputation),
    u.ReputationQuarter = u.ReputationQuarter + (i.Reputation - d.Reputation),
    u.ReputationYear = u.ReputationYear + (i.Reputation - d.Reputation),
    u.ReputationSinceLastCheck = u.ReputationSinceLastCheck + (i.Reputation - d.Reputation)
FROM dbo.Users AS u
    INNER JOIN INSERTED AS i ON u.Id = i.Id
    INNER JOIN DELETED AS d ON i.Id = d.Id;
END

That trigger updates the other interval columns for any Reputation (your total) column change. It's just echoing it to the current interval. This is very cheap, and something we don't have to remember in code at all. The exclusions (AND NOTs up top) are for when we set any intervals directly (like in a recalc!), we wouldn't want the trigger to take action in those cases.

Anyway, that SQL update timed out and here we are. I've built a way to quickly fix this in the future, it's simply a risk tradeoff for extreme performance in play. Upping the timeout isn't an option, as that would (in the 30+ second case) block user requests over their retry interval at midnight and cause 503s. That's not really cool, so we have this fix we can now run (and run fast) if it happens again.

I hope that's either enlightening or gives you something to laugh at. Happy New Years!

2
  • 8
    I have to say - I like how open you are with these issues, and that you post live updates on Twitter - it's actually quite interesting. Thanks!
    – Kobi
    Commented Jan 3, 2017 at 18:49
  • 1
    Happy New Year to you too, Nick! Commented Jan 3, 2017 at 19:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .