25

Lately, we've been making adjustments to how the review queue indicator works. Sometimes when we work on some esoteric feature for a long time, we start to lose track of the ultimate goal. In this case, the goal is to direct more people to the review queue so that more posts are reviewed so that overall quality of visible content on the site is increased. All other things being equal, the site benefits from a diverse group of reviewers who have been around the site long enough to earn the privilege. So we're going to focus on reviewers per week:

Reviewers per week on Stack Overflow

Usually, the novelty effect complicates analysis, but we can use it to see when major changes occurred. The first major change visible on this chart came in December of 2013 when the original new top bar was introduced. The most important change for our purposes is that we started to show the total number of pending reviews (minus the close queue) next to the review link. After the initial spike, the number of reviewers quiesced to a steady increase that mirrors the overall rate of participation around that time. Unfortunately, the number of outstanding review tasks has been a source of confusion ever since. Regrettably, the number is cached and includes reviews that the user might already have handled.

The next increase came around the time we introduced the new profile and activity pages. I can't find anything else from that time period—the spike occurred the week of April 15, 2015. Presumably the new reviewers were inspired by the next badge progress bar. Once people earned the relevant badges (or stopped looking at their own activity pages) the number of reviewers tapered off again.

In February of 2017, we released a version of the top bar that replaced the word "review" with an icon. This time, however, the number of reviewers settled below what we'd have expected. The most probable explanation is that icons are less readable than text.

In April, we replaced the number with light-up indicator that was triggered when the queues had more than the median number of tasks. Unfortunately, this resulted in the indicator being on about half the time, which is liable to cause alert fatigue.

After the top bar was released the entire network in October, we changed the indicator trigger to the current "danger-level" scheme, which warns approximately one hour in ten. We also introduced the red dot notification. This change boosted usership beyond what we've learned to expect from simply changing the interface. Until the seasonal drop off typical of December, weekly reviewers seemed to be returning to a much higher level than after previous changes. Since we didn't adjust the thresholds for much lower levels of site activity, it's likely the indicator was lit up much less often over the winter break than it would otherwise the year. So the drop could be exaggerated by people not getting warned as much as normal.

Not relevant to this site, but we recently tested a more aggressive indicator which we'll be rolling out to smaller sites this week. But it does bring me to another way to look at the review queue: how long do tasks have to wait to get cleared? And looking at the periods before and after the the most recent change, it's obvious that more hands make lighter work:

epoch  reviews first quartile median minutes pending third quartile 
------ ------- -------------- ---------------------- -------------- 
before 1429699              6                     29            111
after  1493478              4                     18             63

The "after" period includes the December lull, so it's significant that the number of (non-skip) reviews has increased. Especially when it comes to the close queue, faster reviews are better than slow reviews. So the new indicator is a real improvement when looking at the overall health of the site.

I know there are still some annoyances for individual users. Perhaps you prefer a different icon or a text link. There's a chance of false positives for people who do a lot of reviews (especially on smaller sites). We'll be fixing bugs that come up of course, but the overall design seems to be working as intended on Stack Overflow. Is there something I'm missing in my analysis?

15
  • For the "minutes pending" can we get 25-75 percentiles too?
    – Braiam
    Commented Jan 25, 2018 at 23:21
  • @Braiam: I'll update my query and report back later today or tomorrow. SQL is so obnoxious about this. :-(
    – Jon Ericson Staff
    Commented Jan 25, 2018 at 23:23
  • docs.microsoft.com/en-us/sql/t-sql/functions/…?
    – Braiam
    Commented Jan 25, 2018 at 23:27
  • @Braiam: Updated. The problem with the percentile functions is that they are analytic functions, which are awesome but make the query more complicated than usual. At any rate, median is a much better stat than average since review duration tends to have a long tail.
    – Jon Ericson Staff
    Commented Jan 26, 2018 at 0:07
  • Well, actually with this we see that it actually improved the higher end of the spectrum by a lot, compared to before, while it didn't have a dramatic improvement on the lower end.
    – Braiam
    Commented Jan 26, 2018 at 2:36
  • 4
    Where's the freehand circles? I feel cheated.
    – user4639281
    Commented Jan 26, 2018 at 4:31
  • 3
    @TinyGiant: Have you tried holding the post up to a light bulb?
    – Jon Ericson Staff
    Commented Jan 26, 2018 at 4:36
  • 3
    Yes, please, one word under each icon on desktop layout of the bar is a recurring request: the only reason found why SO designers didn't want it was because "Achievements" was too long. So please figure out a shorter naming or make a community poll on what icon wording they want ("Achieve.", "Trophies", "Badges", "Wins", "Achievements", etc.) and put back the word "Review" after that.
    – Cœur
    Commented Jan 26, 2018 at 5:21
  • more hands make lighter work:, so do everything you can to encourage more hands. Which conversely means, do everything you can to not discourage more hands.
    – Stephen Rauch Mod
    Commented Jan 26, 2018 at 6:58
  • Are these increases related to more users being able to review? Or to more reviewers? (Did you take the increase of the number of reviwers into account?) Since 2012 many more users has now the hability to review and there are many more things to review too.
    – llrs
    Commented Jan 26, 2018 at 9:45
  • 1
    The most significant rate changes to reviews are clearly from hats. In addition, elections significantly alter review rates and I don't see that in this analysis.
    – Travis J
    Commented Jan 26, 2018 at 17:23
  • @Llopis: Yes, the baseline increase after the first spike (from the black top bar) is mostly due to more reviewers and tasks to review. It's very likely active reviewers as a percentage of users with the appropriate privilege has fallen over the years.
    – Jon Ericson Staff
    Commented Jan 26, 2018 at 17:39
  • 1
    @TravisJ: Those aren't really things we can count on to drive reviews, I think. ;-) In any case, hats clearly don't counteract the normal holiday lull in terms of number of reviewers. I haven't looked, but I'd guess any effect of elections would be lost in the noise.
    – Jon Ericson Staff
    Commented Jan 26, 2018 at 17:44
  • @JonEricson the obvious solution is to have hats all year long in the review queues.
    – user4639281
    Commented Jan 27, 2018 at 1:01
  • @JonEricson I think the graph needs to plot that too or divide the number of reviewers by the total amount of available reviewers and show the amount of work to do. BTW I didn't receive the notification of your reply :\
    – llrs
    Commented Jan 27, 2018 at 11:06

1 Answer 1

18

As this is tagged as discussion... here are a few random thoughts of mine relating to my personal experiences with the review queues.

Personally, I've lost the motivation to do reviewing via the queues. Why? Mainly because of significant amount of bad questions.

Historically I started reviewing when it was unlocked and I as a user wanted to gain review queue badges. This motivation faded over time. I guess I next stepped it up when the 10k tools became unlocked and I felt I could make a bigger impact... but again... that motivation faded, plus I felt the 10k tools could do with some love.

Granted, the red dot did make me click in to the review queues a couple of times so I participated in reviewing again briefly, but again I fear that my brain is simply going to filter out the red dot over time. I just don't have the motivation to visit the review queues regularly.

I still try to do my part, but I tend now to focus on new posts in tags that I'm interested in. My method of "reviewing" in some way, is doing it on questions and answers as they are posted, where there are people active around the posts I'm looking at.

In an ideal world, the quality of the content posted would be better, which would lead to fewer people actually having to do reviews as opposed to there being so many reviews to do that you have to sell the review queues to users.

4
  • 6
    Similar point in response to the question about DAG priorities.
    – jscs
    Commented Jan 26, 2018 at 13:35
  • @JoshCaswell thanks, hadn't seen that particular answer but yeh, similar sentiment.
    – Tanner
    Commented Jan 26, 2018 at 14:02
  • This is a very good point. As @JoshCaswell points out, DAG as been working on question quality. We ran a promising experiment using question templates. Based on those results (which I hope to write up soon) we're working on a new ask page that will guide inexperienced askers to either ask better questions. That said, I think it's fine if some people ignore the dot. (Full disclosure: I do.) By the way, are you aware of filters?
    – Jon Ericson Staff
    Commented Jan 26, 2018 at 16:42
  • @JonEricson yes I am
    – Tanner
    Commented Jan 26, 2018 at 16:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .