53

In the spirit of science, we're going to run a test on the Low Quality Review Queue.

Current Problem

After pulling some stats about how the LQRQ is performing, we confirmed that review is handling a ton of posts, but it's not able to process them fast enough. The review queue gets 60 minutes to handle the post before it also heads to the moderator queue. If it's not completely reviewed in the 60 minutes, then either the moderators or review can handle the post.

In the last 30 days, 11,987 flags have been sent into the review queue. Of these:

  • the community handled roughly 22% of them
  • 4.5% were completed in less than 60 minutes
  • 3% were cleared in 60 to 120 minutes
  • 3% were processed in 120 to 180 minutes
  • 2% were handled in 180 to 240 minutes after the flag was in review
  • and 9% were processed in over 240 minutes

What's Changing?

Currently, we require a post to accumulate 6 Recommend Delete & Delete reviews. Over the next 30 days, we're going to adjust this to 4 Recommend Delete & Delete votes to dequeue a post.

We don't want things waiting in review for hours to be handled by the community, so we're testing if lowering the number of recommend delete and delete votes will move things along.

What Happens At The End Of The Test?

When the 30 days are up, I'll rerun the queries to see if the review queue is able to dequeue posts before the moderators can handle them. Ideally, we'd see an increase in the number of posts being handled by review during the 60 minutes before they hit the moderator queue. After some sanity checking on the accuracy of the review queue at 4 votes, we'll specifically look for the following:

  • If we see an 20% increase in what the review queue is handling, then we will assume this is a success and we'll consider leaving the new settings in place.

  • If we don't see the 20% increase in what review is handling, then we'll have to reevaluate the settings and see what steps to take next.

What's Next?

Just do what you always do. If you review in the Low Quality queue, then continue as you normally would. We expect most users will continue their normal patterns during the test.

I'll be back in 30 days to report what happened.

14
  • 2
    Does this apply only to Stack Overflow? In my experience, the Super User LQP queue almost never handles posts in time.
    – Ben N
    Commented Jan 30, 2017 at 19:49
  • 3
    @BenN This is only on Stack Overflow.
    – Taryn StaffMod
    Commented Jan 30, 2017 at 19:50
  • 27
    Reducing the number of actions needed is going to increase the amount of content reviewed; I don't see why you'd need any experiments for that. Surely the thing you need an experiment for is whether or not 4 reviews will maintain a sufficiently high accuracy of reviewers, while improving the amount of content reviewed. If reducing the number of reviewers results in more inappropriate deletions (beyond some margin that is considered unacceptable, which should probably be discussed in advance), then surely that would be an unsuccessful experiment, regardless of the number of posts reviewed.
    – Servy
    Commented Jan 30, 2017 at 19:52
  • If you graph the number of flags/day, does this last weekend stand out? (A serious question.)
    – Jongware
    Commented Jan 30, 2017 at 19:52
  • @RadLexus On SO Main there were significant less flags. Root cause still unknown. Maybe a major sport event or so....
    – rene
    Commented Jan 30, 2017 at 19:55
  • 2
    @Servy Yes, reducing the number of actions should increase the amount of content, which is why we're also going to be checking the accuracy of review with this test via some sanity checking.
    – Taryn StaffMod
    Commented Jan 30, 2017 at 19:57
  • 8
    @bluefeet I'd like to see that spelled out in the proposal, going over how much testing of it is being done, and what the guidelines are for acceptable/unacceptable changes, because if there is going to be a problem with this change, then that would be it. It would also be worth looking at the stats for current reviews, specifically, how many reviews attracted 4 or 5 delete suggestions but didn't end up being deleted, as those are reviews that would now go the other way. The total number of reviews processes seems like it should be more of an afterthought as far as metrics for the test go.
    – Servy
    Commented Jan 30, 2017 at 20:00
  • I did reviews today and it felt a bit different knowing that less recommend-deletions are needed. More relaxed, less focused. Which is probably not good given that this is about deletion of the answers. You could probably somehow compensate that by increasing amount of audits
    – gnat
    Commented Jan 31, 2017 at 10:08
  • @bluefeet What do you mean by "dequeue"? Will a post that gets 4x recommend delete still be deleted or just left there with the flag handled but the post still on the site?
    – Magisch
    Commented Feb 1, 2017 at 7:22
  • @Magisch it gets deleted, I checked (unless positive score or accepted, as per usual rules)
    – gnat
    Commented Feb 1, 2017 at 10:28
  • 2
    @Servy I pulled some more numbers, there were about 2k posts in the past 30 days that hit review and received >= 4 delete/rec. delete votes. Of these 91 were not deleted by either review or the mods, so about 4%. Part of the validation will be by checking the posts that hit review, gained delete votes but were not deleted to see if we're missing things that should be removed.
    – Taryn StaffMod
    Commented Feb 1, 2017 at 15:55
  • 1
    @bluefeet I do not have stats, but after the new top bar (fixed), it seems that more people (new people) are doing LQP reviews, maybe worth running some stats. Commented Feb 14, 2017 at 16:55
  • 1
    30 days are over. Results please?
    – Pang
    Commented Mar 3, 2017 at 10:01
  • 1
    @Pang [status-completed]
    – Bhargav Rao Mod
    Commented Mar 3, 2017 at 19:57

1 Answer 1

18

Our 30 day test has ended, and as promised here are the results.

Results:

In the last 30 days, there were 22,517 flags that were sent into the Low Quality Review Queue. Of these:

  • the community handled about 48% of them
  • 27% were completed in less than 60 minutes
  • 10% were cleared in 60 to 120 minutes
  • 4% were processed in 120 to 180 minutes
  • 2% were handled in 180 to 240 minutes after the flag hit review
  • and 4% were processed in over 240 minutes

We had a noticeable increase in flags being handled faster by review, instead of lingering for several hours. If review is able to keep up with what is being sent in its direction, then that minimizes what the moderators have to handle.

Additionally, I checked to see how many flags were garnering 3 or more delete votes in review but were not deleted by the community. Of the 10,929 flags handed by the community in review, 169 received delete votes and were not deleted by either review or by a moderator. That's approximately 1.5% of flags which were not deleted after receiving votes to delete, this is down from the 4% we had in the 30 days prior to the test.

During the 30 day test, we saw a drop in invalidated reviews (aka mod-handled tasks) to what they were before the past election and the number of deleted items increased:

Review Breakdown

This is exactly the outcome we were hoping for with the test. We wanted to see review be able to handle items faster, taking the burden off of the moderators but still have accurate reviews taking place.

One observation from the test is there was a small drop in the Looks Good and in the Invalidated - Declined categories. Which means that review is deleting a small number of things that the moderators most likely would have declined. However, we did some sanity checking on those items and considering the purpose of Low Quality Review Queue is to remove junk, we didn't see anything that wasn't so fantastic that it needed to stick around. We will be monitoring this to see if we need to make any further adjustments.

What's Next?

As I mentioned in the original post, if we saw an increase of at least 20% in what the review queue was handling, we would leave the new threshold in place. Well, we exceeded our expectations, so we're going to keep the 4 Recommend Delete & Delete votes on the Low Quality Review Queue.

We'll still be watching to make sure the community can handle what's being thrown at it and adjust as needed in the future.

13
  • 7
    As I mentioned when you first posted the question, of course the number of items reviewed is going to go up if you decrease the number of reviews needed per item. The question is whether or not the accuracy of the reviews remains high, and there's nothing in here that even mentions the accuracy of the reviews, which is really the only thing worth measuring in a test such as this.
    – Servy
    Commented Mar 3, 2017 at 19:47
  • 1
    Small mistake: legend refers 6 graphs, but there are 5. I suppose "Close" shouldn't be here? Also, I may not be reading the graph right, but isn't the green "Delete" line decreasing after Februrary 2017 (start of the test) while you're saying and the number of deleted items increased? Ah, it would be interested to see the percentage of task completed, before the change, with regard to duration (less than 60 min, etc.) so that it can be compared with after.
    – Tunaki
    Commented Mar 3, 2017 at 19:49
  • 1
    @Tunaki The Close item appears along the left side of the chart (I'll double check what that's from). As far as delete (the green line) there is a spike in the point right after Jan 2017 (which is the test month of Feb) there is a noticeable increase in that line - the following decrease is for March which had only about 1.5 days in it.
    – Taryn StaffMod
    Commented Mar 3, 2017 at 19:58
  • is the setting updated from 6 to 4 only at Stack Overflow or this is network wide change?
    – gnat
    Commented Mar 3, 2017 at 20:01
  • 1
    @gnat Stack Overflow only.
    – Taryn StaffMod
    Commented Mar 3, 2017 at 20:04
  • 1
    The 22,517 flags are not distinct per post correct?, if not posts can you give an idea on how many post entered the queue in 30 days Commented Mar 3, 2017 at 20:04
  • @PetterFriberg There were 12,468 posts associated with those flags.
    – Taryn StaffMod
    Commented Mar 3, 2017 at 20:06
  • 4
    @bluefeet What was the reason for the jump in the number of the posts deleted by mods in May and Jun 2016, and the subsequent fall? The remaining jumps clearly correspond to elections i.sstatic.net/OY6gB.png. Apart from that the change has been gradual.
    – Bhargav Rao Mod
    Commented Mar 3, 2017 at 20:16
  • 4
    @Servy we checked manually around 4500 post in 30 days (it's a rough number) and we found 3 "answers" that according to us where incorrectly deleted by review queue. Moderator has now un-deleted those answers, 1 was a clearly incorrect answer (but still an answer), the other a link but if you strip link still an answer, the last one a good suggestion (what if you try..) Commented Mar 3, 2017 at 20:19
  • 1
    @Servy I've added a bit about something we noticed. We did see a corresponding drop in Looks Good & Invalidated - Declined which means that review is deleting stuff that the mods might have declined. But these were minimal, we spot-checked the items, and did not find absolutely fantastic that shouldn't have been removed for being low quality.
    – Taryn StaffMod
    Commented Mar 3, 2017 at 20:43
  • 1
    @BhargavRao I don't know offhand, I'd have to dig into the data to see if I could find something.
    – Taryn StaffMod
    Commented Mar 3, 2017 at 20:44
  • 2
    Since 4 is still reasonably accurate, are you going to try 3 next?
    – davidism
    Commented Mar 3, 2017 at 23:58
  • 2
    @davidism not any time soon.
    – Taryn StaffMod
    Commented Mar 4, 2017 at 2:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .