25

When reviewing the Close Votes, which question gets shown first to a reviewer, the one with the oldest review date or the newest? Or is it some other sort order or completely random?

2

1 Answer 1

14

I'm expanding on the Answer from Shog9 and some of my own experience.

Glossary of the terms used:

  • Close flag: User with less than 3K flags a question for closure
  • Close vote: User with more than 3K votes for a question to be closed
  • (Review) Queue: list of tasks to be reviewed by a reviewer
  • (Review) Task: an entry in the queue to be reviewed by reviewers
  • Review: A completed task where a reviewer chose an action (Leave Open, Close, Edit, Skip)
  • Filter: Tags (max 3) or close reasons to be shown to the reviewer
  • Audit: A test to see if you pay attention
  • Completed Task: either 5 close votes or 3 re-open votes where casted (or 1 dupe-hammer or a binding moderator vote)

What follows is based on observation of the system. In that sense it is speculation.

I imagine there are 3 virtual pools a Task can be in:

  • New Pool: Tasks created due to a first close flag or vote, ordered by Task creation date, descending
  • Active Pool: Tasks that have had at least one review, ordered by last review date, descending
  • Audit Pool: Tasks used as audits

A script runs every 15 minutes to refresh those pools. If a Task is completed it is removed from the pools or if the question gets closed/deleted.

When a reviewer asks for the next task the system:

  1. chooses at random which pool to get the next task from
  • the audit pool is selected approximately 2 times for every 50 tasks
  1. selects the top task from the preferred pool
  2. if this is an audit, add one of the filtered tags to the question and present the task
  3. if the reviewer skipped this task before: goto 1
  4. if the filter (either tag and/or close reason) matches the tags and close reason of the task present the task
  5. goto 1.

The above logic should describe what Shog9 said here:

The more previous reviews a given task has had, the closer to the top of the queue it'll be.
Preference is also given to the most recently queued items, particularly when filtering by tag; depending on your preferences and time of day, this might end up giving you newer (rather than more-reviewed) items.

So the review system tries to give you a balance between freshly close voted questions and ones which are actively reviewed. I expect this balance to exist so new content gets quickly closed when warranted while giving you tasks that are active should prevent votes ageing away. With enough reviewers this system is not unreasonable.

In the SOCVR room we have weekly1 events where the attendees handle the close vote queue by filtering on a pre-determined tag. That tag is chosen from this SEDE query. The query is selecting tags outside of the major 10 tags which means that we can actually clear all questions from the close vote queue. All participants notice that because the ordering/filter procedure takes more and more time if there are fewer tasks in the queue to handle.

When the room is helping in burninating a tag we ask users to close vote questions in that tag about 30 minutes before the start of an event. At the start of the event, filtering on the particular tag gives us the recent close voted questions.

1. Jon Clements forced me to add that we have two or three events every week and daily meet-ups but who cares about those details...
ad. 1. Someone got annoyed by the frequent pings in chat for the events so we now have none at all

2
  • 1
    We now have 3 weekly events to suit different time zones :)
    – user3956566
    Commented Dec 17, 2015 at 13:48
  • @YvetteColomb not anymore ...
    – rene
    Commented Jun 23, 2017 at 17:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .