19

Where we were at in September:

This is a follow-up to Elikill58's question from September.:

Since the new workflow of the review queues, the backlog of questions in the First questions review queue has been increasing and fails to clean all questions that appear.

Others queues, such as First answer or Triage, have lower numbers. In the case of First Questions, never stops going up.

We just reached 10k. We are reviewing posts that were posted 5 days ago... And such as we are not reviewing more than new questions are appearing, it will continue to grow...

Since then, the First Questions review queue has been changed in an attempt to fix this issue.

In October, when the fix was applied, the First Questions queue had "around 14,750 items" in it. The fix managed to remove about 4,000 of these, leaving a queue with about 11,000 items in it.

Today, the First Questions queue has 12,800 items in it

We've seen slow growth in the number of items in the First Questions queue, even with the tweaks applied in October.

But worse, for new askers, the delay in reviewing First Questions has increased from 5 days to 10 (this question was asked on 22 Jan, and the review came to the top of the queue on 2 Feb)

Reviewing on this queue, I've seen questions more than 12 days old.

The First Questions queue is supposed to highlight issues with questions asked by new users. If it takes almost two weeks for a first question to get to the top of the queue and receive feedback, what's the point in having the queue at all?

I check the profiles of new askers to see when they last logged in before I write feedback in a comment, and often their last login was the day after they asked the question.

Where to now?

I don't think the First Questions review queue is functioning as intended. The time it takes for a post to be reviewed needs to be significantly reduced.

19
  • 13
    Reviews are only effective if people are reviewing things. This applies to code reviews as it does with question reviews. If there aren't people to review things, then the queue backs up. The problem has never been the queue size or how many items were in it; it has always been how many people have been incentivized to review things.
    – Makoto
    Commented Feb 1, 2022 at 22:13
  • Yes, all questions are old. Also, this is related. The amount was at 14k, so it slowly decreasing. Even if (such as I said in multiple post) I think some things should be done, it will become better after some weeks
    – Elikill58
    Commented Feb 1, 2022 at 22:25
  • @Elikill58 the number of items in the queue was 11k after SO culled 4k items from the list in October, it's grown by 1.8k since then.
    – Joundill
    Commented Feb 1, 2022 at 22:28
  • @Joundill Yes, they just don't make those cleaning or that was already applied and the amount still increasing. But 12.5k instead of 14k, without staff cleaning it's not so bad ?
    – Elikill58
    Commented Feb 1, 2022 at 22:30
  • 9
    We solve the problem the same way we need to solve most of the world's problems: Remove a large number of humans. I'm thinking around 95% of them. Thanos was wrong. He didn't go far enough. Commented Feb 1, 2022 at 22:54
  • 27
    Something needs to be done to reduce number of questions asked on Stack Overflow
    – Dharman Mod
    Commented Feb 1, 2022 at 22:55
  • 5
    The queue has been fluctuating between 11k and 13k items for weeks. It's not "still increasing", it's just not decreasing. Commented Feb 1, 2022 at 22:57
  • 2
    This doesn't surprise me, not one bit meta.stackoverflow.com/a/411507/792066
    – Braiam
    Commented Feb 2, 2022 at 0:14
  • @user4581301 the man from the Canuckistan solves it again :D lets create a space program and send everyone to Mars.
    – bad_coder
    Commented Feb 2, 2022 at 4:18
  • 1
    @Dharman Something needs to be done to reduce the number of low-quality unanswered and unmoderated questions on Stack Overflow :) I reckon automatically providing more advice to new users and removing more items from the queue (for example new users who ask a question then don't log in again) are potential solutions
    – Joundill
    Commented Feb 2, 2022 at 4:23
  • @bad_coder We don't need Mars. We need a Mutant Star Goat. Commented Feb 2, 2022 at 7:05
  • 3
    @Dharman Annoyingly, feature requests for stuff like that have been kept on read for far too long Commented Feb 2, 2022 at 11:46
  • 3
    Just took a look at what's currently in the review queue, and, well, ... I can totally see why people don't want to drudge through that. Let alone put actual effort into it. Commented Feb 2, 2022 at 13:41
  • 1
    @MisterMiyagi your argument that people are avoiding the queue because the quality of posts is not supported by evidence, evidence that it the graph where it shows that time to complete a review were stable until that event. Again, you ask "Why people avoid the queues" it is caused by the event marked by the dramatic jump in average time to complete the reviews, namely this. See how the same week it was introduced, the graph jumped? That is your why. It doesn't need take 3 comments to see it. Just match the dates each event happened.
    – Braiam
    Commented Feb 21, 2022 at 11:23
  • 1
    @MisterMiyagi because those other queues already had some friction built in, namely you needed to select and read several options, heightening the cognitive load. Albeit I admit that FP were kind of a mess where post were allowed to pass review, when they should be moderated, the method that employed SE to slow it down, was way too heavy handed, and doesn't make good reviews more likely, just makes reviews less appealing overall.
    – Braiam
    Commented Feb 21, 2022 at 17:25

2 Answers 2

19

Yes, this is an issue. But not one we can address with changing the settings of the First Questions queue (I could allow up to 80 reviews per day on the queue, but that wont change anything on a fundamental level). We've done all that we can think of in that area for now (including as of 2022-02-23 changing the sort order in review task selection for the queue, to show newer items first, and thus have more of a chance of helping out with posts sooner after they are posted).

The bigger issue for me here isn't the number of items in the queue. It is the number of reviews being done. Monthly numbers fluctuate, but in January we got around 82.5K new First Questions tasks. Of these, around 28.5K (44%) had their Review Tasks marked as completed through reviews and around 46K (56%) were invalidated. Some of the invalidations were due to items being disqualified because they were performing well (FQ items are invalidated if they pass over a certain score threshold, or answer performance threshold, or are closed/deleted). But most were invalidated because they were in the queue for two weeks with no one to look at them.

Review work in January was done by around 1,500 reviewers. This number is down from 2,200 reviewers in September (who performed 53K reviews), and those numbers have been going down pretty consistently every month (for both unique reviewers and completed tasks per reviewer). So our reviewers are doing a tremendous amount of work (around 20 reviews per reviewer on month, on average), but there are just not enough of them to meet up with the review demands.

To me, the review size isn't the problem. If we had 50K posts in the queue, with 50K being reviewed every day, that would be just fine. The issue is that we don't have enough reviews being done, by too few reviewers. But this itself is only a symptom of potential factors leading to this:

  • Not doing enough to recruit new reviewers/publicize the existence of review queues
  • Insufficient incentives to make the time investment spent on reviewing worth it for folks
  • Insufficient quality control up front to help weed out low quality posts
  • Not enough education and guidance given to new users to help them to write higher quality questions

We have been taking a long look at many of these issues, and have been working on a new initiative (that will be previewed on Meta in a series of posts that should hopefully go up in the coming weeks) that aim to address some of these head-on. I can't give any more specifics at this time, but please be assured that issues relating to and affecting First Questions are definitely on the radar.

12
  • 4
    "The bigger issue for me here isn't the number of items in the queue. It is the number of reviews being done" if only someone has read this post in particular. Easy solution: revert back to the previous way the queues worked. They at least didn't cause panic on the userbase due reviews not being done, and achieve sensible reviewing doing something else. Someone proposed to auto filter the queues, since users take more time reviewing these and are able to do more accurate judgement.
    – Braiam
    Commented Feb 20, 2022 at 10:15
  • 1
    I agree with Braiam. Also, the "too few reviewers" should also be asked about why people left the review, why they stop to review ? Such as there is at least 700 reviewers that stop (without counting new reviewers that take place or left one). But, for me it should be done at the first step: managing question asked (such as proposed in some feature request, with different proposal)
    – Elikill58
    Commented Feb 20, 2022 at 10:49
  • 7
    Here's the actual reason I don't review in the first questions queue: because tag filtering is broken. As we speak I cannot review, because my filter for two of the most popular tags is returning no results.
    – Laurel
    Commented Feb 20, 2022 at 13:30
  • 9
    @Laurel I hadn't been aware of that (unfortunately, no one has yet marked it as status-review). I can repro and think that I have found a solution for that issue. Thanks for reporting, sorry for the delay. Commented Feb 20, 2022 at 14:23
  • 7
    Insufficient incentives is definitely the reason for not doing reviews for me - imaginary Internet points is not enough. Personally, I've more or less boycotted reviews for many years and that's because there is just no way that I'm doing boring, unpaid chores for this company in my free time, that's definitely where I draw the line of volunteer contributions. I'd much rather spend such volunteer time working for non-profit organizations.
    – Lundin
    Commented Feb 21, 2022 at 8:58
  • 2
    @Lundin currently you don't even get imaginary Internet points for reviews. Just badges. And I am happy to hear about your volunteering. Thanks, wherever it is going to. (I am hearing from you that rep wouldn't make a difference either, right). Commented Feb 21, 2022 at 9:01
  • 5
    Here is my opinion that no one asked for. Even if there were generous incentives, I still wouldn't put my foot in the review queue. Why? Unlike answering programming questions which is arguably fun, this is repetitive unpaid work. I have better things to do, especially during this covid-19 era, where my sanity, on working from home 100% of the time, is on the brink of collapse. And even if all that was not an issue, after the controversial firing of mods, I have lost trust and naturally distanced myself from SO. The SO today is different than how it was in the past. Commented Feb 21, 2022 at 11:06
  • @kemicofaghost we know queues are a chore, so the minimum it could be done is to make them as painless as possible. When the workflow of radio buttons + buttons were introduced, that was too much.
    – Braiam
    Commented Feb 21, 2022 at 11:26
  • 4
    Personally, I enjoy review queues, but as I learned and got better at moderating, my number of actions per task increased. In FQQ it is not uncommon for me to edit to fix things, retag, vote up/down, and vote to close. The problem is that I can run out of all of my votes for the entire day (both up/down and closure) in just a single queue. For me, the primary reason I stopped completing as many tasks is because I didn't like having to choose between being able to vote on content that is genuinely useful to me, and voting on content I would never have seen were it not for review queue.
    – Henry Ecker Mod
    Commented Feb 21, 2022 at 16:26
  • 1
    "to show newer items first" It would be interesting to know whether that means some items just never get reviewed, because they are hidden behind an ever increasing wall of new items. (That's not meant to imply it's necessarily bad.) Commented Feb 23, 2022 at 9:20
  • 3
    @MisterMiyagi it is already the case that around half of items done get reviewed, because they would time out before someone would look at them (or folks would just skip them). This means that when you review, you will review something fresher. It also means that you will have a higher chance of the OP responding to comments, and have a higher chance of running into items that are just fine (and would normally be naturally invalidated because of votes received on question or answers). Commented Feb 23, 2022 at 10:54
  • Many posts in the FQQ have reproducibility issues, contain thanks fluff or other trivial things that can be avoided. Many review tasks for these are pointless, as they can (should) be resolved by proper documentation and onboarding procedure, giving them a chore-like feeling. Reviewers are given test tasks to see if they are paying attention. Why not query posters when they post their first question? Hey, there is an image in your post, is this an image of code? If the quality of the questions can be improved, there is no need for a separate Triage and FQQ. Commented Aug 14, 2022 at 12:12
16

The review queue model is broken by design.

As anyone working with computer science knows: queues is only a valid model if they get emptied on regular basis. It's a method of handling peaks of input, so the system can process it over time. But if the system isn't fast enough to process all input, no amount of queues will help - it is doomed to buffer overflow and eventually fail.

SO reviews have been limping along over the years and they worked ok while there was still a high veteran user per new user ratio. However, the company has consistently since 2014 somewhere been working to increase traffic by sacrificing quality for quantity. This tilts the mentioned user ratio as more new users keep pouring in. It affects all of the site, but it's particularly evident in the review queues. At the same time, more and more review queues are getting added: there's 8 of them now.

This doesn't add up, we simply can't have review queues if they never get emptied.

5
  • This argument would have more weight, if there wasn't a simple piece of evidence that simply doesn't support it: this graph shows the average time it takes for a review to complete per week. Note how it's stable until a particular event, and that trend is still there today.
    – Braiam
    Commented Feb 21, 2022 at 11:20
  • 6
    @Braiam "Look here a graph" isn't much of evidence either. If you want evidence, then simply go here stackoverflow.com/review several times per day. If you see the queues each hitting zero items at some point, the system is working. If not, then this answer applies.
    – Lundin
    Commented Feb 21, 2022 at 11:27
  • @Lundin I guess the first part of your answer applies in all cases (queues only work when they are emptied) but the second part does not necessarily apply (the problem is the ratio between veteran and new users). According to Braiam the problem is the interface, which could indeed be part of the problem.
    – Marijn
    Commented Feb 21, 2022 at 11:34
  • 2
    @Marijn Of course everything that slows down reviews add further problems on top of it all. But it's a fact that queues are either filling up or emptying - that they would remain on a steady fixed number of items is highly unlikely. If the number of items ending up in a queue per day exceeds the number of complete reviews, the queues will just keep growing until something gives in.
    – Lundin
    Commented Feb 21, 2022 at 11:38
  • 3
    @Lundin true, but the argument is made that with the old interface the number of reviews completed per day was equal to the number of newly added reviews per day. I'm not sure if that is true but a correlation between the interface changing and the queue size starting to increase does suggest that your analysis of why the queues are filling up is not the whole story.
    – Marijn
    Commented Feb 21, 2022 at 12:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .