41

Since the new workflow of the review queues, the backlog of questions in the First questions review queue has been increasing and fails to clean all questions that appears.

Others queues, such as First answer or Triage, have lower numbers. In the case of First Questions, it never stops going up.

We just reached 10k. We are reviewing posts that were posted 5 days ago... And such as we are not reviewing more than new questions are appearing, it will continue to grow...

Why does this have to be fixed?

  • As I explained, it will continue to increase. So it will be worse if nothing is done.
  • It will lose its utility because users will post a second/third question before the first one will be reviewed.

But how can we fix it?

These are a few suggestions and ideas:

  1. Remove questions with accepted answers. A question that has been answered and seems to be fixed can be considered as well asked and complete, because a reply on-topic comes (for example, this one or this). We also could imagine extending this to questions with well-received answers (no flag and with upvote).

  2. Increase the limit for daily reviews from 40 to 80 when there are over 1k pending reviews. This is a solution that needs community effort so it's not 100% certain that this will fix the issue.

  3. Use Triage data to remove questions from the First questions queue such as this one which is also a first question.

  4. Upvote + edit of question = good question. This solution need more explanation:

    When a question is upvoted, people think that it's a good one which has to be answered. If it's edited, it means people take more time to perfect their question. So when it's a moderator or community edit, it's really relevant because instead of close/mark as duplicate, they decide to upgrade it.

    But that's clearly a dangerous solution because a lot of questions can have those conditions and can also be bad or need action. I think it can be useful in the final case, if anything else works.

  5. Expose the first questions review queue to more users. It's an idea. Even I think it's not a good one. This can make more people interested in reviewing, but I don't know the real viability of this.

Finally, I think 1/2/3 can be good. And maybe just with the first it will really help.

19
  • 34
    40 reviews is already hard work, very few people are going to do 80.
    – greg-449
    Commented Sep 14, 2021 at 12:37
  • 22
    "A question that have been answered and seems to be fixed can be considered as well asked and complete" No, it can't. "When a question is upvote, people think that it's a good one which have to be answer." That someone upvoted a question may mean that they found it useful but that doesn't mean the question was good or even suitable for SO. Commented Sep 14, 2021 at 12:41
  • 10
    Related: What topics can I ask about here?, What types of questions should I avoid asking?. For example, if I ask for the best software to do something, someone may also want to know it and upvote it and another person may even write an answer linking to some external resource. Yet the question would not be suitable for SO. Commented Sep 14, 2021 at 12:47
  • 12
    Another good example are typo questions (like misspelled variable names, extra semicolon, missing bracket etc.) which often receive quick answers because it's easy to pint out the typo. Yet these have their own close reason "typo or not reproducible". The answer may well be accepted (because it does solve the OP's problem) and the question may also receive an upvote (to help the OP get enough rep to upvote the answer). Commented Sep 14, 2021 at 12:51
  • 6
    FWIW 60 reviews a day limit has been tested a while ago (see this announcement). Results didn't look very promising
    – gnat
    Commented Sep 14, 2021 at 12:59
  • 1
    I see and I understand. But I'm confused: in my mind lot fo question that should not be on SO are flagged, so even if it's a bad question with answer, it will be removed. And so, you think we cannot use those data to clean this queue ?
    – Elikill58
    Commented Sep 14, 2021 at 12:59
  • 3
    There is a chance that less people are reviewing due to the recent change to the review queues. They may just wait until most of the bugs are fixed. I expect those reviewers are likely to start reviewing again. Of course there may also be some that are unhappy with the changes and decided to not review anymore at all.
    – Scratte
    Commented Sep 14, 2021 at 13:19
  • 3
    Fewer first questions would be one possibility. Maybe a more prominent search function could help there. Commented Sep 14, 2021 at 16:34
  • 11
    Personally I stopped reviewing FQ in part because of the changes and because it feels pointless. "Share feedback" comments are anonymous for whatever reason so it's impossible to follow up with the poster, if they even respond to a bot, even if comments aren't anonymous the turnout of askers and answerers actually caring and editing their posts seems terribly low. And that's for new posts, I just can't be motivated to give feedback on a 6 days old post where the poster is already lost to time. Either this gets improved in some way or the posts might need to age away after like 2 days.
    – LW001
    Commented Sep 14, 2021 at 18:17
  • 2
    To @Scratte's point, do we have data to help us understand what's changed? The threshold for approving questions is the same. The question eligibility is the same. Can we verify that there are fewer reviewers today? Or that those reviewers are reviewing fewer questions? Alternatively, might there be a bug that's preventing questions from being properly removed after review? Or maybe a timeout metric changed? It's odd to me that the First Answers queue frequently gets cleared. There have always been fewer answers than questions, but the First Posts queue used to regularly end up below 100. Commented Sep 14, 2021 at 19:04
  • 4
    @JeremyCaney There's a hint in "Other action" in "First answers" doesn't fully complete the review that sometimes it takes more than one reviewer to complete a task now. Depending on the action taken. This will certainly slow the clearing of the queue down.
    – Scratte
    Commented Sep 14, 2021 at 19:07
  • 5
    I don't think there is any improvement that will fix this issue. There is just too many Q's coming in for the volunteer pool to keep up with. Manual review is a horribly scaling system and we really need some automation to fix this. That gets a lot of push back though as they don't want to just not accept a post that could, with some work, become a good post. If the user never gets to post it, they just go somewhere else that will let them post it. It also doesn't help that we have plenty of people that can review, but don't for whatever reason. Commented Sep 15, 2021 at 14:45
  • 3
    Reviewing can be soul-destroying after four or five reviews in a day. Commented Sep 15, 2021 at 17:30
  • 2
    @NathanOliver: "we really need some automation to fix this": Certainly, but it should be much better than Quora's moderation bots (they have unspecified IQ). Commented Sep 15, 2021 at 23:37
  • 3
    They just fixed the issue where “Other Action” doesn’t clear the review: meta.stackexchange.com/a/369947/293308. I am assuming it didn’t go back and clear reviews that already have that option but it should help going forward.
    – BSMP
    Commented Sep 17, 2021 at 19:44

7 Answers 7

20

I have just finished a number of updates to the First Questions queue invalidation/eligibility criteria. All are aimed at removing items from the queue that we think have a higher chance of being ok, leaving the queue with a higher concentration of first questions that are really in need of review.

At the time when I was evaluating these changes, there were around 14,750 items in the queue. Next to each item in the list below, I will include in parentheses the approximate number of items that would be invalidated or excluded from First questions due to the given change. Here they are:

  1. MinOwnerRepToInvalidate changed from 250 to 125, MaxOwnerRepWhenAdding changed from 225 to 125. These are rep levels for the question owner at which the question is either invalidated or not added to the queue. (~500 items)
  2. ScoreToInvalidateQuestionInQueue changed from 3 to 2. This is the score for the question in the queue at which it will be invalidated. (~100 items)
  3. Now invalidating questions that have an accepted answer. (~3000 items)
  4. Now invalidating questions that have at least two non-deleted answers. (~300 items)
  5. Now invalidating questions that have a total answer score > 2. (~100 items)
  6. There had been some questions that were added to both Triage and First questions on creation. These are being added automatically to Triage due to low quality heuristics. These are now being kept in Triage and invalidated from First questions (they can still be redirected from Triage to First questions based on review responses there). (~150 items).

These changes are now live, and have caused an invalidation of around 4000 items from the queue (which at the time of writing is now around 11,500 items).

In addition to the above changes, when there are a lot of items in the queue, the max number of reviews per day per user has been doubled for First questions, from 40 to 80 60 (we originally moved it to 80, which led to the queue size decreasing at a rate that was a bit high, so moving it down to 60).

Thanks Elikill58 for your suggestions above, and for all other feedback that we have received related to this. We will continue to explore more ways in the future to encourage reviews in First questions and to help new users.

13
  • 1
    Thanks you for the complete informations and explaination. I know that the limit change from 20 to 40 at 150. So 80 is for ever or only when more than 1k ?
    – Elikill58
    Commented Oct 7, 2021 at 7:36
  • 4
    80 is when more than 1K
    – Yaakov Ellis StaffMod
    Commented Oct 7, 2021 at 11:10
  • 1
    Thanks for the efforts you have made here - the size of the FQ review queue has been noticeably impacted already! One minor tweak I would suggest is to change the threshold for the switch to 80 reviews per person per day: 1,000 seems too low and something like 5,000 (or even 10,000?) may be more suitable. Commented Oct 7, 2021 at 13:46
  • 2
    Also, will a self-posted/accepted answer invalidate the question? Commented Oct 7, 2021 at 13:48
  • 6
    Self-posted/accepted answers will not invalidate First questions Review Tasks.
    – Yaakov Ellis StaffMod
    Commented Oct 7, 2021 at 14:01
  • 1
    @AdrianMole I want to wait and see how the 80/day limit affects things for a few days, then we can reconsider the point where that goes up. The danger is that the queue is being completely handled by a very small crowd of reviewers (taking advantage of 80 per day), and other folks arent getting a chance to review. If we manage to bring the number of reviews way down through this, we'll see about modifying it further.
    – Yaakov Ellis StaffMod
    Commented Oct 7, 2021 at 14:48
  • 1
    There is also the issue (for me, at least) that, until I have 'fulfilled' my daily quota in FQ, the drop-down review menu from the top-bar takes a very long time (waiting in the three square dots timer) to actually present the menu. I assume this is because I have a lot of skipped posts in the queue, that have to filtered out. (Typically, 15-20 secs; one day, it went to 30 secs then just gave up.) Commented Oct 7, 2021 at 14:53
  • 4
    @AdrianMole please post about that separately
    – Yaakov Ellis StaffMod
    Commented Oct 7, 2021 at 15:03
  • 1
    Is there any kind of metric that is being followed to measure the success of this action?
    – Braiam
    Commented Oct 14, 2021 at 15:08
  • 3
    Queue now down to fewer than 1,000 posts ... but we're still getting the 60/day quota. Commented Oct 17, 2021 at 15:45
  • @AdrianMole It was a misunderstanding. The 60/day is triggered by there being more than 150 items in the queue, not 1000. This number is adjustable but it's for all queues, not for First questions only.
    – Catija Staff
    Commented Oct 20, 2021 at 15:59
  • @Catija Thanks for clarifying. I sort of guessed that (sometime after posting my comment), actually, because I noticed my quota went directly between 60 and 20. Looks like we're back to the 20/40 quotas now, though. Commented Oct 20, 2021 at 16:01
  • 1
    Sure! :) @AdrianMole If it's useful, the 1000 is the network default for adding a multiplier, it's just that on SO we've lowered that to 150. :)
    – Catija Staff
    Commented Oct 20, 2021 at 16:03
51

Too many new users requiring guidance

I'd take a step back and consider what the purpose of the queue is. It's not to create busy work, or to allow people to rack up completed reviews to get badges. It's to help new users learn how to use the site. The problem is, there are thousands of questions posted a day, and many of those are by new users. At least in my experience, a big fraction of those questions have significant problems, such as one or more of

  • no code
  • code only inside images
  • no description of the problem to be solved
  • no attempt to solve the problem,
  • no attempt to search for and apply solutions to similar problems
  • no attempt to read the documentation when implementing something
  • a misunderstanding that an elementary tutorial would cover
  • code not formatted as code, which the poster has made no attempt to fix, despite the live preview
  • not in English

Given our sheer volume of questions, there just aren't enough man-hours by our experienced regular volunteer reviewers to help handhold whenever someone new comes along (which is every few minutes) - even if the action that is taken is only voting or commenting.

If the site frequently fails to set expectations and clarify standards before a question is posted, that's something that would be best fixed by changing the UI that new users see when posting. This is not something that should be "fixed" by piling mountains of work onto community volunteers after such questions get posted, in an attempt to guide the new users back on track.

As one person put it:

New users need better awareness of what is expected of them when they ask their first question.

Currently, we rely on a system of Help Center articles that new users never see, a patchwork quilt of Meta articles, and a Tour Page that focuses on site mechanics. We carefully bury any information that might help new users shed their bad forum habits, allowing them instead to crash headlong into a "hostile" user community that expects new users to already know our rules, and we call that being inclusive.

The best off-topic question is the one that is never asked. Allowing sites to run off the rails by providing the lowest possible friction to ask off-topic questions and then saddling the community with cleaning up those questions... well, that's not what most of us signed up for, nor is it the best use of our time. It is probably the least-friendly way to welcome new users.

Most of the professionals who are here come here to contribute and to help others, not to spend all of their time sweeping the floor.

Making technical tweaks to how questions stay in the queue may help, as suggested in the question, but I think a better approach would be to fix the problem at its source. The point of reviews is not to complete tasks, but to improve the site. Better to reduce the need for such tasks in the first place.

Not all the questions are bad, of course - a number are good or OK. But the same sort of reasoning applies - can the tens of reviewers keep up with the thousands of new questions, even for the ones that are good? Even if they can - which they may not - should they (and the community at large) be expected to?

4
  • Sure, we can help new users to don't post bad question ! It really should help, because such as you explained multiple question has problems. But, there is something more: even if they know how to ask, they will finish in the queue. And so, it will not fully solve the issue here (we could not well check questions quality)
    – Elikill58
    Commented Sep 14, 2021 at 20:44
  • 12
    "Too many new users requiring guidance" from Stack Overflow--not reviewers-on posting.
    – philipxy
    Commented Sep 15, 2021 at 5:13
  • 10
    A site like this that sells advertising doesn't really care much about this sort of issue. Click = revenue. New users with crap questions welcome, also free unpaid work to answer and fix crap question is also welcome. They might as well advertise directly yo all schools "Come ask your crap question here!, we have people to writes your codez!"
    – TheGeneral
    Commented Sep 15, 2021 at 8:10
  • This answer doesn't explain something: the number of first post have been constant since several years back and so has been the quality of the questions. How it's that there's a increase of users asking for guidance?
    – Braiam
    Commented Sep 15, 2021 at 21:05
22

I don't like this queue. For me, there's not much difference between this and close vote review queue. However, I understand the desire to have a queue to help new posters and if I only consider what it has been designed to do I believe that...

We should leave it as it is. Don't change anything. It works correctly.

The growing number of review items isn't the fault of the review queue. The problem is that there are too many questions posted. Help vampires are searching for free assistance with their code while completely ignoring the site's quality guidance. People are creating throwaway accounts to ask a single question, get an answer, and never suffer a question ban. Some are even reposting the same question multiple times in hopes of getting that free solution. If anything needs to be changed, it is the onboarding process and limiting the number of new questions.

6
  • 2
    It's not the fault of the queue, sure, but if people are creating throwaway accounts and flooding the site then we need mechanism in our queue to combat this, no? That's what the question is asking. The queue works correctly, but it needs to be effective against low quality content. That's what the queue is for.
    – 10 Rep
    Commented Sep 14, 2021 at 23:07
  • SE is more likely to fix the review queue than to fix the onboarding process as the former is on their roadmap. So I think your last sentence about the onboarding needing to change will.... never happen.
    – 10 Rep
    Commented Sep 14, 2021 at 23:08
  • 8
    @10Rep Technically the latter is on our roadmap, too ;)
    – Catija Staff
    Commented Sep 15, 2021 at 0:11
  • 1
    "The problem is that there are too many questions posted" [citation needed] the number of posts in the queue has been constant over the years.
    – Braiam
    Commented Sep 15, 2021 at 1:56
  • 4
    @Braiam Too many crap questions posted versus good ones. In other words, signal-to-noise ratio is continuing to drop.
    – Ian Kemp
    Commented Sep 15, 2021 at 9:18
  • 1
    @IanKemp still, old news. Dharman is claiming that something outside of SE changed, there should be evidence of it, because we've seeing crap since before 2013.
    – Braiam
    Commented Sep 15, 2021 at 11:30
14

How about reducing the friction on the queue? Since SE started modifying the queue at the start of the year, the average time it takes to complete a single review task has been growing, despite the fact that it's the same number of posts into the queue. From 3-6 hours last year, to +30 hours this year1. That's 10 times more. So, it's not unexpected that the first post queues are growing wildly. The only queue that exhibits a marked improvement in the time to completion times is the close queue, which has other explanations (from reducing the number of close votes, to the invalidation).

The issue with the first post queues is that a single action completes the review and yet they are taking way too much time. The data is still not in for the separated queues, but I expect that first Q's continues to grow until reviewing improves.


1(This includes the time the post waits in the queue before someone sees it)

9
  • Oh that's a good point. We can try to reduce the time to solve review.
    – Elikill58
    Commented Sep 14, 2021 at 15:43
  • 3
    @Elikill58 is not the time to handle a review, is the number of actions required to do so. There's also an issue where some actions that should complete the review, don't do so
    – Braiam
    Commented Sep 14, 2021 at 18:08
  • 1
    What do you mean by "a single action completes the review and yet they are taking way too much time"?
    – philipxy
    Commented Sep 14, 2021 at 19:02
  • 5
    I generally find that the time it takes to assess a post far outweighs whatever overhead has been introduced by the new processes. I'm skeptical that needing to click "Submit" after voting to close a question is the reason why we've gone from clearing all first questions in a day to accumulating over 10,000 in three weeks. Commented Sep 14, 2021 at 19:09
  • 1
    @JeremyCaney the change wasn't done in 3 weeks. It was done at the start of the 2021, which is kinda the same time where the graph starts to grow.
    – Braiam
    Commented Sep 14, 2021 at 19:14
  • 1
    @Braiam: Prior to moving to First Questions, the First Posts queue used to get cleared regularly—or, at least, end up well below 100. The time taken to get there might have slowed down previously, but I never saw it go below replacement rate. The compounding growth we're seeing now is certainly new. Commented Sep 14, 2021 at 19:25
  • @JeremyCaney Well, there's SEDE, try to prove that observation.
    – Braiam
    Commented Sep 14, 2021 at 21:22
  • 7
    @10Rep The post spends 30 hours waiting for review. That's what the query tells.
    – Braiam
    Commented Sep 15, 2021 at 2:25
  • We just changed the sort order for selection of posts in the First questions queue. It is now showing newer posts first (instead of older). This should have an immediate affect in terms of lowering the amount of time between when a review task is created and when it is seen. This should also increase the rate of interactions between reviewer and author.
    – Yaakov Ellis StaffMod
    Commented Feb 23, 2022 at 8:56
9

But how can we solve the problem?

From my own experience and from observing the number of curation tasks performed over time, I would say that more curation activity as a remedy is unlikely to happen. The supply of curation time seems to be quite limited, it's not a very thankful task. And in fact I think that curation should be only one of many things that keep the whole machinery running. Mistakes happen, and people may get things wrong, and that's when tailored feedback can be very helpful, but if the only quality control is a lot of people looking at new content creators, then the whole thing won't scale very well.

I'd like to get to a situation where curators have a lot less to do than now. That would mean people would have to invest a lot more of their own time and energy before publishing content, and as a result, perhaps less content would be published. Certainly, friction should be minimized as much as possible (e.g., having to hit the submit button every time), but that would probably not increase efficiency so much that it would be enough to handle the amount of questions that would need to be reviewed.

I just looked at a few questions in that queue, and the rate of questions showing any research at all is definitely 1 in 10 or less! That's not good. Why are we allowing people to ask questions in this state? I could, in principle, downvote all of these questions for that reason alone, but I would run out of downvotes pretty quickly.

To answer the question. Primarily by trying to minimize the number of bad questions that get posted, whatever way that can be accomplished, rather than by trying to further increase curation activity.

4
  • 1
    I agree. Another queston talk too about how to make people less ask bad question. And that's a good point and yes, we should help people to ask good question. Just for example by checking if there is a "?" in the question for example
    – Elikill58
    Commented Sep 15, 2021 at 21:40
  • 1
    This check will only lead to another misfeature like the title quality "control"... I already see several possibilities of how this ends up in horrible amount of false positives and negatives. Far from all well structured posts have "?" in them. Far from all badly written ones lack one either. In fact, it can be argued that "?" are more likely to be found in poorly asked questions because those askers just use whatever language construct comes to their mind...
    – 0Valt
    Commented Sep 16, 2021 at 0:49
  • 2
    There could be soft close votes (only one required to temporary close a question) to be used instead of downvotes that would close a question for a set amount of time (say, 10 minutes or 4 hours). The OP would know exactly when it would be reopened (it would not be seen as permanent and very negative) and can use the time to improve the question and/or take a quiz based on the help center and/or play the Stack Overflow Onboard Game™. In the soft close period the question would be protected from voting (up or down), but not from comments. Commented Sep 16, 2021 at 1:05
  • 2
    @PeterMortensen But that's all after the fact, i.e. after a typically low quality question has been asked. I would prefer that new users play the onboarding game before asking their first question. And maybe questions should be more structured. Instead of title and body there could be more fields and one of them could be research and filling something in there could be mandatory. Commented Sep 16, 2021 at 5:06
4

An official answer regarding the aging out of review items from the First questions review queue was given by staff in Do posts in the first answers and first questions queues age out?

The script that will remove older items from the queue should run sometime this week. From then on the size of the queue should remain approximately stable over time.

5
  • 20
    Nothing like a simple automated process to hide a problem instead of solving it properly!
    – Ian Kemp
    Commented Sep 15, 2021 at 9:19
  • 6
    Ah, the "sweep it under the rug" strategy.
    – VLAZ
    Commented Sep 15, 2021 at 20:26
  • 4
    aka "bury owns head in the sand" (if I can't see it, it doesn't exist) Commented Sep 15, 2021 at 21:15
  • 4
    @IanKemp That's not exactly new - for long periods of time (including recently), half of all posts in the close review queue age out without actually being judged... and those were posts we had some signal that said "this should be closed" these first posts are just first posts - they could be good or bad. There's too many things to review and too few people interested in doing the reviewing - and that's something we need to fix - we need to better set expectations of newer users about what a good question is so that fewer bad ones are posted.
    – Catija Staff
    Commented Sep 17, 2021 at 4:30
  • A question that 'ages out' of the First Questions queue should probably just age out of the site. Ofcourse that would risk losing good content.
    – ouflak
    Commented Jan 19, 2022 at 13:53
4

One way to make a dent in the queue size would be to have posts removed from the queue if they enter any other queue. The most likely candidate would be the Close Votes queue: I see a surprisingly large number of posts in the course of my daily 40 reviews that already have one or two close votes on them.

In Yaakov's answer to a question referenced elsewhere in this thread, we see that fully closing a question removes it from First Posts (well, why wouldn't it); also (now that the bug is fixed), casting a close vote from within the review also completes the process. However, why not remove it from the queue whenever a close vote is cast on it? Or, put slightly differently: Why should these posts be allowed to swell the size of two queues (First Posts and Close Votes – both large enough on their own)?

Entry into the Triage queue should also remove it from First Posts – but I'm less clear on what user action(s) trigger that.

3
  • The only user action that I'm aware of that puts a post in the Triage queue is the "Very Low Quality" flag. But that also puts the post in a moderator queue, so the flag will get declined. I'm not sure what happens to the post in the Triage queue at that point. The rest of the posts in the Triage queue got there by some system heuristics.
    – Scratte
    Commented Sep 23, 2021 at 22:42
  • @Scratte But "system" actions that put a post into Triage could also be used to remove it from FP. The question indicates that a post can be in both queues. Basically, what I'm saying is that we don't really need any post to be in two queues at the same time. Commented Sep 23, 2021 at 22:45
  • That is true. I don't know if the limit is two, but I wouldn't be surprised if a post could now be in Triage, First Questions and the Close Vote queue at the same time. But I suspect that it actually cannot. I suspect it's not put into the Close Vote queue until the post has exited Triage and/or First Questions. (I reckon they did not change the logic where previously a post could be in both Triage and First Posts, which is why one could get a disputed flag raised in First Posts, by a review result in Triage.)
    – Scratte
    Commented Sep 23, 2021 at 22:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .