56

tl;dr; When flagging questions as VLQ, we're inadvertently sending too much noise to the moderators to handle. Over the past year, the reviewers here on Stack Overflow have shown themselves able to accurately sift through the majority of these flags, but due to various assumptions, the system still isn't giving them the opportunity in a significant number of cases - it's time to hone the criteria for what gets added to Triage and what gets removed.

Background

Triage review was built to help new questions, not hitting a predefined "quality score," get human review. Based on the result of that review, a question will go to one of three places:

  1. The homepage, where they can be answered
  2. The close or moderator flag queue where they can be reviewed and eventually deleted
  3. A new "Help and Improvement" queue where they can be edited

The lifecycle for a question looks like this:

enter image description here

Notice the freehand circles around the VLQ flag... more on this later.

Triage and the other review queues (close votes) should process the majority of these questions, and it does, but I've noticed things falling through the cracks and I'd like to see if we can fix some of those.

Current triage settings work… but there are problems

The queue handles a lot of questions. Since it was launched, nearly 1.1 million posts have been triaged - an average of 60k+ a month. <!---15539---> Depending on the month, anywhere from 17% to 27% of all questions asked go through Triage.

Triage reviews the newest questions and they should be handled in a reasonable amount of time (no huge backlog like Close Review), in order for this to work, we have safeguards in place to control what hits the queue.

Some of the limits include:

  • The question must have been asked in the past 7 days
  • The maximum queue size is 100 200 (as of August 4, 2016) items at any moment

While we've processed a ton of questions with the current settings, the limits are...well, limiting. This results in some questions being pushed to triage, then immediately kicked out for a variety of reasons.

That might not sound like a huge issue, but when a question bypasses triage it goes directly into the moderator flag queue, meaning our human exception handlers need to process it. Moderators are supposed to deal with exceptional things and most of these questions don't qualify for that.

What's bypassing the queue?

Remember those freehand red circles in the question lifecycle, the biggest thing causing questions to bypass Triage is the VLQ flag. If a user flags a question as VLQ and Triage is full, the question goes directly to the moderator flag queue for handling.

In the past 90 days, there have been 7,721 VLQ flags on questions, of these 930 were handled by the moderators. That's a lot of flags that should be processed elsewhere. The mod flag queue should have items that the community cannot handle, so I wanted to take a closer look at why moderators were getting so many VLQ question flags to process.

Breaking down these 930 questions: <!---15588--->

  • 53% were sent to the moderators because triage was full
  • 46% were invalidated from triage and sent to the mods

The queue being at max capacity is a concern, but in analyzing the invalidated questions, I found some holes we could plug.

What holes? How can we fix them?

Problem 1: Currently, when a question is older than 7 days and it receives a VLQ flag it bypasses Triage and goes directly to the moderators. Over the past 90 days, this accounted for 193 questions. Since triage is for new questions, older questions should be handled by close votes or flags which will send it to the close vote queue for processing.

Proposed Solution: We already don't show the VLQ flag when a question is in review, so let's expand the rules. If a question is older than X days it won't be eligible for Triage, so the VLQ flag will be disabled on them. The "X" will be a site setting that can be adjusted, right now we're leaning toward 3 days.


Problem 2: VLQ flags on questions can only be added when the score is <= 0, but they stay active on questions even after it gets a positive score. In the past 90 days, there were 39 questions with a VLQ flag and a positive score processed by the mods. Only one of these flags was marked helpful, so it seems that if a question with an active flag gets a positive score, the flag should be disputed.

Proposed Solution: If the score on a question becomes > 0 while a VLQ flag is active, dispute the flag.


Problem 3: The definition for a VLQ flag on a question is:

This question has severe formatting or content problems. This question is unlikely to be salvageable through editing, and might need to be removed.

Assuming a question doesn't look like a cat typed it and it's somewhat understandable, it seems reasonable that if it receives multiple upvoted answers, it may not be VLQ. It might be off-topic, but not VLQ because multiple users were able to understand it enough to answer it. Again, looking at flags for the past 90 days, we had quite a few questions flagged VLQ that received at least one positively scored answer:

# of Answers Total Posts Score=1 Score=2 Score=3 Score>=4
1 45 28 7 6 4
2 35 14 13 3 5
3 11 7 3 1 0
4+ 26 12 6 4 4

Proposed Solution: If a question has more than 1 answer with a score > 0, then the VLQ flag is not available. In addition to this, if a question with an active VLQ flag receives more than 1 answer with a score > 0, the flag will be automatically disputed.

Next Steps

When it comes to the VLQ flag on questions and how Triage handles them, most of these problems are the low-hanging fruit. Before we move forward in making any adjustments to remove noise in the moderator flag queue...

  • Do you see any red flags that we are missing?
  • Can you think of cases where a close vote or flag wouldn't help get rid of the questions or where a VLQ flag is necessary?
29
  • 26
    Yet another scaling problem. SO now regularly gets more than 13,000 questions per day. This is way beyond what can be reasonably handled, by everybody. Fix the fundamental problem, make it harder to ask for help. Commented May 27, 2016 at 18:14
  • 4
    Actually @HansPassant we're looking at improving the guidance for new users when they go to ask a question. It's something we'd like to focus on this year.
    – Taryn StaffMod
    Commented May 27, 2016 at 18:19
  • Keeping in mind I can use neither queue yet so this may be a pointless question: If the "VLQ" flag automatically becomes unavailable, will it still work in the Help queue? Since that's the only "flag" option for questions that go into that queue, at least from the review itself. If it's not available because the question hit the queue at the end of the third day, say, and the VLQ flag became unavailable due to age very shortly afterwards... Do you folks have a plan for that?
    – Kendra
    Commented May 27, 2016 at 18:19
  • 2
    There'll probably be some small window of time where a VLQ flag from the Help queue would end up going straight to the mod queue, @Kendra. I wouldn't expect this to happen often though; edge-cases on edge-cases.
    – Shog9 Mod
    Commented May 27, 2016 at 18:22
  • @HansPassant - A good suggestion, but how? You have to register to ask a question, and we know that people will burn through Gmail and other disposable addresses to create new accounts when blocked. IP-level rate limiting is already being used, as is a faster-acting question ban. It's pretty hard to keep out people desperate enough to ask questions.
    – Brad Larson Mod
    Commented May 27, 2016 at 18:22
  • 1
    @NathanOliver Bountied questions with a VLQ flag should already hit the mod queue instead of Triage. That is one of the current edge-cases that go to the mods.
    – Taryn StaffMod
    Commented May 27, 2016 at 18:26
  • 1
    Well then never mind. Everything else looks good. Commented May 27, 2016 at 18:27
  • 7
    @BradLarson Don't charge rep for questions, but you can require a rep threshold to ask. Two rep or five rep, while trivial to get for many people, is a much bigger hurdle to jump than making a new gmail account, namely because a rep threshold requires you to wait for and receive a positive response from another human before you can get upvoted or have your suggested edit approved.
    – TylerH
    Commented May 27, 2016 at 19:26
  • 2
    @TylerH - There are problems with a reputation barrier that would lead to frustration and abuse: meta.stackoverflow.com/a/253179/19679
    – Brad Larson Mod
    Commented May 27, 2016 at 20:05
  • 2
    I'm not sure that's necessarily the right attitude @TylerH, we need to spend time on improve the guidance to help new users when they go to ask a question. While we haven't tested anything yet, I think if we improve that it would help immensely.
    – Taryn StaffMod
    Commented May 27, 2016 at 20:11
  • 4
    @bluefeet I think the return from such passive efforts is very limited. Unfortunately the necessary solution is far from feasible, it seems... in order to get people to ask high-quality questions, we need them to be smarter/put more effort in/read more about our site... how can we 'help' someone before they come to us for help?
    – TylerH
    Commented May 27, 2016 at 20:14
  • 1
    @bluefeet: Ah, that wasn't clear. It sounded like Triage would stay at 7 and VLQ would be 3. If they're both at 3 that's probably fine. Commented May 27, 2016 at 21:31
  • 1
    @NathanTuggy I tried to clarify it a bit, hope that makes more sense that they are tied together.
    – Taryn StaffMod
    Commented May 27, 2016 at 21:34
  • 1
    Thank you, thank you, thank you, thank you! Currently about 45% of the SO flag backlog we moderators have to handle consists of VLQ flags. This looks like it'll help reduce that pile significantly!
    – Martijn Pieters Mod
    Commented May 28, 2016 at 1:56
  • 1
    @bluefeet I just found this today. Since we can no longer flag VLQ due to the age what should be done? Commented Aug 10, 2016 at 17:15

1 Answer 1

16

"VLQ flags on questions can only be added when the score is <= 0" AND, as you say, when the question is not under review. That makes them basically pointless to cast on questions. The only result of casting a VLQ flag is a longer time to closure than would happen if you just cast a regular close vote.

Your post seems to assert that there's no useful signal from VLQ flags. But that's largely because the use of the flag is so restricted. There's no useful signal where the signal can't be raised in the first place. That's what should change.

Don't tie people's hands just because they're not using the Triage queue. Instead, while review goes on in the queue, the flag should always be available elsewhere. It should have a cumulative effect, like spam flags do. Set the threshold higher, maybe: 8 or 10. Hit that number, and the question is garbage beyond a reasonable doubt, no exception handling necessary. Just delete the damned thing.

Note that this is the original official definition of the flag: it is a "flag for removal". This was also more recently expressed, just as clearly. If that's no longer the case, it should be made more explicit by changing the wording or title, not just watering down its effect.

13
  • 1
    This would be a big change, but I thought of it when the last VLQ post was made on MSE and thought I'd run it up the flagpole.
    – jscs
    Commented May 27, 2016 at 18:31
  • 1
    It's very rare that a question picks up more than one VLQ flag, of the 7.7k+ questions in the last 90 days only 175 questions got more than 1 VLQ flag. And the most number of flags on a single post was 4. Having the flags linger not in a review queue, puts them in front of the mods when they don't need to see them - that's part of the problem I'm attempting to tackle here.
    – Taryn StaffMod
    Commented May 27, 2016 at 18:34
  • 4
    Since you're saying that, I probably completely misunderstand how this works: isn't it impossible to raise a VLQ flag if one has already been cast (or question entered Triage "naturally")? I continually see questions that are very low quality but unflaggable.
    – jscs
    Commented May 27, 2016 at 18:39
  • 3
    Also, I'm not saying the post should not appear in the queue, but that the flag should be available regardless, letting "review" happen more quickly because you don't have to be in Triage mode to pass judgement.
    – jscs
    Commented May 27, 2016 at 18:41
  • That specific question was either in Triage or the Close Vote review queue (under current review) which prevented additional VLQ flags. If it's not in review for any number of reasons, the flag can be added by other users. We prevent the additional flags because we're working under the assumption that the existing review will handle the question appropriately, adding more VLQ doesn't do much at that point.
    – Taryn StaffMod
    Commented May 27, 2016 at 18:45
  • 7
    @bluefeet how about this: if a user eligible to Triage intends to flag VLQ, you offer them a link to review of this question in Triage queue. This way they would be able to integrate in the review process
    – gnat
    Commented May 27, 2016 at 18:50
  • 1
    That's interesting @gnat and something to consider, in addition to the other issues we currently have with the flag and Triage. I'll have to think about that.
    – Taryn StaffMod
    Commented May 27, 2016 at 18:52
  • 6
    @bluefeet: "We prevent the additional flags because we're working under the assumption that the existing review will handle the question appropriately" ...isn't the premise of your question that review is not currently doing so? This appears to be a circular argument against my idea.
    – jscs
    Commented May 27, 2016 at 18:56
  • 1
    Not necessarily because review does work @JoshCaswell, but there issues with the way it works today - it kicks out old questions, we've intentionally limited queue size. I'm worried about those things because it takes times for the mods to process that stuff when they don't need to. Basically review works most of the time, I'm trying to improve the stuff that's missing review.
    – Taryn StaffMod
    Commented May 27, 2016 at 18:59
  • 7
    @bluefeet Your statement "it is very rare that a question picks up more than one VLQ flag" is because 95%+ of the time the first VLQ blocks other VLQ flags from being applied. So it isn't useful information as a comment on a answer about if "what if multiple VLQ flags simply trashed the question", because the answer presumes that the first VLQ wouldn't block more VLQs... Commented May 29, 2016 at 14:01
  • @Yakk Considering so many questions fail to go into Triage that's not actually the case. If it's a new question that gets a VLQ flag and Triage is full, it's possible for it to get more VLQ flags. I'll look closer at the numbers after the weekend is over. But before we go off in the weeds, I'm trying to alleviate some specific problems with the flag and Triage - older questions get punted out, etc. there are probably additional ways we can plugs holes but this is a first pass.
    – Taryn StaffMod
    Commented May 29, 2016 at 14:08
  • totally agree with this
    – user3956566
    Commented Jun 5, 2016 at 0:08
  • Here's another good one. It makes absolutely no sense that this is not flaggable as VLQ. Straight to deletion is exactly what needs to happen there.
    – jscs
    Commented Oct 10, 2016 at 22:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .