tl;dr; When flagging questions as VLQ, we're inadvertently sending too much noise to the moderators to handle. Over the past year, the reviewers here on Stack Overflow have shown themselves able to accurately sift through the majority of these flags, but due to various assumptions, the system still isn't giving them the opportunity in a significant number of cases - it's time to hone the criteria for what gets added to Triage and what gets removed.
Background
Triage review was built to help new questions, not hitting a predefined "quality score," get human review. Based on the result of that review, a question will go to one of three places:
- The homepage, where they can be answered
- The close or moderator flag queue where they can be reviewed and eventually deleted
- A new "Help and Improvement" queue where they can be edited
The lifecycle for a question looks like this:
Notice the freehand circles around the VLQ flag... more on this later.
Triage and the other review queues (close votes) should process the majority of these questions, and it does, but I've noticed things falling through the cracks and I'd like to see if we can fix some of those.
Current triage settings work… but there are problems
The queue handles a lot of questions. Since it was launched, nearly 1.1 million posts have been triaged - an average of 60k+ a month. <!---15539---> Depending on the month, anywhere from 17% to 27% of all questions asked go through Triage.
Triage reviews the newest questions and they should be handled in a reasonable amount of time (no huge backlog like Close Review), in order for this to work, we have safeguards in place to control what hits the queue.
Some of the limits include:
- The question must have been asked in the past 7 days
- The maximum queue size is
100200 (as of August 4, 2016) items at any moment
While we've processed a ton of questions with the current settings, the limits are...well, limiting. This results in some questions being pushed to triage, then immediately kicked out for a variety of reasons.
That might not sound like a huge issue, but when a question bypasses triage it goes directly into the moderator flag queue, meaning our human exception handlers need to process it. Moderators are supposed to deal with exceptional things and most of these questions don't qualify for that.
What's bypassing the queue?
Remember those freehand red circles in the question lifecycle, the biggest thing causing questions to bypass Triage is the VLQ flag. If a user flags a question as VLQ and Triage is full, the question goes directly to the moderator flag queue for handling.
In the past 90 days, there have been 7,721 VLQ flags on questions, of these 930 were handled by the moderators. That's a lot of flags that should be processed elsewhere. The mod flag queue should have items that the community cannot handle, so I wanted to take a closer look at why moderators were getting so many VLQ question flags to process.
Breaking down these 930 questions: <!---15588--->
- 53% were sent to the moderators because triage was full
- 46% were invalidated from triage and sent to the mods
The queue being at max capacity is a concern, but in analyzing the invalidated questions, I found some holes we could plug.
What holes? How can we fix them?
Problem 1: Currently, when a question is older than 7 days and it receives a VLQ flag it bypasses Triage and goes directly to the moderators. Over the past 90 days, this accounted for 193 questions. Since triage is for new questions, older questions should be handled by close votes or flags which will send it to the close vote queue for processing.
Proposed Solution: We already don't show the VLQ flag when a question is in review, so let's expand the rules. If a question is older than X days it won't be eligible for Triage, so the VLQ flag will be disabled on them. The "X" will be a site setting that can be adjusted, right now we're leaning toward 3 days.
Problem 2: VLQ flags on questions can only be added when the score is <= 0, but they stay active on questions even after it gets a positive score. In the past 90 days, there were 39 questions with a VLQ flag and a positive score processed by the mods. Only one of these flags was marked helpful, so it seems that if a question with an active flag gets a positive score, the flag should be disputed.
Proposed Solution: If the score on a question becomes > 0 while a VLQ flag is active, dispute the flag.
Problem 3: The definition for a VLQ flag on a question is:
This question has severe formatting or content problems. This question is unlikely to be salvageable through editing, and might need to be removed.
Assuming a question doesn't look like a cat typed it and it's somewhat understandable, it seems reasonable that if it receives multiple upvoted answers, it may not be VLQ. It might be off-topic, but not VLQ because multiple users were able to understand it enough to answer it. Again, looking at flags for the past 90 days, we had quite a few questions flagged VLQ that received at least one positively scored answer:
# of Answers | Total Posts | Score=1 | Score=2 | Score=3 | Score>=4 |
---|---|---|---|---|---|
1 | 45 | 28 | 7 | 6 | 4 |
2 | 35 | 14 | 13 | 3 | 5 |
3 | 11 | 7 | 3 | 1 | 0 |
4+ | 26 | 12 | 6 | 4 | 4 |
Proposed Solution: If a question has more than 1 answer with a score > 0, then the VLQ flag is not available. In addition to this, if a question with an active VLQ flag receives more than 1 answer with a score > 0, the flag will be automatically disputed.
Next Steps
When it comes to the VLQ flag on questions and how Triage handles them, most of these problems are the low-hanging fruit. Before we move forward in making any adjustments to remove noise in the moderator flag queue...
- Do you see any red flags that we are missing?
- Can you think of cases where a close vote or flag wouldn't help get rid of the questions or where a VLQ flag is necessary?