70

I've been watching the Low Quality Posts review queue to see whether feeding "not an answer" flags into there would cause any problems. For the most part, that review queue seems to be working quite well for identifying and deleting bad posts with no moderator intervention. I've seen few instances of people clicking "Looks Good" on non-answers and spam, and bans have taken care of the rest.

However, a few flags have come in lately for the other side of this: delete votes being cast on what seem to be good posts. Most of these are short, but they all appear to be legitimate answers to the question asked. The following are just some examples I pulled up now:

I was particularly baffled by the unanimous deletion votes on this answer: https://stackoverflow.com/review/low-quality-posts/4898020 , which even in its first revision was a legitimate answer.

The thing is, this isn't your typical review abuse where people are clicking a button over and over again. These are all otherwise good reviewers making these calls. I honestly don't know how to handle these delete reviews on what seem to be viable answers, or even if I need to do anything.

I'm a little troubled by this predisposition to deleting every short answer that appears in the Low Quality Posts review queue (via a mistaken flag or an overactive post quality heuristic). Should I be? Is this simply people freely exercising a vote on quality, like with a downvote?

8
  • 6
    I've detected some incorrect reviews through the canned comments that the review system allows reviewers to post on reviewed answers. Like for instance an answer that contains no link that got the message about how a link only answer is bad, etc. I agree with you about the cases you show. So it's not just you who is troubled.
    – Louis
    Commented May 27, 2014 at 17:42
  • 3
    But your last example doesn't really answer the question…
    – nietonfir
    Commented May 27, 2014 at 19:49
  • 1
    Agreed, the version of that answer that was reviewed should have been a "works in my machine" comment. The rest look like standard robo-reviewing. Commented May 27, 2014 at 21:28
  • 1
    I am worried about delete votes - as well as downvotes of course, but that is not for today -, without explanation. It is not a problem in case of questions since they get close reasons first, but it is a real problem for answers IMHO. Commented May 28, 2014 at 12:00
  • 2
    Brad, three of those posts do not even bother to explain the code... Why have you not asked for that where the request is missing? That is the first reaction when I see those... Commented May 28, 2014 at 12:03
  • 3
    When I started first reviewing on the site a few years ago, the no explanation one-line answers were perhaps the first thing that bothered me about the review process. At that time I saw that high-rep users routinely got away with it while often low-rep users were chastised for it. Nowadays I think in practice the one-line explanationless answers are generally let through with less scrutiny across all users.
    – demongolem
    Commented May 28, 2014 at 19:22
  • Start with questions.
    – user2140173
    Commented May 29, 2014 at 8:25
  • 3
    A user can post a crappy one-line answer and edit it within the 5-minute grace period, and it will look like the first revision was fine, when actually it was crappy. (Another side-effect of the FGITW problem.) I got a number of my flags marked as "unhelpful" before I worked this out. Commented Jun 24, 2014 at 15:03

3 Answers 3

79

I see a lot of answers in the VLQQ that, at most, deserve a downvote. However, the UI does not offer downvoting. I claim that, human nature being what it is, that this leads some people to make the wrong choice some of the time, and that providing a downvote option would make that mistake significantly less likely.

11
  • 38
    I can see that. If the non-deletion option is "Looks Good", that might be read as implying that you fully approve of the post, particularly with a description of "Looks Good: if nothing is wrong with this answer." People who still have some reservations about the quality of the post might figure that the correct course of action is then to delete. Didn't realize this review queue lacked voting options, unlike the other queues.
    – Brad Larson Mod
    Commented May 27, 2014 at 18:18
  • 63
    @BradLarson The other option is to just change the text of the post from "looks good" to something that conveys, "No so completely awful that it needs to be deleted", which is what it actually means. For example, something like "Good enough" or just "Don't delete". Too many people think that "looks good" means that they need to think it's a good post, but that's not what it means.
    – Servy
    Commented May 27, 2014 at 18:23
  • 51
    @Servy: That's exactly what needs to change, indeed. I still feel terrible clicking 'Looks Good' on bad answers, that are still answers none-the-less. Commented May 27, 2014 at 18:27
  • 2
    wonder how many LQ reviewers would resist the temptation to fall back to "painless" Recommend Deletion after they discover that option to vote down costs them -1 rep
    – gnat
    Commented May 28, 2014 at 7:14
  • 1
    Yes. This is what I was trying to get at in a question I posted here - meta.stackoverflow.com/questions/253831/… . They are questions I feel I should take action on, but the fact I have to do two clicks, and leave the VLQQ means I probably either don't downvote, or maybe even vote to delete questions that shouldn't be. Commented May 28, 2014 at 12:41
  • 2
    @Servy captures it I think. I typically wind up clicking "Skip" on those kinds of questions, but that's just cowardice and shunning a decision. Offering only "it's totally right" and "it should be deleted" is probably not ideal for SO. Even "it's fine" (or "Good Enough") would be fine. Lots of answers offer something to the conversation without being "Looks Good."
    – Rob Napier
    Commented May 28, 2014 at 19:17
  • 1
    I'll add my voice to the looks-good-considered-harmful choir. It subtly frames that question as Good vs. Bad, which of course is not the purpose of the VLQQ process. Like @RobNapier, I often chicken out and click "Skip" instead. I propose "Acceptable" as an alternative. Commented May 28, 2014 at 21:35
  • 7
    I vote for a button reading "Not Completely Awful". Commented May 28, 2014 at 23:44
  • 5
    @RobNapier: I don't know if "It's Fine" is much different than "Looks Good". I'd prefer "Don't Delete" Commented May 29, 2014 at 10:54
  • 5
    A downvote option misses the entire point of the queue: correcting or removing trash from the site. I don't want reviewers mass-downvoting Very Low Quality posts, I want them deleting them! The system adds an automatic downvote on such posts for every helpful VLQ flag, and that's on top of any votes that arose from folks reading the Q&A naturally. Tempering the wording is reasonable however.
    – Shog9
    Commented Jun 3, 2014 at 23:41
  • @Shog9: Understood. But there are reasons that an item is low quality and a downvote indicates that. Posts that are low quality don't necessarily meet the criteria for deletion. I don't have any ready examples. Commented Jun 12, 2014 at 15:56
-1

Regarding the first five reviews you pointed out: these are all in a class that should be titled "snippet-only answers" (the third has since been edited, but rev. 2 was visible at the time, and was just the snippet). In most cases, answers like these are of little use to anyone but help-vampire-y programmers who want us to give them teh codez; there is no explanation and very little actual help is provided. In one case, the question's asker even explicitly asked for more information, because the mere snippet wasn't clear. Even the last one fits into this category on the first revision (which comes down to "Run make in directory X instead of Y"), which is the one that was deleted.

Are these answers worthy of deletion? Not really. They do (usually) technically answer the question, and they can even sometimes be enough help to solve the problem. And certainly, some perfectly fine questions are worded such that merely giving a line or two of code is an alright answer. They should instead be downvoted. But, when presented with no way to downvote, and with the correct action labelled "Looks Good", there're always going to be people recommending deletion on downvote-worthy posts.

All in all, though, the system appears to be working. Enough "Looks Good" votes come in that the posts generally get to live. Unless you see a user delete-voting such posts over and over again, or this becomes a regular pattern, there's probably not much need for direct moderator action here.

3
  • 15
    I disagree with saying that snippet-only answers are generally "of little use to anyone" but help vampires. If it is a good snippet, which shows for example how to correctly use some API, then it often contains just the information which may be missing from reference docs. Though I agree that just code snippet is still a bad answer, and I'd be reluctant to upvote it even when such an answer helped me into the right direction. Actually only asker is probably able to just copy-paste a snippet without understanding, others need to study it first.
    – hyde
    Commented May 27, 2014 at 19:54
  • 7
    Disagree. Sometimes short answers are the absolute best. I don't always need someone to quote MSDN, the Linux archives and post a 100 lines of code just to show me that they can research an answer. Usually I just need the answer. If I want/need to know the nitty gritty, I'm happy to dive in. But the quality of an answer does not, nor should it ever, depend on the word count.
    – ouflak
    Commented May 28, 2014 at 12:07
  • 7
    Agreed that short answers are great. If you can answer a question in a few lines, and it's clear and well-explained, that's perfect. The problem isn't that these are short; it's that they're just a snippet of code with no reasoning, no explanation, nothing about why that snippet works; not even a few lines of a real answer. Commented May 28, 2014 at 18:13
-18

As long as deleted post is recent (so that author can easily find it in their profile page), I wouldn't worry about this.

Thing is, the system is designed to support easy correction when things deviate from happy path ("happy path" here being unambiguous decision of reviewers and voters that post isn't wort keeping, followed by total negligence of its author).

Namely, author of deleted post can easily find it and undelete if they wish - this is exactly what happened in one of the examples in your question.

For the sake of completeness, if it is deemed desirable to minimize deletion-correction cases for certain kind of posts, this could be achieved by tuning review audits selection algorithm to plant enough posts of this kind into the queue as known good audits. Such a tuning would better take into account educational effect of the change, as known good audits make quite a prominent demonstration of what posts are considered good quality.


While we're at it, below are few things worth keeping in mind discussing Low Quality review queue. 1

Oblivious of meta

Review in this queue is supposed to run without any meta guidance.

This is a matter of simple math: there are about 30,000 eligible reviewers (2Kers), while top viewed flags question at MSE has only about 18K views - including views from lower rep users, as well as views from users of other SE sites and from all the internet at all.

No diamonds

Review in this queue is supposed to run without diamond moderator involvement.

This is a matter of simple timing: in straightforward cases, deletion takes only a few minutes. If it was expected that moderators control such a fast paced activity, they would have to be on a 24x7 permanent watch, which is certainly not the case.

Mob rules, balanced

At first glance, it seems that just six-seven 2K users can in theory create a death ring and remove 20 any answers a day through flagging and consequent voting in LQ queue (20 is review limit when queue has less than 1000 items).

But system is indeed designed to prevent anything like this from happening.

To start with, if post author ever notices deletion they can simply undelete it solely at their discretion. And this is not all, system appears to be designed to either dissipate into disputed flag or alert moderators any time when things deviate from happy path ("happy path" here being unambiguous decision of reviewers and voters that post isn't wort keeping, followed by total negligence of its author).

For the sake of completeness, moderators have means to block repetitive abusive flagging (flag suspensions). But if you think of it, system is designed so that most bad flags are expected to be blocked before they even bother a moderator.

When there are 30,000 mostly reasonable reviewers in the queue, chances for a substantially mis-flagged post to pass through without being stopped by regular reviewers are pretty slim.

First round: light as the breeze

Overall, it looks like no risk and no pain are involved for all the parties in the initial phase of dealing with posts in LQ queue. Worst case for flagger is they get painless "disputed" flag. Worst case for author who noticed deletion of their answer is they click undelete.

Bad answers are easy and quick to delete, and when deletion mistakes happen, it's easy to revert for post author.

Related reading:

11
  • 11
    What exactly is the conclusion/general thought behind this post? What I take away from this is "it doesn't matter if we delete too much, people can just undelete it". That is not an acceptable rationale at all. Perhaps you can shed some light as to why you believe some of these answers should have been deleted, considering you're amongst a few yourself. For example this answer is perfectly fine: it describes the problem and provides a solution. What was the reasoning for deletion? Commented May 27, 2014 at 18:11
  • 12
    Full disclosure: you voted to delete on at least one of the disputed reviews. :) FWIW, I do believe there's value in getting this right the first time, rather than relying on "correctional principles."
    – Robert Harvey Mod
    Commented May 27, 2014 at 18:12
  • 6
    I've got to wonder how often a deletion which was in fact against the guidelines does not result in the author undeleting their answer but in the author taking away from the deletion that there was really something in their post that went against established SO guidelines. If they walk away from the interaction with an incorrect notion, this is not something that is going to be visible in the system.
    – Louis
    Commented May 27, 2014 at 18:12
  • @RobertHarvey yes, and I did that with ease exactly for the reasons laid out here: "the system is designed to support easy correction when things deviate from happy path"
    – gnat
    Commented May 27, 2014 at 18:13
  • @Louis worth noting it's fairly difficult to reason about guidelines and following these when there are 30,000 reviewers in the queue, see the section "Oblivious of meta"
    – gnat
    Commented May 27, 2014 at 18:15
  • 2
    @JeroenVannevel the original revision of the answer you ask about reads differently from what you see ("Set <%@ defaultCodec="HTML" %> in the top of your gsp.") As for conclusion, it's fairly simple - there's no way to control 30,000 reviewers, the system appears to be designed to rely on common sense of most of them and easy correction. I only describe what I see
    – gnat
    Commented May 27, 2014 at 18:17
  • 7
    So how do you explain why the posts you recommended deletion on were actual answers to the question?
    – Robert Harvey Mod
    Commented May 27, 2014 at 18:24
  • 1
    @RobertHarvey why does this need to be explained? LQ review deletion popup provides at least two options to vote for deletion on "actual answers" - link-only and no comment. I used it as designed
    – gnat
    Commented May 27, 2014 at 18:31
  • I see that answer as "do this". It might not have been optimal in the first revision but it was still an answer of acceptable quality. What deletion reason do you connect to it? Commented May 27, 2014 at 18:39
  • @JeroenVannevel if memory serves, I picked "no comment" option in the LQ deletion popup. As for what quality to consider acceptable, I am afraid that there's no way to impose uniform opinion on this on all thirty thousands reviewers in the queue (in that sense, it doesn't really matter that I disagree with your assessment regarding this post)
    – gnat
    Commented May 27, 2014 at 18:41
  • 3
    +1 for no. Being concerned is always unnecessary. If there is an actual problem (supported by data) and an actual solution (that works better than placebo), feature request. Otherwise there's probably nothing to do, the system works fine.
    – bjb568
    Commented May 27, 2014 at 20:45

Not the answer you're looking for? Browse other questions tagged .