26

I see lots of Meta posts from people who failed bad audits wanting to give feedback. This question not about that issue.

Take this example: https://stackoverflow.com/review/triage/14764264

The question is clearly unsalvageable and right now there are three "Unsalvageable" reviews and one "Looks OK". The person who voted "Looks OK" may not understand what constitutes a good question or may be robo-reviewing.

It would be really good if the site could provide feedback to those users who seem to consistently review counter to the majority opinion. I understand there are some times where it's not quite so clear and there might be valid disagreement, but I encounter these review quite often: A horrible question that should be deleted immediately gets one or two "Looks OK" votes from low-rep users who clearly don't understand the purpose of SO.

Is there any mechanism, either in place or under consideration, to communicate to reviewers when they are consistently reviewing contrary to the majority? I would expect this would be based on accumulated statistics rather than single instances. I feel this would have a significant effect on the quality of reviews and improve the site overall.

6
  • 8
    We need a simple Accept/Reject ratio for/against the consensus - that is, a percentage of how many review decisions were agreed with by the majority. That would be more clear than - what I just did - checking out your user's activity tab and click the first ten or so reviews.
    – Jongware
    Jan 3, 2017 at 2:03
  • 4
    .. something more meaningful than "xxx has approved 5973 edit suggestions and rejected 10043 edit suggestions" - the numbers don't mean anything until you verify them against the final decision. For one, I think it would highlight robo-reviewers.
    – Jongware
    Jan 3, 2017 at 2:06
  • 2
    So what we need is a review queue to review the reviews, but then who will review the review queue that reviews the reviews? Jan 3, 2017 at 16:12
  • No, the cross-check is automatic and based on statistics. If your reviews are consistently contrary to the majority you get a notice when you review. Jan 3, 2017 at 16:15
  • 1
    related/dupe: meta.stackoverflow.com/questions/298174/… Jan 3, 2017 at 17:46
  • Automating this could lead to providing feedback to the wrong users (for example, a case where the majority were wrong.)
    – Kevin B
    Jan 3, 2017 at 22:12

1 Answer 1

3

A quick look at their recent reviews suggests that they don't quite have the grasp on what "OK" content is.

Especially this one:

https://stackoverflow.com/review/low-quality-posts/14764215

Not only is that answer horrible, the fact that they cosigned on it is terrible.

There's not much feedback that we can provide yet; in all honesty, using a custom moderator flag to alert the mod team to these kinds of reviews is about the best we can do. We can provide feedback, but only the moderator can do what's best in this scenario - give 'em a "vacation" from reviewing.

19
  • I'd really like to see the system gather these statistics and automatically notify the flagrant cases, at least with a gentle reminder. Even if it's only people who vote "Looks OK" when the consensus is "Unsalvageable", it would improve things immensely. Jan 3, 2017 at 2:45
  • 2
    BEWARE, there's a group of users that argue that 1) those answers shouldn't be deleted, 2) nor edit them to add an explanation.
    – Braiam
    Jan 3, 2017 at 2:46
  • 3
    @Braiam: They're entitled to their beliefs but in all honesty, anything that can be improved should be improved in review. Not just that, but there is a very similar-looking answer which was posted years earlier, which doesn't really convey much confidence in the OP's good-natured attempt at answering it.
    – Makoto
    Jan 3, 2017 at 3:55
  • 14
    They're entitled to their beliefs, @braiam, but we are also entitled to ban them from reviewing until their beliefs have changed. :-) It doesn't do any good to have reviewers if the reviewers are all operating under the assumption that everything is fine as-is and never needs to be changed. Jan 3, 2017 at 5:44
  • 4
    I just want to mention this one here: You're doing it wrong: A plea for sanity in the Low Quality Posts queue
    – Floern
    Jan 3, 2017 at 12:57
  • 1
    @CodyGray I'm just adverting that meta is cornering itself into a very poor position: if you can't delete, edit or say that they are ok, then what option would be left?
    – Braiam
    Jan 3, 2017 at 13:34
  • @Floern I whole-fully disagree with everything in there: answers should be either improved or, failing that, removed. We should keep cruft growing.
    – Braiam
    Jan 3, 2017 at 13:35
  • 4
    In suggesting that "Looks OK" is wrong for stackoverflow.com/review/low-quality-posts/14764215, you're going against community consensus, Makoto and @CodyGray. meta.stackoverflow.com/a/256360/1709587 establishes that "Try this" answers are not VLQ, and meta.stackoverflow.com/q/287563/1709587 establishes that wrong answers are not VLQ. Barring total irrelevance to the question, current consensus (as I understand it) is that "Try this:" followed by any vaguely plausible-looking attempt code that solves the asker's problem should not be deleted via the VLQ queue.
    – Mark Amery
    Jan 3, 2017 at 14:06
  • 2
    @mark First of all, there is an "Edit" option in the Low Quality Posts queue, and the linked post definitely needs to be edited. The grammar is incorrect in at least two different ways (which is pretty impressive, since there are only two words), and the code is incorrectly formatted. Both of those things needed to be fixed by an edit. If you click "Edit", you aren't clicking "Looks OK". Second, those answers are talking about why you shouldn't flag. They don't say that we need to keep these answers around in perpetuity. If you have delete vote privileges, you can and should exercise them. Jan 3, 2017 at 14:11
  • 3
    (Also, I disagree with that "consensus". I also disagree that it is a "consensus" at all. It's extremely difficult to judge whether it is, in fact, a consensus, because lots of people tend to upvote Meta answers that explain the status quo, whether or not they agree with it. As far as Shog9's apples thing, to the extent it is not farcical, it doesn't justify leaving anything that looks vaguely plausible. That's someone else's interpretation that they've imposed upon it later, although it does seem to be the one that moderators operate by, which is part of why the site quality sucks.) Jan 3, 2017 at 14:12
  • 1
    @CodyGray While I agree that the post could've been edited, I'm not sure there's much point editing garbage. I disagree about the deletion point. Instructions for mods at least are to reject any VLQ flags that shouldn't have been raised as VLQ flags; I think ordinary members of the community are meant to do likewise? (Which is a bit perverse; I guess that means that if we want such a post deleted, we should "Looks OK" it and then go to the page manually to cast a delete vote outside the queue? But then, the whole VLQ system is irrational and badly designed.)
    – Mark Amery
    Jan 3, 2017 at 14:14
  • 1
    @CodyGray (see also meta.stackoverflow.com/q/339782/1709587, my (flawed and unlikely to be implemented) attempt to bring sanity to the VLQ system.)
    – Mark Amery
    Jan 3, 2017 at 14:15
  • @MarkAmery: I don't disagree on the "editing garbage" piece, but that doesn't mean it's okay to just let it go, either. It does create a kind of double standard which has the implication that all code-only answers are okay (and it really is a case-by-case basis; this one just so happens to fail that case). In that scenario it may be best to skip those reviews and take action on the answer directly instead.
    – Makoto
    Jan 3, 2017 at 16:02
  • 3
    I agree with @MarkAmery and Floern: code-only / "try this" answers should not be flagged as either VLQ or NAA, and should not be deleted. If you think they aren't useful, then downvote them; that is the purpose of downvotes. We shouldn't be removing content that has the possibility of pointing someone in the right direction. (Of course, edits to these answers by either the author or the community should be encouraged.) I've resorted to posting a comment on such answers that both asks the author to add a description, and warns flaggers/reviewers that it shouldn't be deleted. Jan 3, 2017 at 18:22
  • 1
    @scott You and Mark keep assuming that the only options available here are "Looks OK" or "Delete". That is not at all the case. You yourself concede that editing would be appropriate, as is adding a comment instructing the person who posted it on the standards we have for a high-quality answer. Choosing not to do either of those things and clicking "Look OK" is the wrong move, a sign of an inexperienced reviewer who is making the wrong decision in the review queues. Jan 4, 2017 at 11:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .