68

Some time ago I've flagged a poor answer as a low quality. The review votes were:

  • 4 x Recommend Deletion
  • 1 x Delete
  • 3 x Looks OK

enter image description here Image just in case the review is too old and disappears

I don't want to blame anyone for the votes. What's more, I see that the last Looks OK was raised by the moderator. I'm wondering―and this is why I ask here―is there a possibility, that such answer will one day occur in an audit?

Another thing is―and that's why I tag it with ―wouldn't it be better if we made an obligatory tutorial on how to review? I mean... 20 different posts per review queue, which would be chosen by moderators or maybe people with the highest reviews number in a certain queue that would explain how to review this certain queue.

As Looks OK and Recommend Deletion are totally opposite I believe that the review I have attached indicates that there's something wrong. I believe, there's no doubt that reviewers could review better.

Such tutorial might also contain "STOP! Look and Listen" so people would earlier understand how important reviews are. Just my idea. I'm looking forward for your opinions.

14
  • 10
    I'd make it obligatory after first (or better yet, after second) audit failure. This way it would be easier for users understand its purpose. Also it would serve as additional throttling for robo reviewers
    – gnat
    Commented Oct 13, 2016 at 9:53
  • 14
    @gnat I think that might be a better idea. If someone wants to only dip into the reviews occasionally it might be discouraging to have to do a tutorial to be able to do so. You've had to earn some level of trust via participation on the site to access the review queues and not everyone is a bad reviewer so asking for an additional up-front test imho is a no go. However, if you've been banned from review once/twice and want to continue reviewing, then passing a test to do so might work. It'd sort the wheat from the chaff as it were.
    – Jon Clements Mod
    Commented Oct 13, 2016 at 10:37
  • @JonClements Guys, I agree. Is there available factor which says, how many of reviews the user has done ended as he voted? It could also be a reason for asking someone to do the tutorial.
    – xenteros
    Commented Oct 13, 2016 at 10:44
  • 1
    @xenteros how'd that help if there's a split vote? Don't forget that moderators also get auto-flags raised for disputed reviews.
    – Jon Clements Mod
    Commented Oct 13, 2016 at 10:47
  • 1
    @JonClements didn't know about that.
    – xenteros
    Commented Oct 13, 2016 at 10:49
  • 3
    @xenteros well - not entirely matching this case but there are some - see meta.stackoverflow.com/questions/317987/…
    – Jon Clements Mod
    Commented Oct 13, 2016 at 11:04
  • 2
    "I don't want to blame anyone for the votes.". Let me help you with that: robo-reviewers are the worst and should be banned for life. Commented Oct 14, 2016 at 23:13
  • "I see that the last Looks OK was raised by the moderator." there were no moderator involved with that review.
    – Braiam
    Commented Oct 14, 2016 at 23:35
  • @Braiam the source of confusion is probably that the last reviewer is a mod on another SE site. Commented Oct 14, 2016 at 23:40
  • OK, it's nice to discuss, but what should happen to introduce it? Is there at least a chance?
    – xenteros
    Commented Oct 16, 2016 at 16:06
  • @JonClements is there any progress in this topic?
    – xenteros
    Commented Oct 6, 2017 at 8:58
  • Not sure if Jon Clements can answer if there's any progress. Jon Clements is an elected moderator, not an SO employee. I did once bring this post to the attention of Jon Ericson, who does work for SO. He seemed to like it, but I haven't heard about it since. Commented Oct 6, 2017 at 9:18
  • @S.L.Barth I'm bad in HR :p
    – xenteros
    Commented Oct 6, 2017 at 9:19
  • While a forced (long, 20 question) initial tour would be excessive for each queue, it would be useful as an option and be required if the user fails audits. However, better would be clear and easily accessible directions as to what the decision points are for each queue. Basically, while I like this idea, I think there are lower hanging fruit which will take less developer time while having significant benefit to the quality of reviews.
    – Makyen Mod
    Commented Dec 6, 2017 at 17:37

3 Answers 3

38

This is a great idea. It would rely on the reviews being hand-picked, as the current system for generating audits is too error-prone.
The Suggested Edits queue is an exception, as those audits are generated, but there too we need to have hand-picked test cases. There are too many edit reviewers who Reject the audits there and Approve everything else. We want to teach edit reviewers to not only reject obvious garbage, but also to reject things like: spam, copy-pasted tag wikis, backticks around product names, wrong use of quote markdown...

The review tutorial should be obligatory for any reviewer coming out of a 30 day review suspension. In this case, failure in the review tutorial will have the same result as a normal audit failure: a new suspension. This would prevent robo-reviewers from doing damage between suspensions like they do now.

The first time a user completes the review tutorial succesfully, they should get progress towards a badge. We probably shouldn't give that progress multiple times, only once per queue.

The review tutorial should be optional but recommended for new reviewers. We could encourage it with a bronze badge. ("Attentive" - "completed the review tutorial without failing reviews").

The reviews in the review tutorial should occassionally be refreshed. Otherwise robo-reviewers who are forced to go through the review tutorial will learn only how to pass the reviews in the tutorial.

10

Funny that I came up with the same thing today in a different discussion. Unfortunately, this still is relevant, and this suggestion would be a great step forward.

Our review system takes getting used to. And review bans are one way, but rely on the user disrupting the system before being awarded. Lets take some preemptive action with an interactive tutorial. It only has to be 20 questions or so. Handpicked, or designed specifically for the tutorial.

I'd say make half of them clear-cut good or bad and let half of them be a bit more challenging. With the clear-cut bad questions, we should also indicate what type of flag is appropriate in the specific situation. This would be especially helpful for off-topic questions as there is a variety of flags to pick from.

In case someone picks the wrong off-topic flag, praise them for selecting off-topic and guide them to the appropriate flag indicating what were the signs they should pick up on.

2

I think it's a great idea!

Some here suggest that it should only be for those who fail audits. I think it should be for everyone. Sure, I have failed a few audits but I have never got a review ban. However, if there was a good review tutorial maybe I would not press "skip" as often as I do.

4
  • If you use "Skip" a lot, that's great! For the record, the guidelines for reviewing are on MSE. Commented Dec 4, 2017 at 7:53
  • Skipping is not a good thing per se, but it's better to skip if you're not sure. I've read the guidelines, but it can be hard to use them in a practical context.
    – klutt
    Commented Dec 4, 2017 at 9:10
  • 2
    Skipping is like sleeping - the more you mature, the less you need of it. People still learning the ropes should not feel shame to skip, for any reason.
    – Gimby
    Commented Dec 5, 2017 at 10:35
  • I don't feel ashamed. I'm just pointing out the fact that there's nothing positive per se with skipping, because when you skip you - by definition - does not accomplish anything.
    – klutt
    Commented Dec 5, 2017 at 11:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .