22

This is a recurring issue on the site:

2014

There is no shame in using "Skip"

Let us bring an end to the "robo-reviewer" war: Phase 1 - 2

Are we supposed to flag bad reviews?

Additional requirement for Steward or Reviewer badge to help new reviewers learn about using "Skip"

2015

Encouraging attentive suggested edit reviewers to skip every uncontroversial review

How can we stop crazy edits like this from being accepted?

Skippy Hat: Use Winterbash to encourage Skip

2016

BAD "suggested edit" approved by reviewers

How is it possible that this edit was approved?

2017

Can we have some care when reviewing wikis edits? How can we improve our wiki edits?

Looking for stats on how frequency of skip actions correlates with audits failures rate and amount of reviews done by user

Today

I'm bringing this to meta, as it's such a clear case of poor reviewing. It needs to be addressed.

A random user decided to add some code to a question with this suggested edit. There was a comment (now deleted) asking the OP to show the relevant code. I don't know if this is what prompted the edit.

The editors intentions were to help. However, the person needs to be educated about how to use the site.

I gave a custom reason for rejection. I didn't feel the need to reject and edit it to be certain is was rejected, as it was plainly a harmful edit.

But no ... I was wrong.

It seems people need to be continuously reminded and educated to pay attention when they're reviewing and press skip if they're unsure or don't have the time to read it through.

Aside from bringing this to meta, is there anything else we can do to remind people to review with care?

I am pinging both reviewers and the editor to see this post.

28
  • 34
    IMHO we need better audits. The current ones are laughably easy to tell they are wrong. Not sure how the system would work but I think we need to start using edits that were rejected as audits. Commented Sep 12, 2017 at 18:11
  • 6
    True but I bet we would catch a lot more if we gave them something a little less obvious. Commented Sep 12, 2017 at 18:13
  • 13
    This sounds like the perfect use case for a moderator flag and not an opportunity for us on Meta to grab our pitchforks.
    – Makoto
    Commented Sep 12, 2017 at 19:11
  • 9
    Speaking of pitchforks, where did I leave mine?
    – user1228
    Commented Sep 12, 2017 at 19:23
  • 2
    Well, sometimes I feel ninja'd when dealing with suggested edit review. Some "poor" edits I think goes to "no improvement whatsoever" rejection option, but other reviewer approved it - even the OP does (e.g. approved System.lang.out.println instead of proper naming java.lang.System.out.println). Commented Sep 12, 2017 at 23:04
  • 6
    I'm really worried that you missed my point. Mere mortals don't need to do anything in these scenarios. You need to get a moderator involved instead. All we're going to do is make a whole mess of the situation, irrespective of our intentions.
    – Makoto
    Commented Sep 13, 2017 at 2:13
  • 4
    I apologize if you thought my tone was condescending. It certainly wasn't intended to be.
    – Makoto
    Commented Sep 13, 2017 at 2:52
  • 2
    @NathanOliver I once suggested to generate different audits for the edit review queue. Commented Sep 13, 2017 at 6:56
  • 2
    prediction: this is not going to happen. SE team goes to great lengths masturbating over review UI, with a-b-testing and whatnot but apparently has no time to address a simple stats request related to Skip. Looks like they somehow want most reviewers to stay oblivious of Skip
    – gnat
    Commented Sep 13, 2017 at 7:57
  • 6
    Is not using "skip" really the root cause of these bad reviews, though? Isn't it rather robo-reviewers hunting rep and badges?
    – Lundin
    Commented Sep 13, 2017 at 8:01
  • 9
    One of the two reviewers in this case has approved 177 edit suggestions, rejected 6 edit suggestions and improved 7 edit suggestions. It is pretty hard to digest that 93% of the edit requests they got were perfect. Plus, accoring to this query, their approval decisions were reversed by other reviewers 27 times (that is 14%). The other reviewer's edits have a reversal rate of 23%! Commented Sep 13, 2017 at 8:15
  • 3
    @Lundin my theory is, this happens just as often to new inexperienced reviewers who bump into slippery review items and do mistakes believing that they need to make some decision and not just skip. They differ from robo reviewers in that when they end failing audits (or even pass but find out it's too difficult to decide right) they simply drop off reviewing. Request for stats that could confirm or disprove this theory hangs ignored for about 3 months now so we probably will never know
    – gnat
    Commented Sep 13, 2017 at 8:17
  • 3
    @Lundin obviously, you've got to be kidding. On a surface it looks like a typical case when editor simply quotes the code snippet provided by inexperienced asker at some external link instead of inlining it into question, totally legitimate. I almost always skip such reviews because it is too cumbersome to check if this is indeed so (it gets especially annoying when asker provides link to code somewhere in the comments, sometimes even in comments to some answer)
    – gnat
    Commented Sep 13, 2017 at 8:45
  • 4
    @Lundin even that get's rejected sometimes... I once asked a OP to give some code, which he did happily in the comments. I took the code, prettified it and added it to the question with a clear "Added code from comments" Edit reason... It took not even 30 minutes to gather 3 REJECTs... Which by the way made me dropping the question (but that's unrelated to this topic)
    – Mischa
    Commented Sep 13, 2017 at 10:00
  • 4
    @NathanOliver IMO we need better reviewers. Lock people with filters where they can only review edits on the tags they have some score on. They will be more careful since it's something they care about (the language they know of).
    – Braiam
    Commented Sep 13, 2017 at 11:54

1 Answer 1

39

Aside from bringing this to meta is there anything else we can do to remind people to review with care?

No. And calling out specific users on Meta is not an appropriate course of action anyway.

As Makoto already said in the comments, if you see someone consistently making bad decisions about reviews, then raise a moderator flag on one of their posts. Use the textbox to provide your evidence, and then let one of us handle it. As a moderator on other Stack Exchange sites, you know that moderators have tools to see a user's review history, as well as tools to reach out to the user and/or put a stop to any further erroneous reviews.

Even though you, as a regular user, could leave a comment encouraging people to pay more attention when reviewing, it (A) just creates noise that has to be cleaned up eventually by a moderator, so you aren't saving us all that much work, and (B) rarely is as effective as when the message comes from a moderator. Besides, if there is real harm being done by the user's erroneous reviews, then a moderator needs to put a stop to any further "accidents"; a comment from you won't reach them in time, if it even works at all.


As far as using "Skip" goes…I agree wholeheartedly that the option is criminally under-used by reviewers, but…two things:

  1. As you (and gnat) already took great care to cite, we've made this announcement many times on Meta. The reviewers' failure to use it is not for the community's lack of trying. And contrary to certain other options, the review queue's design already makes "Skip" a pretty prominent, obvious choice.

  2. I think it's a red herring in the case you cite (and very likely in others). Those reviewers who approved the edit weren't unsure, so even if they were "Skip" connoisseurs, they very likely would not have used it in this case. They felt sure that it was a good edit, because it was improving the question by adding code.

    In other words, this isn't a review that anyone needed to skip. It's just evidence of a basic lack of knowledge about which types of edits need to be approved and what information needs to be considered when making that decision. If you have ideas about how to better disseminate that knowledge, and/or how to generally improve the review queues, then you should post those as separate feature requests.

5
  • 4
    top viewed "meta announcement" is less than 10K views. Even if we assume that all these views were coming from users eligible to review (2k) this makes for only about 15% of them. I somehow doubt that it's okay to call it reviewers failure not seeing it
    – gnat
    Commented Sep 13, 2017 at 9:01
  • 19
    I'm not blaming the reviewers for failing to see it. I'm calling out the limitations of Meta as a platform for conveying information to users. Another moderator on a different SE site was just telling me the other day that applying a [featured] tag to something on Meta and leaving it [featured] for a week was sufficient to ensure that essentially everybody would notice it. I laughed. Remember Documentation? Every announcement about that was [featured], but when it was discontinued, we still had people saying they never knew about it...
    – Cody Gray Mod
    Commented Sep 13, 2017 at 9:06
  • 2
    "reviewers' failure to use it is not for the community's lack of trying" that sounds slippery to me. Granted you did well at explaining that community did their best to educate and that it's not sufficient but the part of it being reviewers' failure feels somewhat lacking (unless you believe that inexperienced reviewer is guaranteed to have some super-power to see when last one of 5 buttons in review window is a preferred choice)
    – gnat
    Commented Sep 13, 2017 at 9:13
  • when on the VLQ/NAA queue, I only click on "Delete" or "Skip". "Looks OK" is very rare... So if it's bad: "Delete", if it's ... I dunno (or already downvoted): "Skip". Commented Sep 13, 2017 at 9:19
  • 3
    "Remember Documentation?" No.
    – canon
    Commented Sep 13, 2017 at 14:37

You must log in to answer this question.