52

Please see https://stackoverflow.com/review/suggested-edits/5881383

The editor removed the snippet, yet, 3/4 of the reviewers approved it. On a side note, they didn't even fix my typo.

Why was this approved?

18
  • 84
    Because of robo-reviewers.
    – ChrisF Mod
    Commented Sep 29, 2014 at 12:37
  • 9
    Yes, the stats of those who approved it are 224 approved/14 rejected, 154/13, and 190/34. That of the user who rejected it is 365/198.
    – Mooseman
    Commented Sep 29, 2014 at 12:54
  • 4
    @ChrisF any consequences or just stating the obvious?
    – user2140173
    Commented Sep 29, 2014 at 14:12
  • 47
    @vba4all - If I get some time then there will be consequences.
    – ChrisF Mod
    Commented Sep 29, 2014 at 15:05
  • 5
    My first thought was "do you really need a runnable code snippet to demonstrate a text area? Seems a bit OTT" but in context I can see why you decided to do so. Maybe the reviewers just went with the first thought. Commented Sep 29, 2014 at 18:58
  • 9
    @MartinSmith how dare you discourage the snippets™... snippet all the things!
    – canon
    Commented Sep 29, 2014 at 19:23
  • @ChrisF: Consequences AND repercussions? Commented Sep 29, 2014 at 19:35
  • @AbraCadaver - I can't possibly comment.
    – ChrisF Mod
    Commented Sep 29, 2014 at 19:37
  • 29
    The problem it shows is that there is a reward (badges) for clicking your way through and (almost) no penalties for doing it. If I recall correctly, somewhere on Meta, a reason was given along the lines: It helps reduce the review queue size despite some bad reviewers. I entirely stopped reviewing except when I see edits directly in the tags I follow -- outside the queue). Because it usually takes more than few seconds to take a decision. While I try to be apply the usual criteria (does it alter OP's meaning, does it add anything meaningful, etc), it's been approved a long time ago...
    – P.P
    Commented Sep 29, 2014 at 20:34
  • 7
    I wonder, is there any way to objectively measure the quality of reviewers? It's interesting to me that robo-reviewers appear to have a higher percentage of approves over rejects. Could the system give people with those stats more review audits perhaps?
    – DavidG
    Commented Sep 30, 2014 at 10:55
  • 3
    @DavidG Sounds like a feature-request, if it hasn't been suggested already. But it won't help until we get better audits in the Suggested Edits queue. Commented Sep 30, 2014 at 11:02
  • @S.L.Barth Agreed. However, the "better" audits can cause much discussion and frustration. The vandalism edits are obvious -- albeit, perhaps too obvious -- enough, that you ought to see them if you're even half-looking at the post.
    – Mooseman
    Commented Sep 30, 2014 at 11:08
  • 2
    @Mooseman: the "better" audits should be generated. The LQ and CV queues suffer from poor but highly upvoted posts used as audits, I don't want to see that in the SE queue. So SE audits shouldn't be taken from edit suggestions that were Accepted by 3 people, that's a recipe for disaster. Commented Sep 30, 2014 at 11:13
  • Got the same problem but the other way around. I edited the OP's question to make it more sensible as comments were stating the question was unclear. The edit was not approved and the question was later deleted.
    – Rerito
    Commented Sep 30, 2014 at 11:28
  • 2
    @Mooseman, correct. But I edited after the OP was satisfied with my answer (which showed I understood the question correctly).
    – Rerito
    Commented Sep 30, 2014 at 11:42

2 Answers 2

18

Because the edit review system isn't working well. The main flaw is that the system relies on the human edit reviewers to be responsible, careful and actually understand the edit policies.

Look at the statistics from the 4 people who reviewed the edit:

Accept_reviewer1 has approved 224 edit suggestions and rejected 14 edit suggestions
Accept_reviewer2 has approved 159 edit suggestions and rejected 14 edit suggestions
Reject_reviewer1 has approved 370 edit suggestions and rejected 201 edit suggestions
Accept_reviewer3 has approved 190 edit suggestions and rejected 34 edit suggestions

Experience from the flood of crap edits that the site is bombarded with suggest that a careful suggested edits reviewer should roughly have an accept/reject ratio of 50/50 (plus minus some depending on how picky they are).

You can tell that the "reject reviewer" is doing his job well by the realistic number of rejects. Similarly, you can tell that the three others are robo-reviewers, who accept pretty much anything.

As for suggestions about how to solve this, that has been discussed in detail elsewhere.

8
  • 14
    In general, a good analysis. Still, there are some who will skip instead of reject. Also, removal of "too minor" should have changed the rates to favor approval some... Commented Sep 30, 2014 at 12:13
  • 2
    This is a "who watches the watchers" problem. With SO's community moderation paradigm, why can't we have mutual review powers for users above 10-20k? This would allow bad reviewers to be reviewed in turn, possibly by other bad reviewers. Profit!
    – mechalynx
    Commented Sep 30, 2014 at 12:28
  • Edit reviewer review queue :P Actually it is a good idea, though it might be better just to raise the rep cap needed for 1) suggesting edits and 2) getting access to the edit review system.
    – Lundin
    Commented Sep 30, 2014 at 14:20
  • 6
    @Lundin If only high rep were a guarantee of good behaviour...
    – Rawling
    Commented Sep 30, 2014 at 14:35
  • 2
    Wait wait wait, you're saying that someone who has a more even ratio of accept/reject did the right thing!!?
    – gunr2171
    Commented Sep 30, 2014 at 15:25
  • 1
    +1 for the comment by Deduplicator: I hardly ever reject an edit (even if intuitively I would consider it as "too minor"), but instead, just skip it.
    – Marco13
    Commented Sep 30, 2014 at 15:34
  • 1
    We could have an "approve edit power" that grows as (A) your number of total reviews grows, and (B) your accept/reject approaches 50%. Say (approve%)*(reject%) * log_2(total reviews), with a dynamic amount of total "vote power" required to accept an edit (that grows as the size of the edit approval queue approach 0). (accept%)*(reject%) is the amount of information your vote gives (someone who always accepts doesn't produce information when they vote), and the log_2 portion gives more weight to people with more expertise. Commented Sep 30, 2014 at 17:46
  • 2
    @Deduplicator - Could we show the number of skips as well? "misterManSam has Accepted 200 Edits, Rejected 180 and Skipped 500". If robo-reviewers rarely skip, then this would be a good metric to weed them out from those users who skip rather than reject. Commented Oct 1, 2014 at 3:01
49

The edit review system makes rejecting edits harder than approving them.

The result is that those who reject edits are discouraged, and review slower. Those who approve edits are encouraged, and review faster.

This results in people doing edits getting told "good job, do another like that" with a review success, even if the edit in question was poor. This creates a second order effect of more poor edits.

Approve is a one click step. Lots of encouragement. You do have to spot blatantly horrible random spam "tests" and say "that looks like a test". I have yet to see a test that looks merely like a bad post.

The next two are a pair:

Improve Edit requires editing the post, and counts as an approve.

Reject and Edit requires editing the post from scratch.

So if the poster fixes 7/20 errors in the post, you either have to approve, or redo the 7 fixes. So for these two paths, approve is clearly the one the website encourages you to use if the edit contained anything useful. Incomplete/lazy edits are approved more often than they should be, encouraging more of same.

Next we have Reject. If you click on it, you now have to look at a restricted set of options, all of which are narrowly constructed in what seems to be an attempt to discourage you from picking any of them. You either need to do research to justify the rejection, consider the edit attempt to be a misclick, random spam, or prove to yourself that the edit was sufficient to make the original post incomprehensible. Or you can go with "other" and put forward a custom argument why the edit should be rejected, which by association with the other reasons is implied that you should only be done in extreme circumstances.

To vote Copied Content honestly, you'd have to do research and find that content if you are being honest. We are talking orders of magnitude more work than approve. And it doesn't apply here anyhow.

Invalid Edit seems aimed at someone clicking the wrong button, or thinking that edit is the same as comment. Your post doesn't seem to match this -- it looks like an edit, if a crappy one. These happen, but this is again a narrow option from its description.

Radical Change could apply. By the letter of the description, it only applies if the edit would lose the original meaning of the post. While the edit in question does do serious damage, the original meaning of the post remains: "use instead" -- the example is damaged, but the meaning is not lost, as required by the reading of the "refuse edit" choice. Even to detect this damage, you have to either mentally parse the markdown, or click on the render markdown, and determine that the editor had made a mistake. So while I would pick this, by the plain words it doesn't even apply to the edit in question! And it is, again, much harder to justify to oneself than "meh, approve".

Vandalism might apply, but it is not defacing, it is not spam, and it isn't "inappropriate" in the context of defacing/spam.

Which leaves Other, where the person rejecting the edit has to type out "the person who did the edit removed screwed up and caused a code sample to disappear". Again, orders of magnitude more work than Approve.

The powers at be at stack overflow have regularly made the Reject reasons narrower and added hoops to jump through in order to reject an edit, while Approve remains easy, and the tests are good enough to stop a "drinking bird". The result is that the suggested edit queue is full of people who keep on getting rewarded for nearly mindlessly Approving anything that seems vaguely reasonable (ie, isn't random gibberish). Those that Reject reasonably often end up taking so long that even if they matched the number of mass Approvers, they would lose out on most votes.

So the Rejectors do orders of magnitude more work, and see their Rejects getting overridden. Over time they find the options for rejecting made more narrow, and extra hoops added, discouraging them even more.

In short, it was approved because stack overflow edit queue causes bad edits to be approved.

6
  • 25
    RIP "too minor".
    – gunr2171
    Commented Sep 30, 2014 at 15:21
  • 11
    I gave up reviewing Suggested edits due to this, your answer is spot on. Commented Sep 30, 2014 at 19:21
  • "Improve Edit" in particular destroys your will to live. Like InfiniteRecursion, I've given up editing - I'm fed up with risk of slaps on the wrist for getting it wrong, and no realistic chance to improve bad edits. Commented Oct 2, 2014 at 8:53
  • There is a little button named Skip, I am sure they dont even see it.
    – UmNyobe
    Commented Oct 2, 2014 at 16:18
  • @Yakk people would rather approve an edit than skip it.
    – UmNyobe
    Commented Oct 3, 2014 at 7:54
  • @UmNyobe I do not see how skip matters. It doesn't generate an approve nor a reject: it is "abstain", roughly the same as closing the browser window. The edit remains in the queue until certain number of people approve or reject it, skip except insofar as it makes a biased ratio of otherwise approvers/rejectors skip rather than vote does not impact this. I have no idea what ratio of otherwise approvers/rejectors it causes to skip rather than vote, but it serves an important purpose: "I have no idea, so will not speak". Commented Jan 19, 2015 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .