12

Yesterday I learned the hard way (ouch, that hurt) that I really have to work on my discipline when doing further "edit reviews". Which is fine; life is about learning and improving. Leading to this question:

One of the "hints" I read in the comments/answers I got is: having 11.5 times more accepts than rejects ... is probably already an indication of sloppy reviewing. Now I am wondering: what do you think an "OK" acceptance rate should look like?

In more detail: there are many edits/reviews per day. So in the end, the "laws of big numbers" kick in. Meaning: every reviewer should (on average) see edits of the same "quality distribution" in the end (unless you start skipping all those edits that are not your "favorite" tag)

Thus: when you would compare the "approval rate" of "good" reviewers that worked on hundreds or thousands of reviews, I think those rates could be within the same ballpark. Going from there:

  1. Are there any pages somewhere that contain such (accumulated) statistics?
  2. In case you are such a "good" reviewer with hundreds, thousands of reviews - what is your "rate"; and do you agree with that idea that alone that number can be a first indication for problematic reviewing?
11
  • 9
    Approval rate should not be an issue, it's just a hint that something off may be happening but it is in no way conclusive. My stats are probably mostly rejecting a lot more than accepting, that's because I focus on not letting crappy edits pass through. The real issue though, is about approving spam edits.
    – Tunaki
    Oct 18, 2016 at 12:56
  • 10
    If you really want to have fun you can skip all the stuff you think should easily be accepted and instead look for the harder stuff, stuff to be rejected. This works as the robo reviewers will approve the stuff that should be reviewed and you put reject votes on the stuff that should not be. Oct 18, 2016 at 13:17
  • 1
    @NathanOliver But I guess your "algorithm" is incomplete. It would need that final "and as you will be banned all three days for disagreeing with other reviewers you will have to regularly flag reviews so a moderator can come in and declare you-the-minority-rejecter to be correct". Which btw sounds like a good idea to collect "helpful" flags.
    – GhostCat
    Oct 18, 2016 at 13:29
  • 10
    @GhostCat You don't get banned for disagreeing with other reviewers. You get banned for failing audits, which are so obvious that you should never actually fail them, or from a moderator banning you for taking an action so clearly inappropriate that you ought to have known better. Of course, if you see people clearly reviewing inappropriately then you can flag for a mod to look at it.
    – Servy
    Oct 18, 2016 at 13:34
  • 7
    @GhostCat This has nothing to do with "you-the-minority-rejecter". There were links to edits you approved in this other post, like this one. This is putting a link to a, hum, not appropriate site. As the answers are pointing out, there is no "right approval rate", but this is clearly not right.
    – Tunaki
    Oct 18, 2016 at 13:34
  • 1
    Ah, makes sense. Thanks! Good to understand that there is no "automatism" there.
    – GhostCat
    Oct 18, 2016 at 13:36
  • 1
    @GhostCat As others have stated you will not get banned. What you will find is people who are approving things they should not. You can then use a mod flag if you think it is bad enough to have the mods look into the users reviews and decide if they need a review ban. Oct 18, 2016 at 13:56
  • Am I the only one thinking that the approve:reject ratio should be 1:2 going towards more rejections? I feel I see a different site than you guys, never could I ever approve more edits than reject them. There is waaaay too many crap edits, useless edits, edits that leave plenty to be still edited, downright wrong edits, attemps to reply... I don't know. I don't understand how anyone can have approved more than rejected edits. Oct 20, 2016 at 14:04
  • @FélixGagnon-Grenier Thing is - when I started editing I do recall reading "something somewhere" that one should value even small edits; like even fixing just a typo here or there. That is why I approved such edits. Stupid thing: searching for guidance this week; I only found information that goes in the opposite direction; like the FAQ on the answer I accepted.
    – GhostCat
    Oct 20, 2016 at 14:34
  • @GhostCat yeah, there is conflicting opinions on that :) I see it now as: minor edits are valuable, but only when they also fix everything that is to be fixed in the post. Turns out, the vast majority of edits I review that fix minor things lack the fix of other problems. I either reject and edit or improve and edit, if the suggester did all they could in good faith, but lack english proficiency to go further. Oct 20, 2016 at 15:09
  • I could decide to do reviews however I want, as can anyone else. I could decide to only Accept, or only Reject, or what I Skip, or anything else. The stats are partial stats of reviewing workflow. As long as you aim to do the best within your workflow, the stats mean nothing. My workflow was basically as @NathanOliver outlined, so mine are very much skewed towards Reject, with Accept for well-made edits that are not necessarily "easy" to evaluate. Counting time spent and counting Skips would change the stats, but not make them really meaningful. Oct 21, 2016 at 12:25

5 Answers 5

12

A high ratio of Approve:Reject could indeed be an indicator of bad reviewing, at least on Stack Overflow.

Assuming that the quality of edits suggestions is normally distributed, there probably is such a thing as an Approve:Reject ratio that matches the distribution of good and bad edits.

But.. there are a few things that throw a monkey wrench into this machinery.
The first is that we cannot see how many edits a reviewer has Skipped. This is a variable that would need to be accounted for.
The second is that Stack Overflow recently started rate-limiting the amount of edit suggestions that a user can make. This has helped to prevent edit sprees and retagging sprees, which has (probably) changed the distribution of good and bad edits.

A better variable, however, would be how fast a reviewer reviews, and how often they choose Skip. Fast reviewing is a bad sign; Skipping a lot is a good sign.
That's all these numbers are - signs. A fast reviewer is not necessarily a bad one, but most of us cannot review correctly at breakneck speed. A reviewer who uses Skip a lot is not necessarily a good reviewer, but it does suggest they know when to withhold their judgment.

I think your real question is how to be a better edit reviewer, and what early warnings you can use to see if you're on track. I'm not sure about what early warnings we can give, but for what it's worth, I proposed a FAQ for editors once. Several users have contributed to it. Although it is aimed at editors, it may be useful for reviewers as well.

6
  • 1
    Accepting your answer as I find it the one that is "most along the lines" of my question. If I could, I would accept the other answers as well.
    – GhostCat
    Oct 18, 2016 at 15:01
  • 6
    As an anecdotal thing, I tend to skip a lot of reviews when reviewing suggested edits; generally, unless it's obviously good or bad (or my brain has decided to temporarily close shop) I skip, especially if it edits something that I'd need domain-specific knowledge to check if it means the same thing.
    – Nic
    Oct 18, 2016 at 15:13
  • 7
    We don't ban reviewers based on review rates or speed of review. In fact, if you just tell us in a flag that someone approves a lot of edits or reviews quickly, but don't provide any specific examples of bad reviews, we might not do anything with your flag. It's particular instances of bad reviews that I care about. I've known many people who could review quickly and accurately, or whose negative reviews ended up being on deleted spam or non-answers, so they didn't show in their history. It's the decisions themselves that matter.
    – Brad Larson Mod
    Oct 18, 2016 at 15:19
  • @BradLarson The question, IMO, is not about what should be flagged for moderator attention. It's what signs a reviewer has that their reviewing is off, before moderators need to get involved. I thought my answer made it clear that review rate is a sign and not a proof... but I'll consider editing it, to make that more clear if necessary. Oct 18, 2016 at 15:34
  • 1
    How fast reviews go is entirely up to what reviews you get. When reviewing CVs for instance, you can often see that a Q is closable in 3 seconds or so, if its really bad. Same for edits. Sometimes, you see that its bad very quickly.
    – Magisch
    Oct 19, 2016 at 13:31
  • 5
    @Magisch It's a sign, not proof. But I've seen reviewers do 14 (!!) edit reviews in a minute. That is a sign that the reviewer may not be paying attention. In practice, there's always a few reviews in between that require you to look and think carefully, before passing judgment. Oct 19, 2016 at 13:36
11

I don't think that "approval rate" of a reviewer can directly tell you whether they are a good reviewer. For example, someone could approve only suggested edits that should be approved, and skip all the others. There's nothing wrong in such behavior.

However, in practice when someone approves many more suggested edits than they reject, it can tell you that probably they review incorrectly (note that it's only "probably"—you can't judge only by numbers). If you encounter such a situation, you can investigate further, and if you find several suggested edits that this reviewer incorrectly approved, you can notify a moderator about that.

I think that what Tunaki wrote in a comment is exactly what I'm trying to say:

Approval rate should not be an issue, it's just a hint that something off may be happening but it is in no way conclusive.

9

As a >10k user, you have access to the review history for all reviewers. From the last 50 reviews (at the moment I'm writing this), 42 (84%) have been approved, 5 rejected and 3 approved and edited. 84% is (too) high but it could be that some robo-reviewers are active at the moment a stream of high-quality edits has just been posted. You can get more details with a SEDE query - this one shows the aggregates per day and shows that 78% is the current average.

For me, that's too high - my personal approve/reject rate is about 2 to 1. Of course I think that's a good rate (at least on Stack Overflow) but YMMV.

The SEDE query:

SELECT 
  dateadd(day, datediff(day, 0, creationdate), 0),
  SUM(CASE WHEN votetypeid = 2 THEN 1 ELSE 0 END) 'Approve',
  SUM(CASE WHEN votetypeid = 3 THEN 1 ELSE 0 END) 'Reject',
  100 * SUM(CASE WHEN votetypeid = 2 THEN 1 ELSE 0 END) / COUNT(*) 'Approval %'
  FROM suggestededitvotes
  WHERE creationdate > '2016-09-01'
  GROUP BY dateadd(day, datediff(day, 0, creationdate), 0)
  ORDER BY dateadd(day, datediff(day, 0, creationdate), 0) DESC
0
2

There is such a vast amount of robo-reviewers, that looking at the site average approval rate will likely mean nothing.

Given that you review close to all suggested edits and don't skip a lot (which is a perfectly fine thing to do), you will in my experience end up somewhere around 50/50 accept/reject, give or take, depending on how pedantic you are.

New reviewers are almost always overly optimistic and end up with very positive ratios. Usually because they don't know all edit policies yet. But also since a lot of them are robo-reviewers who give up/get banned from edit reviews quickly.

I have no proof, but it really does seem that your accept/reject ratio drop over time, the longer that you review.

I think there are usually slightly more of the good edits than the bad ones, though if you review long enough, you'll now and then encounter someone who's gone on a bad edit spree or copy-from-wikipedia-to-tag-wiki spree, meaning you will usually end up with 20 out of 20 rejects that day.

1

I feel that if you approve more than 75% of all edits (counting all approve/skip/reject) you likely approve edits that don't meet "suggested edit" bar.

I have small number of reviews (~500) with rate 3:1, I think I skip less than 10% of reviews. I usually get to review from posts directly and don't sit in review queue.

There are plenty of suggested edits that fix one single problem in the post instead of all - those get "reject and edit". There are edits adding "thank you" to posts, and edits of posts that can only be fixed by owners.

Note that thinking "would I make that edit myself" is not enough to consider the suggested edit to be of good quality - the bar is higher than one for direct edits as it takes up to 5 people to review the change.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .