46

What follows is a two-phase (pre-access & post-access) "battle" plan to defend SO from future "robo-reviewers",


Phase one (pre-access)

  • Only allow users to access the suggested edits queue if,

    • <W% of their suggested edits are rejected,

    • they have made >X suggested edits (or have an edit badge),

    • more than Y% of their raised flags are deemed "helpful".

    • they have raised >Z flags.

Phase two (post-access)

  • Increase the number of reviewers needed to approve an edit.

  • Increase the difficulty of audits.

  • Increase the frequency of audits.

  • Increase the ban period for reviewers that successively fail audits.

  • Increase the required number of votes to approve an edit (add greater "weight" on reject votes). (Servy's idea).


Ok, all joking aside; these are a few of my suggestions to help at least reduce the number of "robo-reviewers" that are actively approving bad edits (I'm not saying we should implement all of them; I'm just "airing" out ideas that may be useful).

I believe reputation has never been a good metric to measure a user's ability to approve edits, which is why I'm suggesting that we should use the user's editing + flagging history to better determine if the user is suitable or not. Along with harder audits and increased number of reviewers needed to approve an edit we should hopeful be able to drastically reduce the current number of "robo-reviewers", and therefore (slightly) raise the average post quality on SO.

36
  • 70
    "they have made >X suggested edits," - I haven't made any suggested edits as I got to 2K rep before they came in. So under your scheme I wouldn't be able to access the suggested edits queue.
    – ChrisF Mod
    Commented Jun 11, 2014 at 14:05
  • 7
    @ChrisF People that want to review edits should have at least made an edit and know how to make edits. Otherwise how else can you better guarantee that the user isn't going to approve a bad edit?
    – Sam
    Commented Jun 11, 2014 at 14:07
  • 27
    I appreciate that - but as an "old hand" I haven't made any suggested edits so your test is insufficient. I do have the "Strunk & White" and "Copy Editor" badges though....
    – ChrisF Mod
    Commented Jun 11, 2014 at 14:11
  • 6
    I used to contribute edits before registering to SO for a long time, as per your scheme, unregistered users won't be able to propose any edits. They don't do it for rep, their contributions are often very good and honest. Commented Jun 11, 2014 at 14:17
  • 6
    @ChrisF "sorry, you may no longer review suggested edits --Community". But I only have 37 so I may not make the cut either. Even though I probably have a higher reject:accept ration than many others. But, in all fairness, Sam did say he didn't expect all of these to be implemented and I like the effort because I'm tired of reject silly/bad edits just to see them approved.
    – codeMagic
    Commented Jun 11, 2014 at 14:20
  • 4
    @ChrisF Ok, so how about, "they have made >X suggested edits, or have and edit badge"?
    – Sam
    Commented Jun 11, 2014 at 14:24
  • 9
    @Payeli - it's not about restricting people making suggested edits, but restricting people reviewing them.
    – ChrisF Mod
    Commented Jun 11, 2014 at 14:28
  • 2
    Thanks @ChrisF. Understood now. But then robo-editors will increase, they will race to make more edits->access the queue->get badges. Currently there is only one incentive for robo-editors, that's rep. Then there will be two incentives, rep+queue access. 2k users will make unrestricted edits just to get access to queues. Pushing users to make edits for the sake of being eligible to review later, that's scary. Commented Jun 11, 2014 at 14:43
  • 1
    @Payeli Yes, but to get into the queue you would have to have a good accepted:rejected edit ratio, so robo-editors would also be stopped (or at least reduced).
    – Sam
    Commented Jun 11, 2014 at 14:46
  • 1
    Why don't we just get some sort of "moderation reputation" based on moderationy actions? Commented Jun 11, 2014 at 14:51
  • 3
    Maybe you can make meta participation as a mandatory criteria. Like have at least x meta posts with y upvotes. Once they come to meta and see the consequences others face, they will improve. Bans are good, and awareness of prospective bans and troubles might deter them. Commented Jun 11, 2014 at 15:01
  • 9
    The main issue here is that it creates a feedback loop. If it ends up being a positive feedback loop, then that's great, but if you're not very careful, it could just as easily turn into a negative feedback loop. Currently people that just go around spamming super minor edits (often with a rather large percent of invalid edits mixed in) and end up with a very large percentage of their edits approved. Over 80% in my experiences, often much more. If that is allowed to happen at all, then they become reviewers, and continue to accept others making similar bad edits.
    – Servy
    Commented Jun 11, 2014 at 15:19
  • 7
    Since these bad edits are currently being largely accepted, if there isn't a major intervention when this system is first put in place, these same reviewers will keep approving bad edits, letting bad editors continue to become bad reviewers. There needs to be some form of intervention before a change like this is made to ensure that, at least for some short probationary period, we really only approve good edits, so that the people that get into the system are the people who know what should be approved, and so that a positive feedback loop is allowed to be created.
    – Servy
    Commented Jun 11, 2014 at 15:21
  • 1
    Didn´t read everything so far, but PLEASE not more audit. They are annoying enough. (This is not because of personal bad experience. I think I failed nothing in my >300 edit reviews)
    – deviantfan
    Commented Jun 11, 2014 at 15:32
  • 1
    @deviantfan Before audits were in place the suggested edit system was completely in shambles. Approximately zero suggested edits would ever be rejected, this included obvious vandalism, spam, literally anything. If it was suggested, it got approved, usually without ever being read by any of the 2-3 people that approved it. After the audit system was put in place the system went from "making the site worse through its assistance" to "causing a lot of problems, but still doing enough good to at least be worth existing". There were a lot of changes, but audits were the most important.
    – Servy
    Commented Jun 11, 2014 at 15:44

8 Answers 8

28

With that

I believe reputation has never been a good metric to measure a user's ability to approve edits, which is why I'm suggesting that we should use the user's editing + flagging history to better determine if the user is suitable or not.

I do completely agree.

When you join Stack Overflow, you are not very familiar with how this site works. It's different from many other programming-related sites. Some of us jump right in into the FAQs and MSO/MSE and figure out on their own what this community is like, what is considered a good/bad edit, when and how to properly flag. Others learn by method of trial and errors.

From own experience, I assure anyone who ever suggested an edit, flagged something for attention etc. that they will get educated by the response to their actions (suggestions/flags) if they keep monitoring it. This is a great way to learn about the site.

I can't even remember the amount of times I used to get frustrated by having a flag rejected or suggested edit rejected while I thought -   I was right!. There have been multiple times I have researched meta (the old SE before the split) and found out I actually wasn't right!... I also have asked questions to clear my doubts as well as suggested my own ideas to the community what could potentially be improved.

See.. I still haven't mentioned reputation...

This, may have taken a while ( over a year ) but now I am rather confident about reviewing, passing audits, even pointing others in the right directions when they are wrong. I am only human so I do make mistakes but in comparison to what I was "playing" a year ago I have progressed a lot! By participating, earning editors badge, suggesting edits etc you actually do learn a lot about how this site really works and how to help moderating it this is why I completely agree with you and support your idea that reputation shouldn't matter when it comes to "Should I be eligible to review?"


About your points

I like all three except the first one

W% of their suggested edits are rejected,

realistically, you're right. You should have some edits rejected because no-one is perfect but I have seen people with 100% approval rate who surprisingly are not banned and the reasoning even more surprising; where other people said it's possible because you can skip everything you're unsure about and only accept quality edits. I personally disagree with this argument but since someone else mentioned it here on meta while talking about robo-reviewers I am taking this option to consideration.

The other 3 points I do agree with

they have made >X suggested edits (or have an edit badge),

As explained above; everyone should have made at least >X - say X = 50 to get at least a rough idea of what it is like to submit a suggested edit and to see how it has been interpreted by more experienced reviewers.

more than Y% of their raised flags are deemed "helpful".

they have raised >Z flags.

raised flags > 100? Would anyone agree? This could be backed up by a very similar reasoning from the above - you should have had a few flags submitted and monitored the outcome before you are allowed to judge other peoples' flags...

Overall great idea! I am supportive all the way :)

7
  • Interesting, only 0.827% of all SO users have a 100% approval rate.
    – Sam
    Commented Jun 11, 2014 at 16:03
  • 1
    @Sam I think calculating an average approval rate and then calculating the % of people below the average would be somehow more accurate of an indicator :)
    – user2140173
    Commented Jun 11, 2014 at 16:09
  • 8
    Note that it's less than "W% of their suggested edits are rejected" - I think you missed that, and misinterpreted it - presumably we're not talking about reviews here, but edits (i.e. the user needs to have edited a few posts). Commented Jun 11, 2014 at 20:22
  • @Dukeling I see what you mean. But how are you supposed to get some of your suggested edits rejected while there are 25K users who have not rejected a single suggested edit... that's the point here. See the point about some people believing that it's "normal" and possible not to reject any suggested edits for whatever reason.
    – user2140173
    Commented Jun 12, 2014 at 7:06
  • 5
    You don't have to have had any of your suggested edits rejected to review - that's what's meant by "less than" here - you can review if anywhere between 0% and W% of your edits have been rejected. Commented Jun 12, 2014 at 7:56
  • @Dukeling OH! but I don't think I have originally misunderstood the author, I think you have twisted it around as "W% of their suggested edits are rejected" definitely is not the same as you can review if anywhere between 0% and W% of your edits have been rejected
    – user2140173
    Commented Jun 12, 2014 at 8:03
  • 2
    I'll clarify what I meant, a user can only access the queue if less than W% of all a user's edits have been rejected. Or looking at it the other way round 100 - W% edits would have to be approved to access the queue.
    – Sam
    Commented Jun 12, 2014 at 8:05
23

Regarding limiting access: we used to have a much higher bar for reviewing suggested edits. With waaaay more people editing than reviewing, they just piled up. Did you know it's possible to fill the queue? That's right - there's a limit to the number of pending suggested edits the system will allow, and once met no more edits are allowed. That wasn't great.

We could probably afford to raise the bar a little bit though. More on that later... First, your suggestions:

  • Increase the number of reviewers needed to approve an edit.

Done: SO already requires at least 3 people to review each edit, as compared to two elsewhere.

  • Increase the difficulty of audits.

Define "difficulty". Chances are, the only real option here is hand-picked edits; that's been talked about a fair bit, and isn't a bad idea - but it'll take some work to implement. Have no doubt, these will very likely be quite difficult for many reviewers...

  • Increase the frequency of audits.

The frequency is adjusted automatically by the system based on your past audit history.

  • Increase the ban period for reviewers that successively fail audits.

We already do this too.

  • Increase the required number of votes to approve an edit (add greater "weight" on reject votes).

So, increase the incentive for folks to just approve trivial edits decline anything moderately hard to read? How does that help, exactly?

Here's the big problem: for all of the complaining about lousy reviews and bad edits being approved, there's remarkably little agreement on what, specifically, is wrong. Is it...

  • ...Folks changing too much?
  • ...Folks changing too little?
  • ...Formatting vandalism?
  • ...Rep for nothing and privs for free?
4
  • 1
    Wouldn't the correct action on a review that is moderately hard to read be to skip it? As you pointed out it takes more energy to reject an edit (you need to find the right reject reason) than to improve, isn't that a good reason to increase the reject weight?
    – Jack
    Commented Jun 12, 2014 at 15:49
  • 2
    "Improve" already carries more weight than either Approve or Reject, @Jack. Increasing the weight of Reject alone would only make sense if a majority of edits submitted were not helpful; I'm not really convinced that's the case.
    – Shog9
    Commented Jun 12, 2014 at 17:39
  • I don't think the majority of edits are not helpful, but I think it does take less time to approve then to reject, this might just help balance it out (perhaps I shouldn't have used the word energy before, but one follows from the other). However it's probably better to find some way to force/guide those approving to do a better job rather then trying to compete time wise.
    – Jack
    Commented Jun 12, 2014 at 17:45
  • Did it get as bad as the close vote review queue?
    – tbodt
    Commented Jun 14, 2014 at 2:24
10

As an extremely casual queue reviewer, all of the audits I have encountered have fallen into two categories:

  • Blatantly obviously an audit
  • Incorrect.

I'm not exaggerating. Every single time I've been told I've failed an audit, it was because the question or answer that provided a basis for the audit had been mis-handled in the past.

That needs to get fixed first.

8

I don't think your criteria are the best.

  • <W% of their suggested edits are rejected,

This one actually makes sense, but perhaps we should include a time aspect - if they've had many edits rejected long ago, but none recently, it would make sense to allow them to review.

  • they have made >X suggested edits (or have an edit badge),

There's some logic here, but it's flawed.

  • They can make bad edits, which can get approved all the same.
  • They can judge themselves different from how they judge others.
  • They can only make obvious edits to get past this criteria, and have completely wrong opinions on anything remotely borderline or totally inappropriate.

As long as we educate them first (we're lacking this part in reviewing currently AFAIK), I suspect there's only an extremely small fraction of people who will learn from making edits (I also suspect there's a lack of feedback when it comes to rejected edits, which would make this criteria only serve to enforce the first).

  • more than Y% of their raised flags are deemed "helpful".
  • they have raised >Z flags.

Flags have next to nothing to do with editing.

It's like letting the dog sleep inside because he can fetch a stick (as opposed to having learnt to do its business outside).


I believe we only need this:

  • Increase the difficulty of audits (related).

And:

  • Start an "internship" system where your reviews don't actually count / count a whole lot less at first (related).
3
  • 1
    Only make the audits harder if we get a flag/challenge system. There are enough tricky/borderline audits as is. Commented Jun 11, 2014 at 20:57
  • @BradleyDotNET They can be a quite a bit harder without being particularly tricky / borderline, but I agree that we should be able to flag them (this isn't really required for suggested edits at the moment, but it should definitely be implemented because of the close vote review audits). Commented Jun 11, 2014 at 21:05
  • post edit, to the first two points I'd just say that I could get behind a time-based aspect. What I'd probably do is say something like, when you have at least N suggested edits and your most recent N suggested edits have an approval rating of X%, you become a reviewer. (If you only base it on time then I could suggest N edits, have them all rejected, then suggest one that gets approved, and become a reviewer.) But I do like the idea of providing some means of allowing a bad editor who reforms of having a shot at redemption.
    – Servy
    Commented Jun 11, 2014 at 21:07
4

From my post on MSE:

It seems like most robo-reviewers accept because that's just one click, as opposed to three for rejecting. Therefore, making it take more clicks to approve a bad edit should slow down robo-reviewers.

If an edit has any reject votes, show a warning while clicking approve. It would say something like "This edit already has x reject votes. Are you sure you want to approve it?" It would have an "approve anyways" button there. Then there could be a line, and the standard reject form right there.

1
  • This would only make robo-reviewing easier. The robots would approve anything baring warnings and reject or skip the others. This way, they'd just minimize the risk of being caught and you'd get even fewer helpful reviews.
    – maaartinus
    Commented Sep 12, 2017 at 16:43
3

These are great suggestions. I would tweak the post-access part a bit though.

Phase two (post-access)
•Increase the number of reviewers needed to approve an edit.

This carries a risk of just getting more robo-reviewers.

•Increase the difficulty of audits.

This carries a risk of more false negatives (which do occasionally happen).

•Increase the frequency of audits.

They already seem to be quite frequent.

•Increase the ban period for reviewers that successively fail audits.

Maybe.

•Increase the required number of votes to approve an edit (add greater "weight" on reject votes). (Servy's idea).

This is along the right lines.


My additional suggestion would be, that the system should let all reviews that have been started, count towards the final result. That would put a stop to the frustrating situation where you read an edit carefully, decide to reject or improve it, only to see "this edit has already been accepted".

4
  • I'm not entirely sure about that, since that opens up the chance that an edit might pass, but have enough open reviews that it eventually is rejected. However, not being able to disagree with the group by hitting reject on an accepted edit can be rather annoying. Commented Jun 12, 2014 at 14:17
  • Actually, that is the whole point, that 3 robo-reviewers who click an immediate "yes" might be able to be overriden by people who take more time. Commented Jun 13, 2014 at 9:27
  • Then you have the problem of edits being accepted, presented to people and then reverted? I'm not sure that is desirable. Commented Jun 13, 2014 at 11:06
  • 1
    Why on earth would you implement it like that? Far more sensible to say something like "if 3 reviews have started, then don't start any more until 3 results have been accepted (or timeout)", or even "if 3 results have been gained, calculate the end result on these 3 plus all outstanding reviews" (which would lead to different numbers of reviews per edit). Commented Jun 13, 2014 at 12:34
2

If people are robo-reviewing to get badges, then encourage them to use the Skip function by counting 'reviews skipped' towards badge progress. Or remove the badge altogether for certain types of reviewing.

SO is incentivizing reviewing by offering badges and a progress bar; but also disincentivizing the process via review audits/bans, and trying to think of ways to fight human nature of taking the easiest path to success. Edit reviews are currently nearly instantaneous (if I edit a post, it will be approved or rejected in 1 minute or less), so why encourage people to review when there are enough willing people already?

I would also recommend fully removing review incentives if there are issues - don't even show a number getting bigger on the profile page and certainly don't have a 'top reviewers' page.

3
  • 2
    Perhaps most of them are there only because of the badge? Perhaps only a few of them are there because of the badge, but they robo-review, so they're responsible for the queue never building. But it might be a good idea to investigate this. Commented Jun 11, 2014 at 21:11
  • 3
    Disable the badges and see if it affects queue size or review quality. If it is positively affected, leave them off. If the queue gets large or reviews stay poor, turn them back on.
    – bd33
    Commented Jun 11, 2014 at 21:14
  • The badges aren't the sole cause of problems, because this was a widely known problem well before there were reviewing badges, but the badges made the problems quite a lot worse, so they're certainly a major contributing factor to the problem of bad reviewers.
    – Servy
    Commented Jun 11, 2014 at 21:19
-1

A while back I noticed an icon that seemed like it needed to be addressed up in the header of the page. Clicking it took me to the review edit page. Reviewed edits on topics I had past experience with, skipped ones I did not. Rubric for me has been, would I want / accept this edit if it was to my own question, answer.

I see the icon, I review edits.

One day I see a edit that to me seems to clarify the point a little more. Nothing major, but seemed helpful to someone reading the question.

Audit! Failed it. Didn't improve the quality of the question. It was a trivial edit. Scary Warning.

Probability of addressing the queue in the future = ((Good Will / Cost to Contribute) * 0) * 100.

4
  • 1
    +1 The audits are tricky enough as is (hence all the disputed ones). We need a challenge system if we are going to try and make them "harder". Commented Jun 11, 2014 at 20:56
  • 1
    So in order to ensure that we have enough reviewers we should refuse to warn/stop/educate users who make incorrect edits? Um...no? If someone isn't reviewing properly, and is unwilling to learn when told that they aren't reviewing properly, why would we want them to keep reviewing? If they leave voluntarily because they were told they aren't doing their job correctly then that's fine. Certainly better than not telling them they didn't do the right thing. Unless you're simply trying to suggest a more polite wording for audit failure, in which case, do you have a proposal?
    – Servy
    Commented Jun 11, 2014 at 21:01
  • 2
    @BradleyDotNET The problem is that there are too many audits that are just wrong, not "hard". We need a way of ensuring that the audits always expect the correct action, especially if they are there to ensure that reviewers are good reviewers, and not just "paying the slightest bit of attention", less we teach them to take the wrong action. That's a hard problem to solve.
    – Servy
    Commented Jun 11, 2014 at 21:03
  • @SList A simple metric like "fail all edits < 10 characters" is woefully inadequate, not just for the reason you cite, but because sometimes a substantial edit will have a net result of +/- only a few characters. I have abandoned suggesting an edit for this reason on more than one occasion.
    – ClickRick
    Commented Jun 13, 2014 at 12:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .