27

Somebody just pointed out this to me where 3 robo reviewers didn't even have time to properly read the edit.

Only one of those reviewer even has gained rep (a whole 3) in the php tag.

I don't know whether the suggested edit is even correct or not, but what I do know is that neither of those three reviewers could have possibly reviewed the thing.

Am I missing something here? If not: can they be review banned just based on the time spent on the review task? Or do we actually have to wait before somebody sees a wrong review based on content instead of time?

So I guess my question is: should I flag these kinds of reviewers when I encounter them and my assumption is correct about the review time?

15
  • 1
    The thing is that sometimes there are edits which you can review and reject probably in seconds. For example spam edits or if someone submits an edit for the wrong example and you don't want to get banned for that. So I love your idea, but I'm worried that also a lot of good reviewers could get banned.
    – Rizier123
    Sep 15, 2016 at 14:29
  • 1
    It's still requires somebody to manually flag them (me in this case). But I somewhat see your point.
    – PeeHaa
    Sep 15, 2016 at 14:30
  • 2
    @Rizier123 That's the job of the mod to give them eventually the benefit of the doubt…
    – bwoebi
    Sep 15, 2016 at 14:31
  • 7
    How could you have possibly known it added value and wasn't broken in all kinds of glorious ways in less then 2 minutes?
    – PeeHaa
    Sep 15, 2016 at 14:33
  • I really want to get robo-reviewers banned, but I don't want that good reviewers get banned. So maybe a mod gets a notification about "fast reviewers" or something like that.
    – Rizier123
    Sep 15, 2016 at 14:33
  • @Rizier123 nobody has been talking here about automated bans, or am I missing something?
    – bwoebi
    Sep 15, 2016 at 14:34
  • 2
    I'm always wary about using "time" as a metric to make sense out of anything... I'm someone who speed-reads, so I am sure I would trigger a LOT of these flags. But I would like to think that my reviews are on par. looking at the example linked, I could probably read an assess it in < 5 minutes if I was more familiar with PHP. So maybe not an auto-ban, but some kind of flag (with a manual review)would still make sense.
    – Patrice
    Sep 15, 2016 at 14:34
  • @Rizier123 I once asked Shog9, in the Tavern on the Meta chatroom, if the speed with which people reviewed was considered. He said it was; but I don't know what SO does with that information.
    – S.L. Barth
    Sep 15, 2016 at 14:35
  • @bwoebi From the post: "If not: can they be review banned just based on the time spent on the review task?"
    – Patrice
    Sep 15, 2016 at 14:35
  • @Patrice I understood it like "Will mods review ban people solely based on the criterion of time spent on reviewing?"
    – bwoebi
    Sep 15, 2016 at 14:36
  • 1
    @bwoebi so... what's the difference if it's automated or done manually, if the only criterion used is time? Whether it's the system going "too quick -> ban" or the mod going "too quick-> ban"... who cares? the problem is "too quick->ban" without ANY other criterion, not who does it
    – Patrice
    Sep 15, 2016 at 14:38
  • 3
    @Patrice I mean the size/time ratio to be concrete. I.e. small edits are fine to be reviewed in 10 secs… larger ones need one or two minutes at least.
    – bwoebi
    Sep 15, 2016 at 14:41
  • @bwoebi but again, if the action is "too low ratio-> ban", who cares if the system does it automatically or it's a mod. the problem is the "ONE criterion". If we only look at one thing to make a decision, who cares who is the one pushing the button? The issue is the system only looking at one metric.
    – Patrice
    Sep 15, 2016 at 14:43
  • 1
    The difference @Patrice is that it will get pre-filtered by users because they have to manually flag it before it's even being taken into consideration.
    – PeeHaa
    Sep 15, 2016 at 14:43
  • @Peeha and see, that is something I'm okay with. If there is some kind of manual process involved, where a person has to validate if the edit makes sense or not, then that means no one will get banned simply for reading/assessing quicker, as long as the assessment is correct. The "someone has to look, realize it's bad, and flag" is the thing that brings this out of "welp, you just read too fast, so you're banned" territory I'd like to not fall in :P
    – Patrice
    Sep 15, 2016 at 14:44

1 Answer 1

15

Robo reviewers should be review-banned on bad reviews.

As has been pointed out in the comments, some people really can read and judge fast. And in some cases a problem is easy to spot, like most spam postings are.

However, the users who can read and judge fast are the exception. Somebody reviewing at breakneck speed is almost always a robo-reviewer.
And although some cases are easy to judge, most require a bit of reading carefully - it is extremely rare to find a series of subsequent spam posts in any review queue.

The system should (and probably does) alert moderators if a user is reviewing at extreme speed.

What we also need is a better workflow for the moderators; Brad Larson pointed out that the workflow for suspending robo-reviewers is difficult. Since review is an important part of quality control, it might be worth the SO developers time to improve that workflow.

Obviously, the Documentation queue also needs audits, which as I understand it is still in the works. As I see a lot of copy/pasting over there, I hope there will be audits where content is copied directly from Wikipedia or official sources.

When flagging a robo-reviewer, try to find evidence of actual bad reviews. This makes it easier for the moderators to decide if someone is robo-reviewing. Personally, I look for 3 bad reviews in the last 2 days; this should be sufficient evidence that there is a pattern of sloppy reviewing, and not just an unfortunate mis-click.

0

Not the answer you're looking for? Browse other questions tagged .