16

Related: Can a machine be taught to flag comments automatically?

I was thinking about automated flagging and it occurred to me that if an automated flagger can exist and survive, a literal Robo-Reviewer (a reviewer that is literally a robot) could be plausible.

From the perspective of SO/SE policy as interpreted by the moderators, would running a literal robo-reviewer be problematic?

More specifically, would the very fact of having a script perform review queue actions be seen as a violation of the rules of reviewing, or would such a robo-reviewer be held accountable by the quality of its reviews? E.g. if the robo-reviewer was reviewing at least to the standard expected of human reviewers, it would be left alone in peace to rack up badges for its owner, but if not, it (and its owner) would be subject to the same penalties (review bans) for bad reviewing as a flesh-and-blood reviewer would receive for those same bad reviews.

Would it be different if a human were actually looking at each review item, but using criteria other than their own intelligence, knowledge, and skills to perform the review? For example, if a reviewer were performing up to the level of quality expected here, but it became known that the user was rolling DC 10 Low Quality checks (or using a Ouija board, etc.) instead of actually reading the posts, would any action be warranted or would they be left alone to review in peace as long as their reviews did not fall below the quality benchmark?

I'm not asking about the programming feasibility of creating a robo-reviewer of sufficiently high quality to avoid getting automatically review banned due to failed audits or identified by moderators as one who ought to be banned due to blatantly bad reviews, only asking whether a reviewer made of a thousand lines of code would even be allowed within a hundred feet of the SO review queues.

Another way to ask this question is to ask whether the review queues are Outcome Based in the sense that what really matters is whether the correct review action is performed (regardless of how that action is determined by the reviewer - careful study, rolling a die, Ouija board, phone-a-friend, Magic 8-Ball, outsourcing to China, etc.) or whether the process itself matters (must be manually reviewed by a human, account owner must be the reviewer (no outsourcing reviews), etc.). For example, if my past reviews appear acceptable at first glance, but a moderator finds out that I have been using chicken entrails magick instead of my brain to make decisions, would they say "Good on you for finding something that works for you and that produces acceptably high-quality results" or would they be thinking about a review ban "for the principle of the thing"?

Note that I have not actually written a robo-reviewer and am not currently planning to do so - I'm more asking for the sake of knowledge about SO policies and for the sake of others who might be inclined to give a robo-reviewer a try.

3
  • 5
    Gut says "yes" because a script is rubber-stamping questions, and there isn't opportunity to actually review anything since human interaction is scant in that workflow. But let's see what moderators think.
    – Makoto
    Commented May 17, 2018 at 1:35
  • 8
    Consider: If you combine a robo-flagger with a robo-reviewer of similar judgement: all flags from from first robo will be mechanically auto-approved. That may not be what humans want.
    – Cœur
    Commented May 17, 2018 at 16:43
  • 1
    @Cœur true, but isn't that why there are manual review bans? If a moderator noticed a robo-reviewer mechanically auto-approving robo-raised flags regardless of their actual merit, they can (and should) issue a manual review-ban to said offending robo-reviewer (possibly sending him to work on Jabba's sail barge in the meantime). The question then is why the process of banning an incompetent robo-reviewer should be different to that of banning an incompetent human reviewer. Commented May 17, 2018 at 17:01

1 Answer 1

63

The only purpose of review is to get humans looking at posts that've already been identified - by machines or other humans - as potential problems.

No, really. That is it. There is no other purpose. If we didn't need that, we wouldn't have /review.

So if you're circumventing that... It's a bit of a problem.

If you think you have heuristics that are much, much better than what's currently in use to identify... spam, bad answers, bad questions, bad edits, good edits, good questions, etc... Then try 'em out. Test 'em against the human reviewers by flagging or just maintaining a database of whatever you think should be classified a certain way and see how often it is classified that way.

Then share it with the class. If you're good, we could potentially stop reviewing a bunch of stuff.

5
  • 7
    FWIW, I suspect that the secret to writing a reliable review bot would be to skip a lot. A bot would not get bored even if it had to skip 1000 reviews just to find one question/answer/edit that trips enough heuristics at the same time to make it 100% certainly and obviously bad, and the review queues on SO are long enough that even a bot that only reviewed 0.1% of them would still rack up gold badges in no time. But I'm not sure how much SE itself would benefit from reducing the review load by 0.1%. Commented May 17, 2018 at 17:45
  • 1
    Not very much, @ilmari. Truth is, I can tweak various existing thresholds to get that easily - at the cost of essentially "skipping" a tremendous volume of borderline posts that currently benefit from human attention. Knowing ahead of time which 20% of posts entering, say, Triage could be culled automatically would be a huge boon, but that's a lot harder.
    – Shog9
    Commented May 17, 2018 at 18:22
  • 1
    @Shog9 Hmm, could you explain what thresholds you could possibly tweak to review only the perfect set of posts? For those with less time/patience, that might be useful information. Commented May 19, 2018 at 23:34
  • 1
    @BlackVegetable The whole idea of review is that the automated tools are using imperfect heuristics that are increase the likelihood of a post meriting some action, but aren't actually problematic behaviors in their own right. If reviewers start looking for them then they're not actually doing the job that the computer can't do of determining if the heuristic is actually correct in that instance, or a false positive.
    – Servy
    Commented May 21, 2018 at 17:32
  • 1
    I have no idea, @BlackVegetable. A 1% change can be had easily by just ignoring posts from [higher|lower] rep users, but of course at the cost of not actually reviewing any posts from that group at all.
    – Shog9
    Commented May 22, 2018 at 0:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .