32

Why aren't there any review audits in the suggested edits review queue that are correct?

I've noticed that every single audit I've come across in the suggested edits queue should be rejected for spam. It seems like the system just adds random words in random places, making it pretty obvious that it's an audit.

What's to stop a reviewer from simply rejecting every post as spam or vandalism? They will never fail an audit, because all of the audits seem to be due to spam/vandalism.

Obviously, it would be hard to generate correct audits in the suggested edits queue, so why don't we just make them correct audits only if they are approved by all of the reviewers.

Also, those with access to moderator tools can review reviews according to the help center, so maybe only make it a correct audit if a user with moderator privileges doesn't take any action on the reviewers' choices. (I'm not really sure how /tools works, so I don't know how this suggestion would work, if it would at all)

10
  • 5
    Wild guess: because it is to prevent robo-reviewer who is only clicking "Accept/Looks OK" without even reading.
    – Andrew T.
    Commented Mar 24, 2015 at 4:58
  • 4
    @AndrewT. That is true, but the problem still stands that people can reject every post as spam or vandalism, and even if they didn't it's fairly obvious when it's an audit, you just need to look at the post for about a half of a second.
    – Jojodmo
    Commented Mar 24, 2015 at 5:20
  • 2
    If this happened there'd be reviews that the reviewers disagreed with, so you then need some reviewers to review the reviewed reviews...
    – Ben
    Commented Mar 24, 2015 at 5:49
  • 3
    fear of introducing these was justified when there was a "too minor" rejection reason - but now that too minor is gone, "known good" audits probably can be implemented
    – gnat
    Commented Mar 24, 2015 at 7:13
  • 3
    @gnat I still see a lot of bad edits getting approved. The only way "known good" audits are going to work is if they are selected manually. Commented Mar 24, 2015 at 8:47
  • 9
    @S.L.Barth well one way I can think of is to generate such audits. Pick a known good audit post from FP/LA/CV queues, decorate it by adding "hi/thanks", decapitalising "I" and generate edit suggestion to strip that back
    – gnat
    Commented Mar 24, 2015 at 8:52
  • 1
    @gnat That could work. It should probably be done with old posts that have a number of upvotes. Those tend to have proper grammar and spelling, so you'd get the right source material for the audit. Commented Mar 24, 2015 at 9:00
  • As a side note, all reject reasons are treated alike AFAIK. Certainly in other queues the distinctions between the various "this is kinda lousy-ish" responses are not upheld at all. So even additional known-bad audits with different reasons would still be passed with the same "reject with anything" response. Commented Mar 24, 2015 at 14:45
  • I think I reall that reviewers whose votes deviate too frequently from the final consensus already get flagged for moderator attention. Since most edits are approved, that dos not works so well for someone who approves everything, but 100% rejections should stick out.
    – HugoRune
    Commented Mar 24, 2015 at 21:43
  • I suggested an even stronger variant of this in meta.stackexchange.com/questions/267165/…, with some preliminary analysis of the results. Commented Feb 17, 2016 at 1:55

2 Answers 2

19

So... The short answer here is that we couldn't figure out how to make or identify bulletproof known-good edits when we were setting up audits for that queue.

I actually think we could probably get away with something even simpler than the tactic used for known-bad audits though: just create a fake revision that trashes the post, adds spam or something, and then make the "edit" the actual current revision of an otherwise well-received post.

2
  • 15
    But elsewhere, users are advised against editing out spam links. So I can see reviewers rejecting the edit for that reason. Alternative suggestion: use Peter Mortensen's edits for audits.
    – user3717023
    Commented Mar 25, 2015 at 2:57
  • How about all 3 approved in a limited period of time? (may be by users with certain reputation/review edits stats, for instance by those who rejected at least as much as 0.5 * times they approved edit and no less than 100)
    – YakovL
    Commented Sep 7, 2017 at 9:49
9

Audits in the suggested edits queue are primarily meant to stop the one-click robo-reviewers – that is, those who mindlessly click Approve, Approve, Approve, etc.

Robos who use Reject, Reject, Reject, etc. are far less common because rejecting an edit requires 3 clicks at a minimum and, well, these are robo-reviewers, looking for the minimum amount of effort required to get badges. That said, note that it does happen.

The reason why audits in suggested edits are painfully obvious is due to their subjectivity. What should and shouldn't be added to a post through edits is a hotly-contested issue, and that isn't going to change any time soon. Everyone can agree that injecting spam into a post is something that can be declined, and ideally, all audits should be obvious if you are paying attention.

Audits are selected algorithmically, so having a "positive audit" is something that would be difficult to implement, but not impossible. If it's something that would be useful enough for SE to implement, it can't rely on reviews like audits from other queues do, because only three are needed to approve a suggested edit, and a lot of edits are approved that really shouldn't be.

Because you can't vote on edits like you can on posts, it's difficult to distinguish an "okay" edit, which could be borderline and possibly incorrect, from a great one (both of which could be unanimously approved).

So, the audits would have to be generated (like they are now for the "bad" kind). Perhaps it could take a high quality post, deface it by lowercasing "I"s and inserting random commas or unblocking code (or something similar to that), and then show the edit as restoring it to what the actual post looks like?

6
  • But not all audits are spam injections. I copped one that was in Russian, and could clearly be improved by translation into English, and when I did so it was a valid question. I've had others that were also very dubious, and I haven't done too many so it seems that the quality of audits is rather low.
    – user207421
    Commented Mar 25, 2015 at 1:07
  • @EJP Really? I've always gotten insertion of nonsense text (enough so that I can confidently reject as spam/vandalism even if it were real).
    – AstroCB
    Commented Mar 25, 2015 at 1:34
  • Really. See here.
    – user207421
    Commented Mar 25, 2015 at 8:50
  • @EJP Ah - I wasn't clear. I was speaking of suggested edit audits.
    – AstroCB
    Commented Mar 25, 2015 at 10:32
  • Blimey, You mean there are different systems? Why?
    – user207421
    Commented Mar 25, 2015 at 22:32
  • @EJP Mostly for the same reasons described in the answer, actually; I think that's part of what this question was asking.
    – AstroCB
    Commented Mar 25, 2015 at 22:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .