13

I'm playing around the documentation review queue, and I have noticed that it has an excessive amount of review audits.

The audit pattern seems predictable: if there's some amount of continual items for review, every other item is audit. This may lead to gaming, although the Community-generated spams are very easy for me.

The amount is excessive, the pattern is predictable, and the audit type is very limited. Is this an intended thing?

8
  • 4
    Don't know whether it's intended or not, but it keep one alert. I think that it's important to keep a high standard of reviews, specifically in that queue, to make sure that there is a good reason for any change, and that it was approved/rejected by quality reviewers that really pay attention to details
    – Alon Eitan
    Commented Apr 13, 2017 at 11:31
  • 7
    The thing is that there's only boring, spam like audits. This doesn't raise the level of reviews. Commented Apr 13, 2017 at 11:33
  • Ah, well that's another case then :) I rarely see proposed documentation changes, not sure why - I see this There are currently no pending review items matching your filters message, even though I didn't set any filter (But this is not relevant)
    – Alon Eitan
    Commented Apr 13, 2017 at 11:39
  • It's likely the frequency of audits is to make up for the limitations of the audit generation process; like Suggested Edits (and unlike every other queue), it synthesizes fake edits from real posts and a simple nonsense algorithm. These audits are easier to spot, but there aren't likely to be any good alternatives. Commented Apr 13, 2017 at 12:08
  • There probably is a lack of poor examples, as the documentation isn't around long yet. Therefore random spam audits are created, to have atleast some audits.
    – Luuklag
    Commented Apr 13, 2017 at 13:08
  • @Luuklag you obviously haven't spent much time in documentation.... it is full of horrible examples. Granted many have probably been rejected....but that history still exists
    – charlietfl
    Commented Apr 14, 2017 at 16:07
  • @charlietfl: The main problem is that identifying those automatically is very challenging. Audits need to have a comfortable margin of obvious correctness, and there's no obvious way to identify most of those except manual verification, which isn't considered a suitable audit-generation technique. Commented Apr 14, 2017 at 21:57
  • @NathanTuggy I can imagine! Time constraints aside, even doing it manually seems to me like it would be a tricky task since scope and types of issues are so broad and the whole thing is so free form
    – charlietfl
    Commented Apr 14, 2017 at 22:14

1 Answer 1

12

Proposed changes are very similar in function to suggested edits, so we use the same audit generation process for both. The primary goal of these audits is to catch people who are approving changes without spending much time reading them. Believe it or not, the audits do catch people who aren't paying close attention. And some folks fail audits often enough they get blocked from reviewing for awhile.

The audit system tries to deliver more audits to people it thinks aren't very careful. So your individual experience might not be representative. Some people see very few and others see many audits. If it feels like you are seeing too many, might I suggest reviewing isn't a race? (N.B. the algorithm isn't perfect and doesn't know for sure if you are doing careful reviews. But it's often good at guessing whether users are guessing.)

1
  • 14
    Could we also have Documentation audits, where questions are posted as Documentation? This might trip up a few more robo-reviewers. Commented Apr 14, 2017 at 7:36

Not the answer you're looking for? Browse other questions tagged .