-7

I've recently spent some time on the Triage review queue, and have been getting an insane amount of review audits (for example, 1 in 5 posts is an audit). This contributes less to the site, as I spend my time doing the audits and not actually reviewing posts. I think the amount of review audits should decrease when the following factors go up:

  • Reputation
  • Percentage of audits passed
  • Net helpful flags (helpful minus rejected)
  • Number of reviews completed in that queue (or in all queues) Thanks to The Thonnu
  • And possibly more.

What do you think?

4
  • 2
    Maybe the total number of reviews completed in that queue could be a factor as well? For example, someone with a Steward badge should be getting less audits than someone without one.
    – The Thonnu
    Oct 17, 2022 at 16:41
  • 1
    Also, this looks like a feature-request. Is it a feature request or simply a discussion question?
    – The Thonnu
    Oct 17, 2022 at 16:42
  • 1
    @TheThonnu I thought it's a bit more of a discussion, because this is not likely to be something we agree on, but sure, it can be a feature request.
    – pigrammer
    Oct 17, 2022 at 16:44
  • 1
    I find it a hard topic to reason about. Because audits are not really a verification method, they're more there to keep you on your toes and at all times wary. But they still suck in their current form.
    – Gimby
    Oct 18, 2022 at 10:09

2 Answers 2

9

Personally, I think there should be more audits, not fewer.

A trend among reviewing the queues is that people can tend to go on auto-pilot, and make some sloppy misses here and there. The point here isn't to let someone review a lot and then forgive a handful of mistakes, but it's to ensure that the reviews are high quality.

So, more audits in the queue - as ineffective in terms of being incapable of doing nothing else but just making sure you're paying attention - is enough to ensure that reviewers like yourself are, at least, not on auto-pilot.

You're noticing more of them so that's Mission Accomplished™ in my book...

12
  • Well, when the factors I mentioned go up, then that means the user is less likely to mess up, so they can get less audits.
    – pigrammer
    Oct 17, 2022 at 16:46
  • 1
    I don't usually review in the Triage queue, so I haven't seen this for myself, but if it is really around 1 audit every 5 posts, don't you think it's a bit overkill? I mean, yes, audits are good, but that many?
    – The Thonnu
    Oct 17, 2022 at 16:46
  • 1
    @pigrammer: No, you could just be on an amazing caffeine run, or be extremely clear-minded for one set of reviews. Past performance isn't an indicator of future results, after all.
    – Makoto
    Oct 17, 2022 at 16:47
  • @TheThonnu: I'm making some allowances for there to be hyperbole there, but honestly? Out of every 100 things you review, having 20 things to sanity-check that you're paying attention doesn't sound that bad to me. I'd think that it'd be a bit much if for every 100 you reviewed you got 60 audits, and I think that it should be a little less than a 1:1 review:audit ratio, but 5:1 today isn't excessive.
    – Makoto
    Oct 17, 2022 at 16:49
  • 1
    But isn't that why we have the need for multiple people to review? If one person isn't reviewing right, we still have other people who will review it right, because it stays in the queue until two people agree. One person messing up isn't the end of the world.
    – The Thonnu
    Oct 17, 2022 at 16:51
  • @TheThonnu maybe a bit less, it varies slightly. For example, during today's triage run, I did all 40 reviews and got 5 or 6 audits.
    – pigrammer
    Oct 17, 2022 at 16:53
  • 1
    @TheThonnu: Actually, it's pretty bad if one person messes up. Given the volume of posts on the site, you have no way of ensuring that one reviewer will pick up where another one left off. The system is...a bit inefficient in that sense, in that it trusts that the reviewers are autonomous and capable of doing good reviews in relative isolation of one another. On any other Exchange site this isn't an issue since there's orders less posts to review, but on this site, each review is assumed distinct.
    – Makoto
    Oct 17, 2022 at 16:53
  • 1
    @TheThonnu: A very prominent reviewer got put in a very long time-out for making lots of those misses. Sure, the ratio of good-to-bad review is still very low, but looking at the numbers or the encounters of reviews completely misses the point.
    – Makoto
    Oct 17, 2022 at 16:55
  • 5:1 sounds awfully excessive to me. Oct 17, 2022 at 18:09
  • 40:5 is 8:1. I smell hyperbole and anecdata. Put some real numbers and data around this, @pigrammer. Track how many audits you get over the next week, and how often you are passing and failing them, then report back. A single run where you got an 8:1 ratio is hardly definitive, especially since there's a random element in there. Oct 17, 2022 at 20:32
  • @HereticMonkey The ratio was 5:1 when I wrote the post, then it droppped to 8:1 after I finished reviewing.
    – pigrammer
    Oct 17, 2022 at 20:35
  • @pigrammer That just solidifies the need for better data. You'll likely find that after a few more attempts the ratio will increase further. Oct 17, 2022 at 20:37
7
  • High reputation means you're good at answering or asking questions, not necessarily that you're good at reviewing.

  • Audit frequency is already dependent on how often you pass or fail them. Pass more and you'll get less of them, pass fewer and you'll get more of them.

  • I guess if you flag you know what spam is or what rudeness is. I don't see how that helps with knowing how to do reviews.

  • I don't necessarily think if you do hundreds of poor reviews, you should get fewer audits. There has to be better quality control than sheer numbers.

0

Not the answer you're looking for? Browse other questions tagged .