30

To my mind, a review audit question should be so obviously good that if you're paying any sort of attention, you'll immediately be able to spot it as an audit. Recent audit questions I've been getting are... not obviously good. I obviously don't know what the algorithm that selects these questions is doing, but based on my observation it tends to favour high-upvoted questions.

That's a problem when you get tech support tickets posted as questions. Now, maybe this was just a fluke, but could we maybe add a little bias to the algorithm so that it prefers older (i.e. actually high quality) high-upvoted questions for auditing purposes?

13
  • 8
    If you can spot that it's an audit, then you undermine the purpose of audits. The point of auditing people's reviewing is not just to catch people robo-reviewing. It's to make sure that they're reviewing up to our standards. That means you need to catch people who are close voting inappropriately, or not closing appropriately. And so forth. Commented Nov 26, 2017 at 21:49
  • 4
    That question has 16 straight upvotes. What other criteria would you suggest?
    – Martijn Pieters Mod
    Commented Nov 26, 2017 at 22:59
  • 7
    @NicolBolas The point of SO's audit system is to make sure reviewers are paying attention. With SO's implementation, a reviewer detecting an audit means the system is working in one of its intended ways.
    – TylerH
    Commented Nov 27, 2017 at 0:27
  • 4
    I don't see how just waiting longer will have the desired effect. If it sat around for that long and attracted only upvotes and no other moderation activity, waiting until it's 2 weeks or a month old is only going to delay how long until the audit gets chosen, not prevent it. A mishandled post is a mishandled post no matter how long it takes to get selected.
    – animuson StaffMod
    Commented Nov 27, 2017 at 3:16
  • @MartijnPieters Exclude anything from the [android] tag as an audit?
    – Ian Kemp
    Commented Nov 27, 2017 at 5:13
  • 3
    @IanKemp: we have plenty of excellent [android] posts that are audits. There will always be flukes, in any tag. If you think it is a bad audit, downvote the post or vote to close it (either of which disqualify it), and move on.
    – Martijn Pieters Mod
    Commented Nov 27, 2017 at 7:48
  • 11
    @NicolBolas IMO audits should only catch robo-reviewing. If one day, an algorithm is able to decide if a post follow the changing "SO standards" or not: then human-reviewing will be useless.
    – ben75
    Commented Nov 27, 2017 at 10:00
  • 2
    Yeah, if spotting an audit undermined the purpose of audits, "Review Edit" audits wouldn't look so much like the "editor" had a stroke.
    – Daniel F
    Commented Nov 27, 2017 at 13:32
  • 25
    The main problem with the audits is that they are catching people who review correctly and add the incorrect review towards a review ban when they review correctly. This has been discussed endlessly on meta, in fact it feels like 80% of meta is about this. The consensus is 1) Yes the audits are broken, and 2) no, SO will not do anything about it, they are busy with more important things like monochrome top bar icons.
    – Lundin
    Commented Nov 27, 2017 at 13:59
  • i mean... 16 upvotes is a decent criteria to look at, but it's certainly possible for a post to get a large number of upvotes and still be terrible. Waiting longer won't change that. Maybe just blacklist android questions from being audit examples of good quality posts.
    – Kevin B
    Commented Nov 27, 2017 at 22:49
  • @KevinB That's what I suggested, but Martijn Pieters doesn't seem to agree. I know that blacklisting a particular tag is a little extreme, but considering the overall quality of Android questions are miles below even the poor-quality questions that are now the accepted norm on SO...
    – Ian Kemp
    Commented Nov 28, 2017 at 4:51
  • 1
    @NicolBolas "If you can spot that it's an audit, then you undermine the purpose of audits." -- then audits are doubly broken, because with a little reviewing experience under your belt it's relatively easy to develop a feel for what an audit question looks like and check it out before reviewing. I don't do much reviewing any more because the system is so broken, but when I do I find it quite easy to spot audits. Commented Nov 29, 2017 at 3:57
  • 1
    The target question is now closed as off-topic. Meta effect? Will that question now be an audit failing people if they DON'T flag it for closure? That would be the epitome of irony. Commented Nov 29, 2017 at 4:01

2 Answers 2

4

How about this:

  • Select a post as a potential audit based on the current audit-selection criteria.

  • Show this post to some random users (with high audit pass rates) in Review to determine whether it's actually valid as an audit.

    It would not be an audit for those users, closer to a regular review (whether it should be different from a regular review is up for debate).

  • If the number of users taking the same action as expected is above some threshold (or just 100%), make it an audit.

  • If it's below the threshold, don't make it an audit.

6
  • I wonder how much spurious reviewing this would add to an already-bogged-down review queue, particularly for close votes? If we do something like this, perhaps we could also decrease the threshold required to review the close vote queue to increase the throughput?
    – theMayer
    Commented Nov 28, 2017 at 18:48
  • 1
    The whole point of audits is that there are lots of people using the review queue and doing things wrong. If you let other people in the review queues be the ones to judge what action should be taken, then whey those other reviewers do it wrong (as they inevitably will) you have bad audits.
    – Servy
    Commented Nov 28, 2017 at 18:49
  • @Servy Note that I'm suggesting this as an additional validation step for the existing system, not a replacement of it, and I'm not suggesting we go in completely the other direction than what the system thinks (the only alternatives are "make it an audit requiring the result the system already wanted" and "don't make it an audit"). Commented Nov 28, 2017 at 19:08
  • @Dukeling And yet the result of that additional step, particularly for the queues that perform the worst, is bad audit being kept and good audits being removed.
    – Servy
    Commented Nov 28, 2017 at 19:10
  • @Servy I didn't say it was a perfect suggestion, it just seems better than what we have now, which is, AFAIK, the system selecting audits based on what an arbitrary portion of the user base thinks - at least with reviewers there's some quality assurance beyond "managed to post some useful posts", and these are users who actually want to help (except for those who just want badges). We probably need to better educate reviewers if we want them to improve, but that's a separate issue. Commented Nov 28, 2017 at 19:16
  • at least with reviewers there's some quality assurance beyond "managed to post some useful posts" But there isn't. All you have with reviewers is that they posted enough "useful" posts to get the rep minimum to review. and these are users who actually want to help (except for those who just want badges) That's a pretty big exception though. That's precisely the problem. If they were a small minority this whole thing wouldn't be an issue to begin with.
    – Servy
    Commented Nov 28, 2017 at 19:28
1

I asked the same question which generated a fairly high degree of interest.

As some of the commenters on this point have mentioned, the purpose of auditing is not simply to catch robo-reviewers, but to reinforce site standards. It says so explicitly on the audit, once you've failed.

I think it's fair to say that if an audit is to catch robots, it shouldn't be a product of a robot itself. Manual cultivation is required. And if we can't do that, let's up the privilege level for reviewing, disable the audit system, and be done with it.

10
  • the purpose of auditing is not simply to catch robo-reviewers, but to reinforce site standards That's not what SE said when they created it and implemented it. It has been brought up as a possible side benefit, but it's not the goal of the system.
    – Servy
    Commented Nov 28, 2017 at 18:51
  • 1
    And if we can't do that, let's up the privilege level for reviewing, disable the audit system, and be done with it. This has been shown to not be a solution. Reviewer quality is simply uncorrelated with reputation once you get to about the level where you're using most of the queues. When you look at the stats of people failing audits enough to get banned regularly, it's not just the low rep users.
    – Servy
    Commented Nov 28, 2017 at 18:52
  • Regardless of what the intent was, the reality is that the system is utilized to enforce standards. If not, it would not contain the words "you should click 'leave open'..." when you fail.
    – theMayer
    Commented Nov 28, 2017 at 18:52
  • 3
    Does the audit system actually add value (more than its cost, anyway)? Has this been asked or studied?
    – theMayer
    Commented Nov 28, 2017 at 18:54
  • It's there to ensure that people are "trying". It's not there to ensure that they're actually doing a good job. It's not attempting to do that, and it doesn't often accomplish that. (It occasionally helps people improve, incidentally, but it's not a primary purpose, nor something it consistently does.)
    – Servy
    Commented Nov 28, 2017 at 18:54
  • Yes, it adds value, as can be seen by how unusable the system was without audits, which was the case for quite some time. The problems that they cause (which is honestly mostly just some frustration in reviewers ever once in awhile when they hit a bad audit, and need to learn to ignore it) absolutely pale in comparison to the amount of damage they've prevented (keeping out thousands and thousands of users just clicking buttons randomly to increase their review count, causing enormous amounts of damage).
    – Servy
    Commented Nov 28, 2017 at 18:56
  • 4
    Seems like it would be easier to get rid of the review count
    – theMayer
    Commented Nov 28, 2017 at 18:58
  • So people abusing review was indeed a problem before the incentives were in place (the review count, the rankings, the badges, etc.), but those incentives absolutely made things worse. Lots of people (myself included) said many times that these incentives should be removed, as they only ever encouraged people to do reviews who weren't interested in reviewing well. You can see for yourself how convinced SE was of that argument.
    – Servy
    Commented Nov 28, 2017 at 19:00
  • 1
    Up until about a year ago I actually enjoyed reviewing. However my reviewing has dropped off to near zero. The reason for this was that I was actually trying hard and still somehow failing edge audits for bad audit examples that had a couple up votes from uninformed users. At the same time there was heated debate about the meanings of the different audit flags. For months I was fine and then all of a sudden my flags started getting rejected. I was then in "audit hell" as there is a post on this; basically my flags weren't aging off fast enough and one of the audits put me back in fail.
    – Shawn
    Commented Nov 29, 2017 at 21:06
  • (cont.) and whats worse is flags weren't being reviewed for a couple weeks. So I could get re-banned on audits from weeks ago and I could not go pull those flags off once I knew their meanings had changed. I got tired of it and basically stopped reviewing. I just won't deal with the frustration of being punished for something with an always changing subjective meaning and with different meanings on different queues.
    – Shawn
    Commented Nov 29, 2017 at 21:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .