146

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the low S/N ratio and the users willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

39
  • 35
    Just a nitpick, but I think the failure messages are trying to alienate users who fail. The idea is that we don't want people reviewing who don't know what they're doing. Of course, that logic only works if there aren't bad audits, and judging by the frequency of complaints here on Meta, it does seem that the audit system is imperfect. Interesting question, I'd be curious to see this data as well.
    – Cody Gray Mod
    Commented Feb 9, 2016 at 5:51
  • 2
    That people actually manage to fail suggested edit audits is evidence they must work at least sometimes.
    – TZHX
    Commented Feb 9, 2016 at 6:01
  • 12
    Hmya, this is backwards. There is no audit to check users' competence at asking questions or posting answers or editing posts. That's because those are already easily measurable. Review audits were instituted because reviews are not easy to measure and SO users were complaining bitterly about the quality of reviews. If you want to audit the audits then you first need to propose an objective measure of success. If you find one then we don't need audits anymore. Commented Feb 9, 2016 at 6:04
  • 8
    No one complains on meta when they pass an audit, and very rarely are complaints about failed audits actually bad audits. That said, I have found instances after the fact where an audit was actually bad, but the reception on meta was abhorrent. As with on the main site, you could post the same question on two different days, and get a surprisingly different response each time. You can take a bad audit out of the audit pool by down voting and close voting the question (or down voting the answer if that is the case.
    – user4639281
    Commented Feb 9, 2016 at 6:32
  • 22
    From my (very subjective) point of view, the audit system is a complete failure. I got a 30 day suspension before, so take what I say with a grain of salt. I used to just review things as I saw them, and that worked fairly fine until I got hit with a few (in my opinion) BS audits that got me review banned. Now, instead of just reviewing how I see posts, I usually first check if its an audit (I've since had BS audits as well but at least I knew it was an audit and could research the arbitrarily "correct choice the system wanted from me beforehand) and then review.
    – Magisch
    Commented Feb 9, 2016 at 7:09
  • 17
    So in conclusion, audits currently only serve to annoy me.
    – Magisch
    Commented Feb 9, 2016 at 7:09
  • 6
    It must also be said that if I was interested in robo-reviewing, audits wouldn't stop me. If you know how to look for it, its usually painfully obvious what is and isn't an audit. You can even automate looking at wether or not it is one with a userscript, so I question the general sense of them, too.
    – Magisch
    Commented Feb 9, 2016 at 7:53
  • 10
    Somewhat subjective, but I definitely think audits have improved my review quality. After I got review-banned for a day and then for a week, I started to skip a lot more things I was unsure about, and think about what I was going to do before clicking the button. Is this an accepteable, reasonably good quality post ? Does it answer the question ? I guess I also know a lot more about the site now than I did when I had 500 rep. Commented Feb 9, 2016 at 8:49
  • 5
    @JimGarrison IMO that particular question can be salvaged by editing. Remove the request for off-site documentation and it becomes a question about syntax. There must be better examples of bad Triage audits. Commented Feb 9, 2016 at 9:52
  • 23
    It's very clear that you never used the review system before the audits were put in place. LIterally a significant majority of reviewers never read any post and simply clicked the same button on every review, unconditionally, until they were out of reviews, every day. The queues as a whole largely had a net negative effect on the site as a result, as spam posts sent into the queue, instead of being flagged and deleted, would get upvoted (if in first/last post), approved (if an edit) etc. You literally couldn't get enough people to reject obvious spam edits.
    – Servy
    Commented Feb 9, 2016 at 14:24
  • 4
    If you were to remove audits (and not replace them with a new feature to accomplish the same goal) then you'd be better off removing the entire review system rather than go back to what it was like before. All that said, audits don't make someone who's actually trying to review, review better. They just make sure the people that aren't trying at all don't use the review system at all. They've (largely) accomplished that goal. They of course haven't done what they aren't even attempting to do and make people actually reviewing review better.
    – Servy
    Commented Feb 9, 2016 at 14:25
  • 3
    @TomášZato I would strongly encourage you not to release that script. If you do, and people use it, then it means more sloppy reviewing will go undetected. That means more bad/destructive edits will be approved, more garbage questions will live on when they should be closed, and so on. In other words, it makes the entire site less useful for everyone.
    – elixenide Mod
    Commented Feb 9, 2016 at 19:55
  • 3
    @TomášZato Thanks for not breaking the system! :) I agree, many of the audits are obvious. Unfortunately, many users still fail them because they are really robo-reviewing. Some people hit rates as fast as 2 seconds/post, believe it or not. (And no, they are not doing a good job at it...) Anyway, just please keep in mind that there are lots of people trying to game the system or earn various badges, and a script that helps people circumvent the audits will do more harm than good.
    – elixenide Mod
    Commented Feb 9, 2016 at 20:56
  • 8
    @EdCottrell If there is a problem with the click rate surely the answer is to detect the click rate and slow the user down to the point that he gets exasperated? I gave up reviewing here years ago because of bad audits. Whatever the motivation, the present system is certainly deterring people beyond the robo-reviewers. Is that a good thing?
    – user207421
    Commented Feb 10, 2016 at 1:45
  • 7
    @EdCottrell "Good reviewers don't have to worry about audits" -- I consider myself a "good" reviewer but have been banned for failing two audits (both of them bad) in 2 months. I'm ambivalent about abandoning the review process because I actually care about the quality of SO even while feeling overwhelmed by the abysmally low signal-to-noise ratio in the questions. However I'm reluctantly reaching the same conclusion as EJP, and will probably stop using the review queues until an audit dispute mechanism is in place. Commented Feb 10, 2016 at 2:53

5 Answers 5

139

I've talked about this before, but I strongly believe that as long as we have badges for reviewing, we must have audits to prevent abuse. We already have evidence for this, because the short period of time between when badges were added for review tasks and when audits were put in place was an absolute trainwreck. It took years to clean up all the spam that not only was approved but was upvoted in that short period. Completely removing the audits now would be a disaster.

I'll once again point out that I think there's a better way to address the few problematic audits, allow people to dispute individual audits. There are bad audit cases out there, and I think it's best to focus on identifying and removing those rather than throwing out the whole system.

But you wanted some numbers. At present, 144 users are banned from review on Stack Overflow. I went through by hand and reviewed the audit that led to each ban. Of those 144 bans:

  • 14 users were manually banned by moderators for approving spam, vandalism, or other abuses.
  • 28 users were banned for approving suggested edit audits. There is no way that those should ever be approved. They're complete gibberish.
  • 23 users were banned for approving obvious spam questions. No debate on those.
  • 22 users were banned for approving spam answers or obvious follow-on questions.

60% of the bans are ones that I don't think anyone could argue with at all. That leaves 57 audit-related bans that could at least be discussed. Of those, 17 of the audits came on questions and 40 on answers. Looking over the 40 answer audits that weren't spam or obvious non-answers, these failures came on approval of link-only answers, very poorly written ones, and others that community members have flagged as non-answers or very low quality and then were deleted.

The audits that you're referring to, and the ones that people complain the most about on Meta, are audits on questions. These come mostly from triage or the first posts queue. None of the current review bans come from failed audits in the close votes queue. 12% of the bans came from these debatable questions, so let's take a look at the specific audit failures (my apologies to the reviewers, but we need specific cases here):

In general, I see failures for votes on highly-voted questions that people thought were unsalvageable. I can see differences of opinion on what's acceptable with these. Personally, I think that was the wrong call on most of these questions, but that's my opinion.

Looking over the entirety of these bans, I sure am glad that almost all of these people aren't able to review after looking at the kinds of things they were approving. Without audits, 60% of these people would gladly approve spam or vandalism, and a large portion of the rest would approve low quality content coming in to the site.

For the few debatable audit cases, I fully support adding the ability to protest those and remove them from circulation.

25
  • Is interesting that suggested edits leads.
    – Braiam
    Commented Feb 10, 2016 at 0:11
  • 3
    Considering you're a stack exchange employee, can you give any perspective on when/if this "dispute audit" feature will be implemented?
    – Magisch
    Commented Feb 10, 2016 at 8:41
  • 7
    @Magisch - I'm merely a moderator, not an employee. I don't work for Stack Exchange, and I don't know any internal product development plans. I can only suggest things to them, as any community member would.
    – Brad Larson Mod
    Commented Feb 10, 2016 at 13:59
  • 7
    @BradLarson can you suggest them to hurry it up? ;)
    – Gimby
    Commented Feb 10, 2016 at 14:05
  • 10
    "These come mostly from triage " IMHO, the triage system is broken because the terrible wording of the triage options leads to far too many reviewers approving questions that should be closed, leading to disputed flags for the few people who understand what the close reasons are.
    – Raedwald
    Commented Feb 10, 2016 at 14:23
  • 6
    @Raedwald - It isn't just a lack of clarity in instructions. Of the 23 users banned for approving spam in questions, 12 of them came from triage. No matter how poorly the guidelines are explained, someone voting "Looks OK" on advertisements for Chinese rock crushing equipment clearly doesn't care. But yeah, poor reviewers or bad instructions, there have been problems with triage disputing correct flags: meta.stackoverflow.com/a/297163/19679 . I haven't seen quite as many complaints or bad cases since these changes went into affect: meta.stackoverflow.com/a/300787/19679
    – Brad Larson Mod
    Commented Feb 10, 2016 at 15:34
  • 6
    This is an interesting analysis, but I think only looking at bans skews the numbers. When I used to visit the review queues, my action on getting an audit was simply to stop reviewing. I was failed by several bad audits, but never got banned by the system: I just banned myself. It seems likely that the people who keep going until they get banned are far more likely to be roboreviewers than people who are successfully deterred by the doorslam UX of review audits.
    – Dan Hulme
    Commented Feb 10, 2016 at 15:42
  • There are 144 bans, but how many active reviewers are there? Unless someone can come up with a better measure of "active," how about number of reviewers who have completed at least 10 reviews in the past month? To @DanHulme's point, do we have any data about active reviewers before the badges were added? (Do we really want review badges if they result in behavior that bad?)
    – jpmc26
    Commented Feb 10, 2016 at 17:11
  • 1
    @DanHulme - To use the triage review queue as an example (the most controversial when it comes to audits), over the last 30 days, 4029 people have reviewed at least one post. Among those, 363 failed at least one audit, and 56 of that group did not pass any audits, were banned, or quit reviewing after an audit failure. 22 of the above review bans were from triage and would have occurred during this period.
    – Brad Larson Mod
    Commented Feb 10, 2016 at 18:05
  • 1
    @BradLarson - That seems like a high rate of failure. I am curious, what was the total reviews to audits failed ratio (across all reviews); and also, what value (perceived or measured) was gained from the review in that month?
    – Travis J
    Commented Feb 10, 2016 at 21:14
  • 1
    @Trilarion - All it takes is a single reviewer to approve or upvote a post in the Late Answers or First Posts review queue and it will never be reviewed again. There's no majority approval required there. A single terrible reviewer can allow dozens or even hundreds of spam posts, non-answers, or gibberish on the site if it wasn't for audits. Even for suggested edits, the terrible reviewers spamming reviews overwhelmed the larger number of good reviewers, and quite frankly the worst reviewers tend to be active at the same time, so the same block of reviewers can approve a series of posts.
    – Brad Larson Mod
    Commented Feb 11, 2016 at 15:32
  • 2
    Moderators do regularly analyze the advanced review statistics we get, to see if anyone stands out. By doing so, we can find those who are abusing review and gaming audits and manually apply bans and suspensions. However, I've not found there to be any consistent patterns in these beyond the obvious "approving 100% of the time" or "takes less than 3 seconds per review" metrics (which audits already take into account). It's hard to pick out someone who approves every third spam post they see by looking at pure review stats, even with a human analyzing patterns.
    – Brad Larson Mod
    Commented Feb 11, 2016 at 15:36
  • 2
    It's not that a small group of banned reviewers account for all reviews, it's that a small group of reviewers account for almost all bad reviews. When the audits first went into place, they only banned a few hundred people from review, which was much lower than I expected. What also surprised me was that approvals for spam, vandalism, non-answers, etc. stopped almost overnight. We found that this small handful of terrible reviewers accounted for almost all the damage that bad reviewers had done on the site. Someone clicking a button as quickly as possible can do a lot of harm.
    – Brad Larson Mod
    Commented Feb 11, 2016 at 15:57
  • 2
    @TravisJ - It would take me a long time to total up the number of reviews, but for the top 5 most-audited reviewers in triage in the last month, they performed a total of 1839 reviews, were exposed to 305 audits, and failed 2 of them. Eyeballing these, I'd say a rate of 1 audit failure per 150 reviews seems consistent across this queue. To compare, there were 3200 people who reviewed at least 5 suggested edits in the last 30 days. 264 of them failed at least one audit, and those audits should never be failed under any circumstances. That's a lot of inattentive reviewing.
    – Brad Larson Mod
    Commented Feb 11, 2016 at 16:11
  • 1
    @TravisJ - For First Posts and Late Answers, absolutely. They're our primary means of detecting incoming spam that makes it past all the system safeguards. Non-answers, too. The Close Votes queue is necessary, or the community moderation of SO would break down. Low Quality Posts has offloaded a lot of work from moderators onto the community, helping us to focus on more urgent matters. Suggested Edits is harder to gauge, but I think it's an overall positive for the site. Triage is the one I don't know much about, and is still an experiment from what I gather.
    – Brad Larson Mod
    Commented Feb 11, 2016 at 20:17
11

This is broken windows stuff - audits are an active and visible part of a culture that is all about quality. Beyond any objective metrics, they have a very real psychological role in cementing that image of quality.

This may sound foolish to some of the more rational and calculating among us, but for a lot of people I think it contributes in a significant way to the everyday mindfulness that regular users are bringing to the site. How much better is the custodianship outside of the review queue, for example, just because everyone has that sense of quality in mind? I don't think you can measure that so easily.

Having police stop the occasional drunk driver isn't just about catching the offenders or being right every time they pull someone over - it's about promoting a culture where most people don't even consider breaking the rules and, further, take some pride in that point. If you measure success by the number of drunk drivers caught you're sort of missing the point.

Sure, the audit system isn't perfect. Nobody can dispute that, but its function goes far beyond its ostensible role.

1
  • 3
    I'm glad you pointed this out. One of the principles of criminology is that the most effective way to reduce crime is to increase the likelihood that the criminal will be caught (read the excellent work of Cesare Beccaria, "On Crimes and Punishments"). You make the salient point that the audit system isn't necessarily meant to improve review quality, but to create a perception that abusive reviewers are likely to be caught. That's how you measure the audit system-- how likely is it to catch an abusive reviewer? If the likelihood is high, it is a successful system. Commented Feb 11, 2016 at 18:30
7

I'm not aware of any hard evidence although one could probably gather it (see below).

But first a reminder: Even without the existence of hard evidence it can still be that audits improve review quality because there is also no hard evidence of the contrary, as far as I know.

How would one get hard evidence?

  1. Divide the reviewer into two homogenous groups (similar rep distribution).
  2. Shows audits to one group, do not show audits to the other group.
  3. Measure the review quality of both groups (with a review review system where review reviewers vote on the accuracy of reviews, possibly with review review audits ;)).
  4. Compare both groups.

My personal experience: Most review audits tend to be obvious (extreme) cases. Their value is very limited. They are more like attentions tests, maybe effective against robo reviewers. Then there are some really dubious cases where it's actually not clear the audit is correct (see various meta questions about audits). And finally there is a certain amount of helpful audits - I do not pass all of them but sufficiently many to not be banned from reviewing. Mostly they remind me of taking more time to review and not deciding hastily and skipping more often. That's what I like them for. But they also take valueable time for real reviewing away - so is there a positive benefit in the end? My gut feeling tells me that some kind of feedback is valuable but it should not waste too much time. It's roughly okay as it is.

Alternatives? Maybe. One could tell reviewers how correlated their judgements are with the average judgement of all reviewers. Also show them the cases where they are deviating and maybe they learn something from it. Or install a review review system. The minimal waiting time between reviews is already an alternative counter measure. But these all also mean a lot of additional overhead and there may be totally valid reasons to produce deviating reviews in single cases - so it would have to be more like a statistical measure.

3
  • 3
    The investigation of some recent cases by Brad Larson is actually quite impressive. Commented Feb 10, 2016 at 10:42
  • 1
    You mentioned showing reviewers when they deviate from the review result; there's a feature request for that. It's only helpful for users who actually want to improve, so it doesn't really help the problem audits are trying to solve. Also, I'd be told my justified rejections disagree with robo-approves, which would be annoying (though at least I could repair the damage with a rollback). Commented Feb 10, 2016 at 22:36
  • @JeffreyBosboom Thanks. Nice. Unfortunately I didn't get it to work right now. Will debug later. Commented Feb 12, 2016 at 12:34
-7

Your question reads like a rhetorical question. Regardless it is an interesting question. I'm not aware of any statistics about the audit system and I doubt there is. -> apparently there is, see other answer.

How would that statistic gather data? Compare failures before and after audits and check if the number decreases? But like stated in the comments, some reviewers get more cautious and just pick 'easy' tasks. Which is maybe what the audit wants to achieve: Just process tasks you are absolutely sure about.

The statistic you want is already there:

Did the number of failed audits for you personally decrease over time? If no, then the audits did not achieve their goal. If yes, they obviously did.

9
  • 10
    There are stats all right. However these are only accessible to moderators and employees.
    – BoltClock Mod
    Commented Feb 9, 2016 at 8:58
  • 1
    The number of failed "good" audits has certainly decreased over time. However, that number is insignificant (<10) compared to the number of reviews I've completed since reviews were introduced. The number of failed bad audits is higher (10-20). I can't give accurate numbers since I'm currently banned and can't get to the review history tabs. Commented Feb 9, 2016 at 9:08
  • 1
    "Just process tasks you are absolutely sure about" I already do that. I skip LOTS of reviews in areas I don't know. But the bad audits are a regular fixture. Do you really want to discourage experienced users from participating in reviews? OK, I'll just "review" questions as I read them in the normal flow outside the review queues. I don't need to do reviews, I just thought it was a way the system made it easier to spread the moderation task among experienced users. Maybe I misunderstood the purpose of the review queues? Commented Feb 9, 2016 at 9:18
  • @BoltClock do these stats only include the number of failed audits or do they give insight about the quality of audits? How do you measure that?
    – lisa p.
    Commented Feb 9, 2016 at 9:25
  • @JimGarrison this already answers your question: The audits - in your case - failed their goal.
    – lisa p.
    Commented Feb 9, 2016 at 9:26
  • 3
    My question has nothing to do with one specific individual, and is not a "rant" as you seem to think. I see a deeply flawed mechanism that seems to choose audits based on upvotes alone. I believe the bad audits reduce the effectiveness of the review queues by discouraging active participation from experienced users. I want to know if the audit process is being measured to determine if it improves the overall effectiveness for the entire site. I don't believe it does. Commented Feb 9, 2016 at 9:33
  • 2
    My number of failed audits dropped drastically (currently 0 after the ban), but only because I now look for audits before voting on my best judgement. It has happened several times before to me that I examined a post and then failed an audit on it, and afterwards I still would have voted the same on that post.
    – Magisch
    Commented Feb 9, 2016 at 10:05
  • 1
    I just have the impression that you already think the system has to be changed, which is most likely necessary, and want data to back it up. It wouldn't help you if audits turned out to actually improve overall effectiveness except for a few individuals. I can remove the answer if it doesn't resolve your problem.
    – lisa p.
    Commented Feb 9, 2016 at 10:26
  • 2
    I agree that some form of audit is necessary given the robo-reviewers (i'd call them zombie reviewers myself). However, an appeal mechanism is needed. The current system is equivalent to letting a poorly implemented AI act as judge, jury and sentencer with no possibility of appeal. There's lots of science fiction horror stories about just such a scenario, and it mirrors the current ethical debate about AI in the sphere of autonomous weapons. I'm getting off topic but there's a common thread here. Commented Feb 9, 2016 at 22:17
-16

Just a simple suggestion from my side:

if userpoints > 2500
if user has not failed 25 audits
if user has at least done 250 reviews
excempt user from audits until bad behaviour.
then restart but with having to pass 50 audits, then 100 audits etc.. ;-)

Something like this, to reward the actual hard working reviewers who know the site well.

6
  • 7
    "excempt user from audits until bad behaviour." - define bad behavior.
    – Gimby
    Commented Feb 10, 2016 at 13:34
  • 1
    johnny be good does all good, then get some trolls in life, johnny be good is pissed off at the world and takes revenge by allowing crap on stack overflow by accepting spam answers. Good guy gone bad basically :-) Who knows who takes the swingline stapler away... Commented Feb 10, 2016 at 13:36
  • 2
    So basically you can't define it either then ;) "Bad behavior" cannot be measured, especially because on SO you vote on questions and answers, not people.
    – Gimby
    Commented Feb 10, 2016 at 13:39
  • 1
    yea, let's just classify under "user needed moderator intervention because he set the building on fire" as bad behaviour, that'd be great. Commented Feb 10, 2016 at 13:39
  • 9
    How do you tell if Johnny is accepting spam answers? The only way to do so is to test him with a known spam answer. That's basically what auditing is already doing.
    – Mage Xy
    Commented Feb 10, 2016 at 15:30
  • I probably qualify as one of those "working reviewers", with about 30,000 (that's thirty thousands) reviews. Thing is, what you suggest doesn't look like a reward to me. Audits help me stay focused and attentive, I don't feel any need to get rid of them
    – gnat
    Commented Feb 11, 2016 at 12:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .