38

I understand there is a legitimate purpose to making sure a user is paying attention when going through the review queues. However, in my opinion, a pre-requisite to doing this is to have some level of assurance that those audits are reasonable and predictable. This has been brought up before, but apparently is still not being done. Why waste review throughput on spurious audit questions?

Here is an example. This question appeared in the close audit queue. It was asked six days ago and had 11 up-votes (now it has some down votes).

Image of Question

At best, the question should have been edited for grammar and content by someone at some point. It's difficult to tell what the person is trying to get (what code will satisfy his requirements... indeed, what are his requirements?). At worst, it's off topic ("hey, tell me how to re-write my working code" is off-topic by definition). All of this has culminated in at least three answers, all of which have copious up-votes (after less than six days?) but none of which look similar nor have been accepted.

The Purpose of Audits

Audits have, ostensibly, one purpose - to prevent "robo-reviews" - meaning that they provide some degree of assurance that the user is actually voting in line with what others have thought. There are, in theory, a handful of algorithms which are selecting these questions from among those in the real world, and users are temporarily banned from reviewing as a result of failing too many audits (and that threshold is very low).

However, I think it's easy to argue that audits have a secondary purpose, and that is to train users to correct their decision making approach. As this discussion shows- an egregious example of a bad audit- the "audit failed" message says "There are no major problems with this question. You should click 'Leave Open' ..."

There is a term for this - when a student learns a skill, it's called "Transfer of Training." Transfer can be both positive and negative. Negative transfer of training occurs when a student is worse off for having learned the lesson.

So, if audits are going to serve as a learning purpose, in addition to an automated check, shouldn't there be some human involvement in cultivating good teaching examples?

To me, this question would not be a good candidate for an audit, and actually might be considered "negative transfer of training." I still think it should be closed, and I voted accordingly. What do you think?

19
  • 13
    Because we would need an audit review queue, to select posts to be used as audits. And it will probably be full with robo-reviewers and the audits will be even more crap but now at least you could complain on meta about users that selected the wrong audit instead of the current cold and impersonal system. Also note that your single down vote on that question pushed it out of the audit system. So posts are selected as audits based on manual selection. The down voters somehow left the building ...
    – rene
    Commented Nov 15, 2017 at 21:08
  • So you're saying that the answer is "it is too much work?" What assurance do we have that the current system does more good than harm? In this case, I almost wonder if there was some fraudulent voting (it sure seems odd for a question so badly formed to have so many up-votes). Wouldn't it be more appropriate to have a bank of, say, 100 questions and just recycle those?
    – theMayer
    Commented Nov 15, 2017 at 21:19
  • 1
    For fraudulent voting you can always raise a moderator flag and explain your concerns. In general the audit system works and stops the true robo-reviewers. There were a tricky audit causes an issue the audit is often resolved by community voting and a meta post will help to let moderators look into any review bans and if the ban is caused by a bad audit, mods often tend to rectify that. So I'm not convinced it is a huge problem, albeit annoying, and I'm almost 100% sure a full manual system wouldn't be better.
    – rene
    Commented Nov 15, 2017 at 21:35
  • 11
    I've suggested something similar in the past. All they need to do is let the algo spit out 100 potential audits, then ask for mod volunteers to review the list and pull out good candidates. Within a couple days you'd have 30-50 candidates that could go into rotation. Maybe do that once a quarter, depending on how many audits the average reviewer burns through a month.
    – user1228
    Commented Nov 15, 2017 at 21:50
  • 1
    We can't be sure where all the votes came from but I asked Shog9 and his check seems to confirm that the specific example has been on the Hot Network Question list for a while: chat.meta.stackexchange.com/transcript/message/6495254#6495254 so that could explain an higher then normal number of votes on that specific Q/A pair. And enough upvotes (IIRC 6 or more) and with no down votes, the post becomes eligible for being an audit.
    – rene
    Commented Nov 15, 2017 at 22:01
  • @Will was that suggestion a post you wrote or did you rant in a comment? if the former, I can't find it, not on MSO, nor on MSE
    – rene
    Commented Nov 15, 2017 at 22:16
  • 9
    cross-site duplicate: Bring a “human factor” into review audit composition/selection. Also related: Review audits and “I understand” button
    – gnat
    Commented Nov 16, 2017 at 6:50
  • 2
    @rene I wouldn't call my comments rants... but yeah, I never created a FR for it. I think gnat's dupe above (still) is pretty good and roughly what I would like to see enacted (still, because it showed I already upvoted it).
    – user1228
    Commented Nov 16, 2017 at 14:16
  • @Will fair enough, and I brought my humor as I assumed you would bring yours. So it's "rants" ... ;)
    – rene
    Commented Nov 16, 2017 at 14:26
  • 2
    To be fair, I get audit questions once in a blue moon. But about 3/4 of the audit questions I get are themselves questionable. So, I do think the magnitude of the problem is perhaps not fully understood - there is no systematic way to collect data, except for people posting here for discussion. I mean, if the audit questions were always good, I wouldn't fail them. I don't robo-review. I do a few at a time, when I have time, and I don't review if I don't have the time to do it properly.
    – theMayer
    Commented Nov 16, 2017 at 16:17
  • 1
    FYI as of 18min ago that particular question is now on hold.
    – robinCTS
    Commented Nov 17, 2017 at 7:48
  • 4
    IMO the audits hardly ever teach, they're basically just there to make sure you're paying attention. Their secondary purpose is to provide enough annoyance that people are triggered to come to meta where they have a chance of learning something. Results are... mixed.
    – Gimby
    Commented Nov 17, 2017 at 8:27
  • 1
    The audits are broken, everyone knows it and there's been countless of meta discussions for years. Still SO won't do anything to fix them. However, the root of the problem in this specific case might be that too many users up-vote anything they see?
    – Lundin
    Commented Nov 17, 2017 at 8:33
  • 2
    Why don't we use information given by "Skip". As I understand it "Skip" either mean "Not proficient enough in the subject matter to Audit" or "What should be done with this question is unclear to me". Good Audit questions should should not have too high a skip ratio for the second kind of skip (of course there is the issue os distinguishing between the first ands second kind of skip, maybe relying on tag badges and question tags ?).
    – kriss
    Commented Nov 17, 2017 at 10:10
  • 1
    @NathanTuggy which I upvoted :)
    – user1228
    Commented Mar 29, 2019 at 13:11

0

You must log in to answer this question.

Browse other questions tagged .