-5

We all know that the auditing system is flawed. Several users have come across faulty audits. Multiple times resulting in bans. I recently came across an audit in the Close Votes. As you can see here there is no well formulated question here:

NOTE: (The example below, may or may not be a valid question, but it should not be used as an audit, because even if you are paying attention it's quite ambigous on what the answer would be)

enter image description here

Yet, if you navigate the the Question, you can see that despite this question being of such poor quality, it has several upvotes and 0 votes to close:

enter image description here

This happens multiple times a day and most people don't even care enough to report it. Seeing as this is in the Close votes queue, it's quite ambiguous what happen's when you vote to close an audit in close votes

I think it's time we attempt to solve this with another queue: audits review queue. This will be tricky to implement but I think I've worked out most of the spec.

  1. Audits which people have failed will now have a dispute button.
  2. Audits review queue should only be open to people who have gotten a gold badge for reviewing, and have access to all review queues.
  3. Reviewers review system picked Flagged and system picked audits.
  4. Anyone banned by an audit which failed review is automatically lifted.

Auditing the audit queue, Will be an issue so we will need to revise some spec's for this. Audits for the audit queue will work in two ways.

  1. Some audits for the audit queue will be hand written by moderators.
  2. When enough user's review an audit, an audit can be added to the audit's audit.
  3. Once the queue starts working properly there will be no need for hand written reviews but, but moderators should still have the ability to submit them.

This will help remove bad audits from the queue, and take a small load off of moderation once the queue starts working properly.

P.S Can a moderator remove that posted audit from close votes audit?

22
  • 4
    And then we need to implement a system of audits in the audit queue, and a way to flag such audits, and... I haven't voted on this question but it does seem like kind of work for work's sake. If you fail an occasional audit that's hardly fatal, if you fail lots you're probably doing something wrong. Commented May 4, 2018 at 15:06
  • 6
    Please don't assume we haven't read the full question before downvoting. Some of us do read fast, and your post really isn't that long.
    – Kendra
    Commented May 4, 2018 at 15:08
  • 1
    I know you're suggesting that a feature be introduced, but I'm not sure I mesh with your rationale. What makes that question bad? Do you know a lot about Facebook plugins and are able to objectively say that this question is poor in that context? When would you reach for the "Skip" button as opposed to taking a guess at this one's quality?
    – Makoto
    Commented May 4, 2018 at 15:10
  • @Makoto, That's just an example, but this is a poor question because there is no question there just a statement
    – johnny 5
    Commented May 4, 2018 at 15:14
  • 3
    A question does not need to contain a grammatical question in order to be answerable. It's entirely sufficient for a question to merely have a problem statement where the question is implied to be how to solve that problem. I don't know enough about the topic to know if more information is needed to solve the problem, but what it's asking is clear enough. Forcing people to add, "how do I fix this?" to the end of their question isn't actually making the questions any better.
    – Servy
    Commented May 4, 2018 at 15:14
  • 2
    @johnny5: I emphatically disagree. This seems to be a perfectly reasonable question in the context of developing Facebook plugins. It also seems to be the case that several others are also encountering this issue. If you don't know if the question is good or not, just skip the review. There's no value in taking a stab in the dark since you're not really helping if you just guess at quality.
    – Makoto
    Commented May 4, 2018 at 15:15
  • @RobertLongson, Yeah you will need to implement it but, thats why I marked it as a feature request. I've seen alot of questionable audit, and this will help create a stronger audit system instead of banning people ambigously
    – johnny 5
    Commented May 4, 2018 at 15:16
  • @Kendra, instead of downvoting or commenting on symantics, it would be nice if you provide an argument against my proposal instead of just saying you read this
    – johnny 5
    Commented May 4, 2018 at 15:18
  • @Servy, I Agree this may not be the best example of a poor audit. But shouldn't audit's be Clear in their intent, and not left open ambiously, this queue could help resolve that problem
    – johnny 5
    Commented May 4, 2018 at 15:18
  • 3
    @johnny5 I'm not sure you've comprehended the recursive rabbit hole nature my previous comment. Commented May 4, 2018 at 15:19
  • @johnny5: All audits can do is see if you're paying attention. That's their intent. If you get caught not paying attention, then the audit has done its job.
    – Makoto
    Commented May 4, 2018 at 15:22
  • @RobertLongson, My argument is that works is expected because to be done because I'm making a feature request, the spec's of my post point out that audit's which pass success enough times will be uses recursively to audit itself. I've never seen the source of stackoverflow, but that seems like a simple thing to do considering the audit system is already in place we would just need to replace where it got that feed from.
    – johnny 5
    Commented May 4, 2018 at 15:22
  • @Makoto, Yeah, my point though is that the audit system is flawed, and even if you're paying attention you can still Fail and audit. This would take out some of the ambiguity of audit's
    – johnny 5
    Commented May 4, 2018 at 15:24
  • 1
    ...It'd be a bit simpler if you admit that you weren't clear about the subject domain of the question, and you couldn't objectively determine if the question was or wasn't good. All this talk about "good" and "bad" audits is superfluous, since audits are really only meant to determine if you're paying attention. But, I think I'll leave it at that.
    – Makoto
    Commented May 4, 2018 at 15:26
  • 1
    That question is highly representative of the meat and potatoes of Q+A at SO. Programmer runs into a wonky error message, no idea how to fix it. It got so many upvotes because other programmers run into the same problem. The only thing wrong with it is that nobody has discovered a solution for it yet, that does require somebody finding the question first. Deleting it obviously won't help that. Sorry to break the news, but audits are designed to weed out reviewers that get this wrong and can't learn to get it right from failing them. Feature, not a bug. Commented May 4, 2018 at 15:53

2 Answers 2

4

We all know that there are occasional bad audits that slip into the system. Voting rings, bad flags, nonobvious spam cases, etc. all can trip up reviewers by injecting improper posts as audits. In my experience, the vast majority of audits are clear and obvious, but bad cases do end up being used occasionally.

For years, I've been banging the drum for a means of disputing individual audits. The system I describe there is simpler, with a single button for someone to dispute an audit they failed. That audit would then go into a moderator-only queue of disputed audits. Moderators would accept or decline these audit removal requests. If a moderator agreed that an audit case should be removed, it would be taken out of circulation and the audit failure stricken from the user's record. If the moderator disagreed, the audit failure would stand. Anyone found to be abusing this system would be subject to lengthy review bans.

I don't believe the volume of disputed audits would be such that it would overwhelm existing moderators, from judging how relatively infrequent these complaints are and from a glance through recent audit failures. As a result, I don't think the complexity of exposing this to non-moderator users is warranted.

At present, moderators have limited means of removing a normal audit from circulation, usually by clearing mistaken spam flags. With an moderator-only audit review list like this, we could have the ability to mark more general cases as bad audits and avoid a chunk of the irritation people have experienced in review.

4

Audits are not meant to test if you can press the correct button, they're meant to test if you are paying attention.

In your particular example there text at the top of the post

Our system has identified this post as possible spam; please review carefully

Well, this post clearly is not spam - something is wrong. Go to the actual page, check things out. Take a second to assess the situation, then pass your judgement.

That's the point of audits.

4
  • 4
    The post being not spam doesn't mean no action is merited. This is the close vote queue. Determine if the question needs to be closed, and choose an action accordingly. A post can be not spam and still merit closure.
    – Servy
    Commented May 4, 2018 at 15:16
  • @gunr2171, as Servy points out that is not the only reason you can vote to close the audit. That audit is ambigious and even if you are paying attention you can fail.
    – johnny 5
    Commented May 4, 2018 at 15:25
  • Also you shouldn't voting to close spam anyway. That is what makes this ambiguous, so the solution is to remove the "possible SPAM" header from the CV queue Commented May 4, 2018 at 15:47
  • I didn't say anything about what button you should press on the review.
    – gunr2171
    Commented May 4, 2018 at 16:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .