8

I would like to re-open the conversation around a formal way to dispute or flag ambiguous audits.

TBH, I haven't been banned before, but I have seen my fair share of questionable audit questions and we have all seen the meta posts that those of us frustrated enough to do so will leave on meta when we don't understand or agree with a failed audit.

If there was a mechanism to flag an audit review then perhaps we can act as a community to clean up or improve the currently automated audit review system. It would be great if the result of a review was to remove the audit question, then it would remove any record of users who completed that audit, but ultimately I would be happy with a first step of ensuring that the question was removed from the queues altogether.

If the purpose of Audits is to enforce rules, conventions, and/or standards, then the reviews as a bare minimum should not be ambiguous, ideally, they should be clear in the intent and if possible should include specific guidance on the correct resolution and the correct reasoning. If we can identify ambiguous or misleading audits then we can all help keep this community a better place.

Formalizing the process for audit review flags does add another queue system, but it simplifies the manual moderation effort that we hope goes on when we post in meta, it should reduce the number of frustrated posts like these or any in the disputed-review-audits and hopefully result in better education across the board:

and, hopefully, we can avoid these types of discussions: Suspended from reviewing due to questionable audits

On a personal note, failing an audit that looks wrong really de-motivates you to continue. I want to help with the queues, but, clearly, the queues don't want my help. I understand this is an automated and imperfect system, so provide a mechanism where we can help.

7
  • 2
    Nicely done, thank you - it does make sense to separate the concerns here so as not to accidentally tank the feature request. Although since it involves dev time, it is highly unlikely that anything like this will be actioned upon in a reasonable timeframe even with community support, so I would not get your hopes up too much. Jan 4, 2022 at 3:48
  • 4
    I think that too many users failing an audit will click the "challenge audit" to make it useful. In my experience, it often happens that a user will think "the audit is wrong, not me", and only in fraction of the cases that's actually right.
    – yivi
    Jan 4, 2022 at 8:06
  • Though I can understand that failing an audit that you think is incorrect should be flagged, I'm not sure it is the right solution. I can, like yivi suggests, see it being misused. Though not ideal, I think that if someone truly feels that an audit isn't correct then posting about here here, on [meta.so], with a link to the review and a detailed explanation of why it's not correct is the best place. The community can discuss it then, and mods/staff can act on said audit if the community feels the OP is correct.
    – Thom A
    Jan 4, 2022 at 10:58
  • @yivi I think the solution to your objection is simply to build a system that's resilient to such abuse – in this case, if a flagged audit comes back from "audit review" as valid, then the author should likely get some associated punishment as a result. The fact that most audits are fine also doesn't diminish the problem at hand in my opinion; as mentioned in the post above, a truly bad audit failing an otherwise good reviewer is a really detrimental thing that simply should not be happening. We need a better system in place here.
    – zcoop98
    Jan 4, 2022 at 15:49
  • 3
    The process could be substantially improved. Having some way for users to indicate there's a bad audit would be good. However, it feels like this proposal assumes there's an existing method of manually removing posts from being audits when that, currently, isn't directly possible. Currently, posts are removed from being audits by applying some change to the post (usually some type of vote: up/down/close/delete/etc.), so it no longer fits the automatic audit selection criteria. The main advantage moderators have in doing that is we can undelete and unlock a post and re-flag spam as R/A.
    – Makyen Mod
    Jan 4, 2022 at 15:58
  • 1
    I happened to request something quite similar here for essentially the same reasons - it's frustrating when an audit is incorrect and it's frustrating that there's nothing that can be done to prevent others from having the same experience. Thinking about it more though, I wonder why audits even need to be automatically generated in the first place? Surely it wouldn't be that hard to have SO staff create a list of 100 or so good audits for each queue. Yes, cheating might be a bit more common, but I'm not sure if that would really matter.
    – JKillian
    Apr 24, 2022 at 3:41
  • Great point @JKillian what is it exactly that the SO staff actually do around here anyway? It's one of those situations like unit testing by re-creating the same logic path, the audits should be testing us using known axioms, even if they are selected from actual questions. But as soon as you automate that then the integrity of the entire system comes into question. Apr 25, 2022 at 20:48

0

You must log in to answer this question.

Browse other questions tagged .