26

I'm not sure I understand this particular audit review. I failed an audit by flagging it as primarily opinion based. Before selecting an audit action, I opened the question in a new tab to see that it was "put on hold" as "primarily opinion based". I read through the question and agreed with the other users' flag of the question denoting that it was primarily opinion based. Link to Review

The question was asking for a coding preference, where there wasn't an incorrect answer. Since it was flagged within the last day and the question was from mid-July, it seems that the review audit rotation still had it as a valid question despite being flagged. According to the review audit, the correct answer was to mark it as "looks good."

Maybe the review audit rotation doesn't update quick enough to prevent something like this from occurring? Was the question still a good question even though it had been flagged as put on hold? Did I do the right thing by flagging the question during the review audit (if so, why did I fail?)?

I've seen that there are tons of questions on Meta regarding failed audits. I have searched and don't think this is a duplicate, but here are some related questions:

Regarding Duplicate

As for the last question (I do not understand why I failed), this is a different case. The post's largest upvoted answer says:

After all, the audit system won't use a closed question as a "good post" audit.

I'm saying that this exact thing happened to me, which is contrary to expected behavior. In essence, I flagged a question that was already flagged by other users and still failed the audit. It's been confirmed by the moderators that I did the right thing (in the comments below). Thus, it seems this is a unexpected behavior (eg: a bug) due to the triage queue using a cache of questions for auditing that may not always be updated, which is different from the other post.

15
  • 1
    Wait the question is on hold, you flagged it unsalvagable, and failed?! Commented Aug 12, 2019 at 14:20
  • Yes. Is that on me? Commented Aug 12, 2019 at 14:21
  • 4
    Probably the 9 upvotes and 0 downvote messed up the audit system...
    – Andrew T.
    Commented Aug 12, 2019 at 14:23
  • 1
    Does the review system take closure alone into account?
    – Suraj Rao
    Commented Aug 12, 2019 at 14:23
  • 5
    @technogeek1995 I don't think so, unsalvageable was absolutely right IMO Commented Aug 12, 2019 at 14:23
  • 15
    I unbanned you - I don't have time to investigate the timeframe of the audit vs the closure - your review response was ok.
    – user3956566
    Commented Aug 12, 2019 at 14:26
  • Should I delete this meta question or keep it alive? Commented Aug 12, 2019 at 14:27
  • 11
    @technogeek1995 Keep it alive, bugs need fixed or at least the situation explained Commented Aug 12, 2019 at 14:27
  • up to you - may as well delete it
    – user3956566
    Commented Aug 12, 2019 at 14:28
  • 2
    @YvetteColomb from the timeline it was closed ~4hours prior to audit
    – Suraj Rao
    Commented Aug 12, 2019 at 14:28
  • 1
    @SurajRao probably caching then
    – user3956566
    Commented Aug 12, 2019 at 14:28
  • 2
    At least in the close vote queue, if an audit is invalidated (for example from a close vote on the post), then the review immediately becomes unreviewable, I would expect that to mean it wasn't caching (at least not in the same way), maybe triage works differently though Commented Aug 12, 2019 at 14:32
  • 7
    This one has been used as an audit quite a few times. All those users either passing or failing the audit may deserve a second look...
    – yivi
    Commented Aug 12, 2019 at 14:33
  • I do not think, that this is a very clear case for either side; this question will always be a bad audit candidate. It is not even very clear "primarily opinion based" either; Remember: The user is asking if he may use "finally" in this case or if it is a clear abuse of this mechanism.
    – Ctx
    Commented Aug 12, 2019 at 14:46
  • Possible duplicate of I do not understand why I failed this audit
    – gnat
    Commented Aug 12, 2019 at 14:49

1 Answer 1

4

As you and Yvette both suggest, the problem here is caching.

The posts used to generate audits for the Triage, Close and Reopen queues are expensive to query. To avoid huge delays (and excessive db load) when generating an audit, the system queries all eligible posts and then caches the results, pulling IDs from the cache on future requests until the cached list is exhausted; only then are results refreshed.

Normally, this works pretty well - the criteria are pretty strict for known-good audits, particularly in Triage: to make the list, a question will have been already interacted with by multiple people who could have voted to close it but did not. It also needs to have a reasonable score, can't ever have been downvoted (even if the vote was later retracted) and have been on the site for between 10 and 30 days without being migrated in or bountied.

To give you an idea of just how strict those criteria are: as I type this, there are precisely 204 questions on Stack Overflow that qualify for use as known-good audits in Triage. Even if I were to look back an additional two months, the total would remain under 1000. We put a lot of work into these criteria and they're generally pretty solid...

However... Nothing actually prevents a question from being closed between the time when its ID is cached and when it is used. And, sooner or later that happens. It happened to you, and it's happened 13 other times in the history of Triage review (out of a total of 213,028 "known-good" audits).

So I'm leaving this marked as a bug, because it is. An extremely rare one, but a bug none the less.

2
  • 3
    When you are retrieving the ID from the cache, are you not checking its state before delivering the result to the end user? That pre-check does not require a lot resources.
    – KarelG
    Commented Sep 4, 2019 at 6:36
  • could you recommend course of action for those who fail such "invalid cache" audits? Do they escalate to meta, or maybe flag for moderators, "I failed audit because of known bug, see meta.stackoverflow.com/a/389106"? Or maybe something else?
    – gnat
    Commented Sep 4, 2019 at 12:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .