Summary
I would like to suggest replacing the current system of algorithmically-selected one-off† audits with a process of feeding audit candidates from the current algorithm through a queue to select the best of them. These would be highly reusable and culled of any dubious or wrong results, like posts that were incorrectly deleted, Haskell confusion, etc.
Immediate benefits
- Good reviewers reviewing at least a bit more total, in part due to the relaxation of the chilling effect. All (existing) queues would benefit, but especially CV.
- Less angst from reviewers that don't want to risk the ego hit/waste of time from bad audits; less cognitive dissonance with mods admitting they fail 20% of audits(!!!).
- Considerably fewer meta posts from normally-competent reviewers getting banned or wanting an audit disabled; slightly fewer meta posts from rightly-banned reviewers complaining; less moderator time spent on debating the validity of X specific audit.
- Audits would no longer be shown twice to the same reviewer, as occasionally happens now. While we're at it, perhaps we could also prevent audits from showing to users that originally helped generate that audit candidate through flags and votes.
Scope for further improvements in training reviewers
With a stable review audit queue, several more possibilities present themselves for improving reviewer training. This would reduce mistakes, and in turn increase efficiency. It would also likely, again, increase buy-in and therefore spread the load better.
- Adding more detailed rationale to individual audits by customizing the approval button to open a menu of defined reasons. And, therefore, training potentially-good reviewers to actually be good, since the majority reason can then be automatically shown every time the audit is failed (or passed, for that matter). This kind of just-in-time training has a lot of potential, most of which we are currently wasting, and some of which the present audit system is actually using to mis-train reviewers by accident.
- More flexibility in including subtypes of audits that are presently hard to realistically include in the algorithm without tons of false positives/false negatives (especially Suggested Edits, which has never had any audits that require approval to pass). This would in turn reduce mistakes in those types of reviews.
- An opportunity to relax the candidate-generation algorithms to allow more subtleties in; assuming audit reviewers pick good solid audits that are still unambiguous, this will also reduce bad reviews and teach better judgment. Some posts are straightforward enough to evaluate, but had some unusual circumstance that influenced the votes enough to be ineligible for the current algorithms — for example, extensive and effective editing after receiving multiple downvotes, which left the question in a good state but did not prompt enough voters to reverse their votes.
Basic quantitative analysis of practicality
If each of these audits is used for auditing as many as 100 different users before being retired, and SO needs up to about 15% of ~15000 (let's say 2250) reviews as audits every day, then about 23 newly-validated audit cases per day (across all review types) would maintain the flow easily. There's 26 ♦ mods and almost 4000 30k users. Assuming each audit case must be approved by 1 ♦ mod or 5 30kers, keeping up with this flow requires only about 1 approval per ♦ mod per day, or 5% of 30kers doing one each, or 0.5% of 30kers doing 10 each. (The other 10 per day could be rejections or simply not hitting their cap.)
Interestingly, in the years since this was originally asked, review throughput has gone down, while ♦ mods and especially 30kers are more numerous, so the load per audit approver could be now less than half the already modest figures.
Some implementation details
The queue would be populated by a scheduled task every minute or so ensuring that it has at least twice‡ as many candidates as the number of audit cases lacking from the desired pool setting. For example, if there's a pool target of 10 active audit cases for close-vote review, and there's currently only 5 in the pool, the queue would be populated with 10 candidates; if four were reviewed and only one was suitable for audit, the next time the task ran it would bring 6 candidates up to 8 by adding two. The timeframe would need to be set to allow enough time for audit approvers to see the queue entries and go through them before the pool goes from almost full to empty. A day's worth (as determined from recent trends) seems like a good start.
In other respects it should be a fairly standard queue, I guess, with three buttons: Valid, Questionable, Skip. (The second one is a vote to kill the audit. Or maybe just kills it immediately without any further reviewing.)
Once used to test a given user, an audit case would (presumably) need to be marked and not used against them again. But since audits would only go through their 100-user lifetime once in general, this marking would expire pretty quickly and the table of known-used mappings would stay a manageable size.
Costs
- An unknown (to me) amount of dev time:
- setting up a new queue type
- rewiring the audits to pipe through it
- setting up worst-case "we're totally out of audits!" fallbacks (probably just use of the current system)
- tuning the pool size to ensure it doesn't run dry, and spinning up the initial transition
- (presumably) ensuring that audits are not reused for a given reviewer
- Some minutes a day per mod and some fraction of the 30k population spending a similar amount of time. The problem of scaling appears to be solved.
Negative side effects
- Some time spent on user reeducation after the switchover to clarify that audits are now (almost) always hand-validated for the express purpose, and failures are therefore no longer likely to be excusable.
Nothing else has so far been brought up.
Notes
I seem to remember suggesting a shortened form of this before, but I can't find the post in which I did, either here or on M.SE. Maybe it was in comments. Probably just as well, since I thought I recalled it being rejected as taking too much effort for the mods, which can't be right. (Although perhaps this objection was raised to a more primitive form of this proposal that did not allow large-scale reuse of all audits.)
An alternative to opening to all 30kers would be to allow gold-badgers to vote for audits, and only on their badged tags. Analyzing the exact numbers required for this would be somewhat non-trivial, so I left that off for now. Similarly, requiring Steward to vote on audits would have some subtle or not-so-subtle implications that would be rather tedious to calculate at present; if that seems like a fruitful line of discussion, I could try it. The original version of this post assumed a 20k privilege, but I've switched to 30k to be conservative and removed the previous 20k analysis (and a lot of the permutations) for simplicity.
As another check, reviewers in this queue could be limited to 10 audit (in)validations per day instead of the usual 20 queue entries. This should not hinder normal operations much; it would simply require somewhat more reviewers to hit their cap, but there should be more than enough.
I'm not sure if there are any other SE sites that really need this, at least at present, but if it works well here, others might adopt it. It shouldn't hurt, in general; almost all the cost is fixed (in the economic sense), for the coding. Quite a few sites have insufficiently robust voting to make use of the current algorithms reliably, and would likely only be able to enable audits at all with something similar to this scheme, which allows loosening criteria substantially without reducing audit quality.
Because audits would come (initially, at least) from exactly the present algorithms, there could be no real drop in audit quality unless more good audits were rejected than bad ones. If desired, audit voting could operate on a veto/full-agreement basis, where one vote is sufficient to drop the audit from consideration; the volume of audits needed is not at all likely to be high enough to cause shortages in the automated selection supply this way, and as long as the reviewers are reasonably careful even this polarizing effect would likely increase average quality somewhat and worst-case quality greatly.
Some review queues share some or all of their audits (FP/LA; Close/Reopen). This very slightly reduces the number of audits that need to be generated in total, but doesn't change the number per day significantly.
I'm tempted to tag this as a burninate-request, because guess what we can do with disputed-review-audits?
†They're not really one-off, strictly speaking, but AFAIK they're not deliberately reused; the algorithm appears to simply select a fresh candidate each time one is needed, which may or may not have ever been used for auditing before.
‡Yes, that's another parameter that could easily be changed for efficiency, although because it's a feedback loop it doesn't really need much tweaking. If the pool routinely runs dry, though, increasing the multiplier and increasing the pool size would both help.