34

Summary

I would like to suggest replacing the current system of algorithmically-selected one-off† audits with a process of feeding audit candidates from the current algorithm through a queue to select the best of them. These would be highly reusable and culled of any dubious or wrong results, like posts that were incorrectly deleted, Haskell confusion, etc.

Immediate benefits

  • Good reviewers reviewing at least a bit more total, in part due to the relaxation of the chilling effect. All (existing) queues would benefit, but especially CV.
  • Less angst from reviewers that don't want to risk the ego hit/waste of time from bad audits; less cognitive dissonance with mods admitting they fail 20% of audits(!!!).
  • Considerably fewer meta posts from normally-competent reviewers getting banned or wanting an audit disabled; slightly fewer meta posts from rightly-banned reviewers complaining; less moderator time spent on debating the validity of X specific audit.
  • Audits would no longer be shown twice to the same reviewer, as occasionally happens now. While we're at it, perhaps we could also prevent audits from showing to users that originally helped generate that audit candidate through flags and votes.

Scope for further improvements in training reviewers

With a stable review audit queue, several more possibilities present themselves for improving reviewer training. This would reduce mistakes, and in turn increase efficiency. It would also likely, again, increase buy-in and therefore spread the load better.

  • Adding more detailed rationale to individual audits by customizing the approval button to open a menu of defined reasons. And, therefore, training potentially-good reviewers to actually be good, since the majority reason can then be automatically shown every time the audit is failed (or passed, for that matter). This kind of just-in-time training has a lot of potential, most of which we are currently wasting, and some of which the present audit system is actually using to mis-train reviewers by accident.
  • More flexibility in including subtypes of audits that are presently hard to realistically include in the algorithm without tons of false positives/false negatives (especially Suggested Edits, which has never had any audits that require approval to pass). This would in turn reduce mistakes in those types of reviews.
  • An opportunity to relax the candidate-generation algorithms to allow more subtleties in; assuming audit reviewers pick good solid audits that are still unambiguous, this will also reduce bad reviews and teach better judgment. Some posts are straightforward enough to evaluate, but had some unusual circumstance that influenced the votes enough to be ineligible for the current algorithms — for example, extensive and effective editing after receiving multiple downvotes, which left the question in a good state but did not prompt enough voters to reverse their votes.

Basic quantitative analysis of practicality

If each of these audits is used for auditing as many as 100 different users before being retired, and SO needs up to about 15% of ~15000 (let's say 2250) reviews as audits every day, then about 23 newly-validated audit cases per day (across all review types) would maintain the flow easily. There's 26 ♦ mods and almost 4000 30k users. Assuming each audit case must be approved by 1 ♦ mod or 5 30kers, keeping up with this flow requires only about 1 approval per ♦ mod per day, or 5% of 30kers doing one each, or 0.5% of 30kers doing 10 each. (The other 10 per day could be rejections or simply not hitting their cap.)

Interestingly, in the years since this was originally asked, review throughput has gone down, while ♦ mods and especially 30kers are more numerous, so the load per audit approver could be now less than half the already modest figures.

Some implementation details

The queue would be populated by a scheduled task every minute or so ensuring that it has at least twice‡ as many candidates as the number of audit cases lacking from the desired pool setting. For example, if there's a pool target of 10 active audit cases for close-vote review, and there's currently only 5 in the pool, the queue would be populated with 10 candidates; if four were reviewed and only one was suitable for audit, the next time the task ran it would bring 6 candidates up to 8 by adding two. The timeframe would need to be set to allow enough time for audit approvers to see the queue entries and go through them before the pool goes from almost full to empty. A day's worth (as determined from recent trends) seems like a good start.

In other respects it should be a fairly standard queue, I guess, with three buttons: Valid, Questionable, Skip. (The second one is a vote to kill the audit. Or maybe just kills it immediately without any further reviewing.)

Once used to test a given user, an audit case would (presumably) need to be marked and not used against them again. But since audits would only go through their 100-user lifetime once in general, this marking would expire pretty quickly and the table of known-used mappings would stay a manageable size.

Costs

  • An unknown (to me) amount of dev time:
    • setting up a new queue type
    • rewiring the audits to pipe through it
    • setting up worst-case "we're totally out of audits!" fallbacks (probably just use of the current system)
    • tuning the pool size to ensure it doesn't run dry, and spinning up the initial transition
    • (presumably) ensuring that audits are not reused for a given reviewer
  • Some minutes a day per mod and some fraction of the 30k population spending a similar amount of time. The problem of scaling appears to be solved.

Negative side effects

  • Some time spent on user reeducation after the switchover to clarify that audits are now (almost) always hand-validated for the express purpose, and failures are therefore no longer likely to be excusable.

Nothing else has so far been brought up.

Notes

I seem to remember suggesting a shortened form of this before, but I can't find the post in which I did, either here or on M.SE. Maybe it was in comments. Probably just as well, since I thought I recalled it being rejected as taking too much effort for the mods, which can't be right. (Although perhaps this objection was raised to a more primitive form of this proposal that did not allow large-scale reuse of all audits.)

An alternative to opening to all 30kers would be to allow gold-badgers to vote for audits, and only on their badged tags. Analyzing the exact numbers required for this would be somewhat non-trivial, so I left that off for now. Similarly, requiring Steward to vote on audits would have some subtle or not-so-subtle implications that would be rather tedious to calculate at present; if that seems like a fruitful line of discussion, I could try it. The original version of this post assumed a 20k privilege, but I've switched to 30k to be conservative and removed the previous 20k analysis (and a lot of the permutations) for simplicity.

As another check, reviewers in this queue could be limited to 10 audit (in)validations per day instead of the usual 20 queue entries. This should not hinder normal operations much; it would simply require somewhat more reviewers to hit their cap, but there should be more than enough.

I'm not sure if there are any other SE sites that really need this, at least at present, but if it works well here, others might adopt it. It shouldn't hurt, in general; almost all the cost is fixed (in the economic sense), for the coding. Quite a few sites have insufficiently robust voting to make use of the current algorithms reliably, and would likely only be able to enable audits at all with something similar to this scheme, which allows loosening criteria substantially without reducing audit quality.

Because audits would come (initially, at least) from exactly the present algorithms, there could be no real drop in audit quality unless more good audits were rejected than bad ones. If desired, audit voting could operate on a veto/full-agreement basis, where one vote is sufficient to drop the audit from consideration; the volume of audits needed is not at all likely to be high enough to cause shortages in the automated selection supply this way, and as long as the reviewers are reasonably careful even this polarizing effect would likely increase average quality somewhat and worst-case quality greatly.

Some review queues share some or all of their audits (FP/LA; Close/Reopen). This very slightly reduces the number of audits that need to be generated in total, but doesn't change the number per day significantly.

I'm tempted to tag this as a , because guess what we can do with ?

†They're not really one-off, strictly speaking, but AFAIK they're not deliberately reused; the algorithm appears to simply select a fresh candidate each time one is needed, which may or may not have ever been used for auditing before.
‡Yes, that's another parameter that could easily be changed for efficiency, although because it's a feedback loop it doesn't really need much tweaking. If the pool routinely runs dry, though, increasing the multiplier and increasing the pool size would both help.

13
  • 30
    Yo dawg I heard you like review queues so I created a review queue for audits for review queues.
    – PeeHaa
    Commented Jan 24, 2015 at 9:31
  • 8
    @PeeHaa Something like that, yes. The meta is strong with this one. Commented Jan 24, 2015 at 9:35
  • 3
    Let's implement this if those 20K-ers cleared the Close vote queue or at least stopped the dialy incline...
    – rene
    Commented Jan 24, 2015 at 10:33
  • @rene One more review/day average (at most) for each reviewer is not going to make much difference to the workload. Also, the one queue that would most show the effects of increased good/decent reviewer count would, naturally, be Close Votes. Even if 5 20kers stop CVing entirely but 15 3kers start, that's arguably a net win. Commented Jan 24, 2015 at 10:36
  • 2
    I like your optimism about the negative side-effects but given the fact that even high rep users come to meta to complain about wrong audits I doubt that behaviour will change. In the current system it is easy to fix a wrong audit: simply visit the question and unleash your [close|up|down]votes and/or (un)delete powers to get it fixed, with help of some peers from a chatroom. The effect of the failed audit will only be felt a couple of days (due to the more audits you get)
    – rene
    Commented Jan 24, 2015 at 10:42
  • @rene Can you elaborate on exactly what you see as a negative side effect of this proposal there? Certainly, in the current setup it's not impossible to deal with bad audits, but this is usually a) done after the fact b) done with considerable noise c) done considerably less often than bad audits occur and d) only marginally useful for future reviewers. This proposal addresses all of those. Commented Jan 24, 2015 at 10:45
  • @rene Oh, also, I did say "somewhat" less complaints by good reviewers. ;) Commented Jan 24, 2015 at 10:46
  • What I miss is the number of failed audits in total vs the number of complaints on meta. I don't think it is that much of a problem (as in there are bigger problems that the dev, mods and users can spend their time on).
    – rene
    Commented Jan 24, 2015 at 11:00
  • @rene Weighing cost/benefit is a reasonable thing to ask (though it's not a negative side effect, just a question of priority), but I certainly do not have the data to determine how many failed audits there are. A [status-deferred] on this proposal would be quite nice to have, though. Commented Jan 24, 2015 at 11:04
  • 5
  • 1
    @rene I do appreciate that answer (although I strongly disagree with his assertion that hand-picking doesn't scale: it clearly does, from the numbers I've given, and extremely well no less), and his suggestion would be quite good if implemented. But there's no sign of that happening, so until then, addressing the noise and the problem is still profitable. Commented Jan 24, 2015 at 11:21
  • Relevant: meta.stackexchange.com/questions/188780/… Commented Jan 24, 2015 at 12:18
  • 1
    It's instructive to note that, in the five months since I asked this, another 400+ users have reached 20k, out of less than 3300 total at this moment, while reviews per day have apparently gone up by roughly 10000. SO is still growing rapidly in some ways! Commented Jun 12, 2015 at 3:44

2 Answers 2

15

Thank you for the detailed feature request. It's a refreshing break from the usual "X is broken, fix it!" requests that we usually get. This being said, I'm skeptical.

Highly likely benefits:

  • Less moderator time spent on debating the validity of X specific audit.

Probably less but is it going to be lower enough to counter the work needed in the new queue? I am quite certain that people are still going to complain about audits, seeing as many complaints are illegitimate. (e.g. Someone letting through what is clearly spam, failing the audit and then complaining.)

Even when the complaint is legitimate, moderators do not need to be involved in the resolution (for instance). I realize this does not entail that moderators did not read the complaint or did not perform any moderation action on the complaint (e.g. deleting comments).

Looking at disputed-review-audits, it seems to me that 1 dispute a day is an overestimate of the rate at which audit disputes come up on Meta. I'm thinking it is enough of an overestimate to account for deleted questions but I could be wrong. Only someone who can access deleted questions would be able to quickly get statistics that include these. 1 dispute a day is not much.

  • Considerably fewer meta posts from normally-competent reviewers getting banned or wanting an audit disabled.

I don't think normally-competent reviewers get banned. To get banned one has to fail audits to the point of no longer being "normally-competent".

Potential benefits:

  • Slightly fewer bad reviewers, if the algorithm is relaxed to allow more subtleties in and mods/20kers pick good hard audits that are still unambiguous.

No thanks. The audit system is designed to quickly weed out robo-reviewers. It is not designed to be a bar exam. If you start adding subtleties to the system because you are trying to teach people how to review, what you are going to end up with is more people complaining. "Ah, yes you clicked 'leave open' but if you look at the 34th sentence in the 3rd paragraph, you'll clearly see the OP is asking for an external resource. Gotcha!" And they'll complain even more stridently than they do now because they audits will have been hand-picked. They'll feel like someone is out to get them. Right now we can tell them that the algorithm is stupid.

Then it seems to me that the biggest benefits this proposal would provide would also for the most part be provided by Brad Larson's proposal. His proposal seems simpler to implement and would require less overall human intervention.

4
  • 2
    Brad Larson's proposal is a lot like Western civilization: it would be a fine idea! But somehow it hasn't been implemented yet. So while I do support it, I think raising alternatives is reasonable. As far as the rate of disputes goes, it's the iceberg problem: for every meta post, there's a good 10 cases that weren't posted about. I've posted at least twice myself, but there's probably another dozen or more audits that I thought were questionable, but didn't feel quite confident enough in to bring up. A lot of reviewers would be too nervous about taking the dispute plunge for us to see them. Commented Jan 25, 2015 at 2:49
  • 1
    I should perhaps have rolled "subtler audits" into "specific audit reasons", because the former largely depends on the latter; if you can tell someone in the audit why it was (in)valid, that's pretty handy and lets you go just a bit further in improving skills, which is explicitly a goal of the current system. As far as "normally-competent" reviewers, well, if they're banned because of 2+ bad audits, whose fault is that? Surely not their own. Even one could make the difference between "off streak, need to slow down a bit" and "ouch, stopped cold". Every bad audit counts. Commented Jan 25, 2015 at 2:53
  • 4
    Well, I guess it could mean that I'm not a "normally competent reviewer", but I once got two review bans in fairly short succession. To get those two bans, I only failed one audit where I regretted my vote. The other were bogus audits, IMHO. I have done about 5000 reviews in total, so I'd like to think that I'm not totally incompetent. Commented Jan 25, 2015 at 3:46
  • 6
    Also, I don't think everybody will post here for every questionable audit. So your statistics based on disputed-review-audits posts are not conclusive. I have a feeling that it might be much more common that people simply stop reviewing. Commented Jan 25, 2015 at 3:47
12

To add to Louis's excellent answer, I'll provide some statistics:

There are a total of 131 , including those deleted. They span 281 days, putting this at roughly 1 disputed audit every 2 days.

During this same time period, there have been 283,545 audits and 18,697 failures, meaning we get roughly 1 audit disputed out of every 2,164 shown, or one out of every 143 audits failed.

What does this mean for your proposal? Nothing yet. There's a huge variation in how many reviews (and hence audits) individual users do. A total of 17,424 users were audited during this period, with an average of 16 audits per user and a maximum of 2,935 audits for a single user. In theory then, we could get away with generating 10 or so unique audits per day, as long as we were careful to track who got them and never re-show them. There are a few confounding factors though...

  • We probably shouldn't be showing you audit tasks based on posts you've already interacted with. This doesn't always happen, but we do try to avoid this now. Of course, this means that we shouldn't audit anyone approving the audits with the audits they've approved.

  • We'd have to be a bit cautious about changes made to questions after they've been picked as audits. We can use a snapshot of the audit post at the time it was approved, but we should still avoid using known-bad questions if, for instance, they get undeleted / reopened, or known-good questions that get deleted.

  • If we run out of audits, we're gonna have to fall back on the automated systems. Which means we can't really slack on those either, since all it takes is one person every two days to not like one of 'em and we're back where we are now.

  • Folks disagree with moderators and 20K users all the time. Just because the system trusts them doesn't mean the folks failing audits are going to. So we'll still get some number of disputed audits.

But let's assume the best case: 10 audits per day. That's at least 5 20K reviewers, reviewing at least 10 potential audits every. single. day. To save answering one meta question every 2 days.

This is why Brad's proposal is so appealing: if we're going to have folks reviewing audits, might as well focus on audits that folks think are actually problematic. I'd probably turn that around though: only review audits that've been reported a couple times and only invalidate them after review.

5
  • 1
    I don't think the other benefits of this proposal are quite so negligible as all that, though. "It saves on meta questions" is less than half of what it's for. In particular, substantially improving the teaching ability of audits in ways that are presently highly impractical (e.g., explaining what an audit was designed to catch) would fit in quite smoothly, and I consider that useful in general. As far as running out of audits, that should be extremely rare, given that the queue keeps populating itself until the pool is full, meaning that rejections just pull more candidates in. Commented Jan 25, 2015 at 2:58
  • 1
    For improving the teaching ability of audits, I rather like this suggestion - again, it doesn't require additional busywork from trusted users, but does offer them the chance to share their expertise upon finding a suitable opportunity for it. Keep in mind, the automatic selection algorithms are extremely conservative - known-bad audits in particular tend to be unmitigated garbage, which does not exactly lend itself to teaching subtle aspects of reviewing.
    – Shog9
    Commented Jan 25, 2015 at 3:36
  • Yeah, I looked through that a couple times already. It's OK, but suffers a few problems: "good reviewer" is not well-defined (though basing it on something beyond review queue behavior is a good start) and may actually be quite difficult to define robustly; less importantly, the mixing of manually-selected and auto-generated audits is not clearly addressed. And while audit quality would likely improve somewhat, the increase is not as notable as this proposal. Commented Jan 25, 2015 at 3:47
  • 3
    It's also worth considering that a lot of disputed audits aren't reported; a lot can be found from complaining in chat rooms, but once you report one disputed audit, posting another one isn't too tempting (what's the point if you already know what to do to counteract it?).
    – AstroCB
    Commented Jan 26, 2015 at 4:25
  • 5
    And not all the ones reported are valid either, @AstroCB. Both are reasons for wanting a better method of reporting - if we can't be sure there's even a problem, trying to fix it is doomed.
    – Shog9
    Commented Jan 26, 2015 at 20:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .