-19

In my opinion, the review audit system is broken and the suspension system is just a tad bit too harsh. If I am correct, and I don't know if there is a cap on suspension days or not, the amount of time someone is suspended for grows exponentially every time they are suspended (i.e. a fifth suspension for a user would result in a 32 day ban). I also think that there are too many audits that are not actually correct (i.e. a bad post that was recieved well counts as a good post in an audit).

Here are my suggestions to fix this:

[1] Moderators or users with high reputation should manually choose audits. If my assumption is correct, there is some kind of algorithm that SO uses to pick audits, and it seems to get audits wrong a lot. If the mods/high-rep users just had the option to set a question as an audit with a correct answer, I think that this problem would go away.

[2] Change the suspension time from being exponential to logarithmic. Everybody makes mistakes, and while I think that this current system is fine for new reviewers, there are veteran reviewers that might fail an audit and get suspended for 64 days if they have been around long enough. If the system used a logarithmic function, mistakes for high-rep users would still be penalized, just in less of a harsh way. A suggested function to use would be ||log_base_2(x)*10|| (|| || is a rounding-down function), because the high rep user gets suspended for 25 days, a lot better than 64, and the amounts a user gets suspended for start to get pretty similar as # of suspensions grows larger.

Any thoughts?

6
  • 4
  • 5
    You don't get a review ban for failing one audit. You get them for failing several during a period of time. As time passes, previous failed reviews or bans age and eventually no longer count towards your next ban.
    – Lundin
    Commented Mar 29, 2019 at 7:42
  • 2
    But well yeah, the review audits are completely broken and have been so for many years. SO doesn't care and will not fix it. The solution is to stop reviewing until SO employees have to do it themselves. Only then will something happen.
    – Lundin
    Commented Mar 29, 2019 at 7:46
  • 6
    I agree audits are often garbage, but I can't help but point out that 'a fifth suspension for a user would result in a 32 days ban'.... Well I mean, when you've been told four times to be careful and don't seem to pay heed to it, I don't think a month is harsh :/. Now.... It would indeed be better if the audit selection system was less flawed, but this is (to me at least) nowhere near harsh....
    – Patrice
    Commented Mar 29, 2019 at 11:17
  • 1
    I appreciated the recommended solutions presented here, even if I don't agree with the suggestion of manually run audit system due to workload concerns. Is the suspension function truly exponential based on all previous audit fails, or just the last so many day's worth? Based on comments here and here I wasn't sure. Commented Mar 29, 2019 at 15:12
  • 2
    +1 manual review of generated audit candidates. -1 too harsh.
    – user1228
    Commented Mar 29, 2019 at 20:05

1 Answer 1

25

I'd be down for improving how audits are selected, whether that's a concrete suggestion on how to tweak the algorithm or a complete overhaul of the entire system, but two things to keep in mind:

  1. Don't overestimate the frequency that audits are bad/wrong. Your estimation of that is going to be biased by the "evidence" you see on Meta due to strong selection bias. The vast majority of audits are accurate, but nobody ever talks about those. They either pass them and move along, or they fail them and are embarrassed.

  2. Picking (or reviewing) audit candidates manually would be a lot of drudgery. We really don't want to introduce another review queue to pick the review queue's audits. But I can't argue against introducing a human factor to review audit selection.

    As a moderator, I wouldn't mind having a button that says "make this an audit" when I process a flag. I come across a lot of good candidates. But I don't think that would be enough audits, given the massive number of reviewers on Stack Overflow.

An even better option than manually-curated review audits is an appeal mechanism (something a bit more formal and scaleable than a Meta question, which is the currently-recommended escalation procedure).

As for your second suggestion...no. The suspension system is nowhere near "too harsh". You've also got some mistaken assumptions here. In particular:

...[T]here are veteran reviewers that might fail an audit and get suspended for 64 days if they have been around long enough..."

This is not how it works. The only way that would happen is if that "veteran reviewer" was also a horrible reviewer, and then they deserve those escalating suspension periods. History is only taken into account to escalate suspension time periods if it's recent review history. Otherwise, the time resets to the starting point (which is 2 days).

Audits also aren't catching all of the problems. I hand out a fair number of manual suspensions for reviewers making completely indefensible decisions in the review queues, as do other moderators, and those aren't even a drop in the bucket. Most of the people I end up suspending haven't been caught by audits.

The only thing worse than not enough reviewers is bad reviewers. Not enough reviewers just means that stuff's not getting reviewed. Bad reviewers mean that stuff is getting reviewed and handled incorrectly, so garbage is getting enshrined on the site. And, the reality is, we've got plenty of reviewers right now anyway, so even that's not a problem in need of a solution.

Maybe the real solution is to stop gamifying reviewing? Cleaning up the site should be its own reward, and if you're not motivated by that, then fine. There are plenty of other ways for you to contribute.

5
  • I do think that reviews are somewhat gameified, and I do think that shouldn't be a thing. An appeal mechanism does sound like a good idea thouhg!
    – Jodast
    Commented Mar 29, 2019 at 5:12
  • 2
    I've deserved my review bans. I will say after a failed review that seemed unclear, I use the skip feature more often. Commented Mar 29, 2019 at 15:14
  • 3
    "But I don't think that would be enough audits, given the massive number of reviewers on Stack Overflow." But does every reviewer actually need to be given their own audit? The only real reason to get lots of different audits is so that users don't figure out which posts are audits, and then tell their buddies, and frankly there are way easier ways to figure out if a post is an audit (like just clicking the link to go to it and see if it's deleted). So since you can give most/all reviewers the same audits, it should scale just fine. You only need enough audits for the most active user(s).
    – Servy
    Commented Mar 29, 2019 at 16:40
  • 5
    You could also make a chatroom/meta post/etc. for people to suggest good audit candidates, which if kept active, would mean mods would only need to rubber stamp suggestions. You could even use the existing mechanism as a fallback if/when the mods fall behind, meaning audits would be better if the mods are feeding them posts, and it's at least as good as the current system if they fall behind.
    – Servy
    Commented Mar 29, 2019 at 16:43
  • I agree with most of this answer, but I'm not really sure where this idea of manual audit approval being some sort of unsustainable drudgery comes from. Just the diamond mods could take care of all the audits for maybe 5 minutes a day a piece… and that's assuming you take several minutes to confirm each audit. Commented Apr 5, 2019 at 4:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .