50

https://stackoverflow.com/review/close/8759750

enter image description here

I really don't even want to elaborate on the answer to how on Earth this could be a good audit example.

I've been having some discussions off and on with one of the CMs about this type of failure happening, and I'm really at a loss as to how to weed these out of the pack.

At times, I think with this close vote queue, it's time to throw out the baby and the bathwater and rip the tub out, i.e., completely change the dependence on upvotes for question selection (and perhaps take into account other criteria such as the number of times the audit has been failed as a function of a typical reviewer's track record).

How can something like this count against a diligent reviewer? Honestly. I have expressed to said CM that it really seems like there's no effort to change these measures of a good review audit. In fairness to him, he did ask me for suggestions, but my suggestion really is "tear out the criteria and start over". I hate to say it, but most of the time these failures are just shrugged off with a "the system is flawed, deal with it," and that's extremely frustrating.

23
  • 8
    The number of votes is the reason it got selected as an audit - the criteria requires upvotes only, at least 5 of them I believe.
    – nobody
    Commented Jul 11, 2015 at 1:03
  • 9
    @AndrewMedico Fine, I get the theory, but is that even a remotely reasonable way to evaluate a reviewer? I'm a mod on two sites and I've been a member of SO for 4 years and no matter how I squint at that, it looks like utter garbage to me. I should point out that this is not the first time this has happened to me.
    – jonsca
    Commented Jul 11, 2015 at 1:04
  • 1
    I don't think it's a good idea at all... just saying that's they way it is currently.
    – nobody
    Commented Jul 11, 2015 at 1:09
  • 9
    Yeah, you got caught by a poor review. Total bummer, that. Downvote the question so that it's less likely to be selected as a review candidate.
    – Makoto
    Commented Jul 11, 2015 at 1:36
  • 6
    No audits is no solution. But better disputing, that would be nice: meta.stackexchange.com/a/188790 Commented Jul 11, 2015 at 1:38
  • @Makoto Thanks for the condescension. This has happened to me more than a handful of times over the past couple of months. This is a bigger problem than "geez, sorry".
    – jonsca
    Commented Jul 11, 2015 at 1:38
  • 2
    So would linking to this answer be better, then? The audit system is definitely in need of some fixing, but the fact that those sorts of questions do come up in audits is well-known. I'm not entirely sure what you're looking for from us here; do you want to discuss a way to fix the audit system, do you want to complain about the audit system, or are you now review-banned from this particular audit failure?
    – Makoto
    Commented Jul 11, 2015 at 1:44
  • 2
    @Makoto I'm pretty sick of the whole process, so I'd certainly like to bring up those points of discussion. Good reviewers shouldn't be punished for volunteering their time only to have it slapped back in their face over something that's absolutely not correct. Whether I'm banned is not germane and is frankly none of your business.
    – jonsca
    Commented Jul 11, 2015 at 1:47
  • 3
    @jonsca there are literally hundreds of audits per day and and there are on average zero posts on meta complaining about audits. While there are likely bad audits that don't get reported, it still stands to reason a vast majority are reasonable. What would you prefer? A human approve all audits? Given the volume necessary, that would me 2 or 3 people 100% dedicated to approving audits. Given human falibility, they are still going to approve bad audits. So what do you want? Commented Jul 11, 2015 at 2:07
  • 1
    @psubsee2003 I've added what I want. Toss out the criteria. Have new criteria that doesn't trap people who are trying to spend their time improving the site. Like I said, this has happened to me more than a handful of times, and it's a bit sickening. Assuming I am a relatively new user and have no idea of the criteria that is used to select audits, what am I supposed to learn from the above?
    – jonsca
    Commented Jul 11, 2015 at 2:10
  • 3
    @jonsca tear out the criteria and start over is not a valid suggestion. It is a sentence that says "the status quo doesn't work but I have no better options, so fix it immediately". This has been going on since the audits started almost 3 years ago. I'm not trying to be difficult or condescending, but I'm tired of the bitching and complaining. Real problems need real solutions, not just requests for other people to fix it. Commented Jul 11, 2015 at 2:18
  • 3
    @psubsee2003 Without revealing anything that might not be known, they did completely overhaul the criteria for the Triage audits. This is feasible. I'm tired of people saying "Oh, well, I click through them and so you should just do that," and it's about time that something is done about it. Perhaps it takes someone with 4 years of experience (and probably around 20000 reviews network-wide) along with extensive moderator experience to say that enough is enough.
    – jonsca
    Commented Jul 11, 2015 at 2:21
  • 7
    @psubsee2003 I don't think the number of meta posts is a reliable measure for how many bad audits there are. I suspect that most people will either just continue reviewing as long as they don't see the bad audits enough to get banned, or they will stop reviewing entirely. Patricia's answer below captures the sentiment very well. I review much less than I used to myself, and the bad audits are a main reason. Commented Jul 11, 2015 at 16:35
  • 1
    @psubsee2003: Re human-validated audits, I have a rather thorough proposal exploring the feasibility of that here -- no need to throw around random guesses at the burden involved when I've put a lot of work into making detailed estimates. Commented Jul 11, 2015 at 19:27
  • 2
    @NathanTuggy: Though as Shog9 says, you only have at best anecdotal evidence that there's enough of a problem to warrant throwing away the current system, or at least install a preferred one for first-shot-handling. A better disputing system like meta.stackexchange.com/a/188790 is needed for that though. Commented Jul 12, 2015 at 21:03

3 Answers 3

55

There is a simple solution to your frustration: Just stop reviewing. SO is a complex collection of games, not a job. Pick the games you enjoy, and skip the ones you find frustrating.

Different people have different levels of toleration for the audit system. Mine is very low. I tried reviewing, found the audits frustrating enough to make it not fun, and so I don't review any more.

If there are enough audit-tolerant reviewers to keep the system operating, there is no reason for SE to put any effort into refining the audit system. If too many people were to decide the current system is not fun, there would not be enough reviewers to keep up with the queues, and SE would have motivation to try to design a better system.

Continuing to review while finding the audit system frustrating enough to complain about is a waste of your energy, and does nothing to force changes.

8
  • 14
    That, I think, is some of the sobering truth about it. I still have chosen to give my time to a network of sites that is striving after an overarching principle. It's just highly unfortunate to be treated as if I'm badge hunting or damaging the network in some way, when there's clearly checks and balances already in play for close votes.
    – jonsca
    Commented Jul 11, 2015 at 7:06
  • 10
    This is a terrible answer, and the implications of it are even more so. Choosing to vote/moderate should not discouraged, especially these days when SO is full of low quality, duplicate, bad grammar questions and answers; indeed, voting should be encouraged and the system should be improved where possible. Your solution is to surrender, and worse - accept a bad system because a majority chooses to suffer in silence. Sorry, this should not be the highest voted answer to this question.
    – Léo Natan
    Commented Jul 12, 2015 at 21:01
  • 17
    @LeoNatan I don't accept the bad audit system. I am taking the very strongest action within my power to get it fixed. Going on reviewing is surrender and acceptance of a bad system. That is what lets SE get away with not bothering to fix it. If everyone who dislikes the current audit system followed my path rather than suffering in silence I believe some of the existing proposals for improving it would be implemented within weeks. Commented Jul 12, 2015 at 21:18
  • 1
    related: Close Votes review: I'm going on a strike!
    – gnat
    Commented Jul 13, 2015 at 6:23
  • 4
    You have summed up the reality very well. If you don't like it, "stop reviewing". Well done on stepping up and saying what a lot of people won't. Commented Jul 13, 2015 at 6:47
  • 1
    Absolutely. I make up my own games instead, such as improving really bad posts manually. Plenty of SO games to go around!
    – halfer
    Commented Jul 13, 2015 at 13:44
  • 2
    -1 This kind of advice is what we call in the Netherlands: "voting with your shoes", i.e. you walk away without making any complaint, leaving the people in charge having to guess what the reason is of people leaving. Complaining should always be done at the place where it can make a difference. If (this) meta is not the proper place to do so, which is? If there is none, something is very off.
    – Jan Doggen
    Commented Jul 13, 2015 at 14:17
  • 7
    @JanDoggen I agree it is better to complain first, so that the powers-that-be know what is wrong. There are several meta threads that include complaints about the audit system with constructive suggestions for improvement. If anyone ever wanted to find out why people like me don't review, it would be easy. I just don't see any point in going on complaining when it makes no difference. Commented Jul 13, 2015 at 14:45
18

Yes, bad audits exist:

Lots of people are actively ignoring SO's scope, upvoting bad questions because "they want to be nice", downvoting good questions "because he criticized me" and the like. And sometimes, noone who gives a damn comes along that fast.
So, we have lots of bad signal.

As Charles Babbage famously said sometime in the 19th century:

On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

You want to have infallible audits?
Either get lots of perfect volunteers for selecting audits and getting yelled at for anything anyone ever fails (I doubt you'll find a single one, and anyway, it does not scale), or fix all those idiots upvoting off-topic or otherwise bad content, not flagging/closing where appropriate and the like (Sisyphus greets).

There are proposals for making things better, like adding a "This audit is incorrect" choice, but you really don't want to simply rip audits out: Remember that audits are primarily for detecting and dealing with robo-reviewers, and they succeed in filtering out most of the really bad guys. Nearly all audits are far too clear-cut to pose any kind of challenge for anyone else.

Also, one has to fail multiple audits to get banned, and unless you get all the duds, that's really unlikely.

3
  • 7
    I click "I understand" with the implication that "I understand that this review is based on a GIGO algorithm", then go downvote/flag the question.
    – o11c
    Commented Jul 12, 2015 at 19:36
  • I agree with most of this, but I would very much like to see anything other than bare assumptions that manual audit generation cannot scale. Surely someone has run the numbers to show why it doesn't scale! Otherwise, taking that position just seems like premature optimization, which, about 95% of the time, is the root of all kinds of evil. Commented Jul 12, 2015 at 21:09
  • Perhaps those with knowledge can do some kind of audit marketing to combat this type of bad press. Maybe if we saw how useful the audits actually are, we could understand better. Commented Jul 13, 2015 at 6:50
10

The review system has some problems. You can review a post negatively that you genuinely feel is crap, but if the algorithm does not agree with you then you fail. In addition the system tricks you by anonymizing the user, lowering their reputation significantly and lowering the score of the question/answer.

Sadly I have found that the best way to do reviews is to look at the actual question/answer. So for every single review I click the link to actual content, and if it does not match (deleted/not deleted; closed/not closed) then I know it is a test.

What happened to review bans?

1
  • 4
    "The algorithms" simply reflect what the masses who saw the question decided. GIGO is nice, eh? Commented Jul 12, 2015 at 3:58

Not the answer you're looking for? Browse other questions tagged .