27

I just "failed" this review audit.

https://stackoverflow.com/review/low-quality-posts/11663700

Apparently I should have labeled this answer as spam. Except that... it clearly wasn't.

The question in itself is one from 2011 that wouldn't pass current standards because it is asking for product recommendations. All the other answers are also links to products.

The product in this answer is, as the answer clearly identified, an open-source framework called "Summerbatch" that is distributed under the Apache 2 license. All of this is correct.

I don't see any reason at all why this answer should be classified as spam, or even deleted at all, as it is a good answer to a question that is bad by current standards.

I could see some rationale for downvoting it if you believe that bad questions shouldn't be answered, but deleting it is - as far as I know - reserved for things that aren't answers at all (or really spam/offensive).


To the duplicate suggester: this is a different audit, and in this case it is clearly not a spam, unlike your suggested duplicate, where even the OP indicated that he was doubting whether it was spam or not (and since it was a link to a youtube video advertising a commercial product without disclosure that it was such in the original posted, it was clearly spam). My case and your suggested duplicate are entirely different.

6
  • 5
    I take it you have voted to close the question already? Commented Mar 17, 2016 at 8:04
  • 11
    Today I learned that a post with a clearly spammy link is not necessarily spam, so I'm not really sure how anyone would ever be expected to know that this type of answer is spam. Someone is wrong here in their definition of "spam".
    – Cody Gray Mod
    Commented Mar 17, 2016 at 9:18
  • About the only time I've ever encountered spam here is via review audits. Don't take it to heart, it's just the mechanics.
    – user207421
    Commented Mar 17, 2016 at 10:30
  • 1
    What I find curious is that you chose "Looks OK" rather than "Edit". Putting aside the erroneous spam issue, there are other, obvious problems with that answer that would warrant some editing. Commented Mar 17, 2016 at 10:40
  • @Jean-FrançoisCorbett That was an option. I typically don't do while reviewing close/low quality posts that unless it looks like I could make a big improvement but it's a good suggestion to look into that more often. Commented Mar 17, 2016 at 13:05
  • 3
    I have always been against using spam in the LQRQ.
    – Braiam
    Commented Mar 17, 2016 at 14:41

2 Answers 2

33

Firstly, as Chris mentions in his answer it's always best to be wary of new answers to old questions - however, this is a bit of a weird one which I'll try to explain without too much boring detail.

The answer in this audit was spam flagged by six community members which caused the Community user to lock and delete the answer. Now, I do agree with you, that although it's not a great answer, in isolation it looks like a good faith answer to the (albeit now off-topic (and now closed)) question. If it's any consolation, I'd have probably failed the audit myself, and looking at the audit history of the post - you and I wouldn't be the only ones.

What you couldn't have known was that the same user posted the same answer to a different question (which was also spam flagged and deleted like above).

I'm guessing that the reviewers that monitor the new answers to old questions queue thought little of one answer coming in containing a url to a bad question, but upon seeing another one with the same url by the same poster in a short space of time, then proceeded to use their spam flags - hence where we are now.

The post should fall out of the audit system sooner rather than later. I wouldn't worry too much about the occasional failed audit - sadly no automated audit system can be perfect and this is a case where - let's just say it's not been exactly great.

If you'd have received a review ban on this - we'd have lifted it immediately after it was brought to our attention as audits shouldn't require you to do research to put the jigsaw together before making your choice.

It's frustrating, but for now, shrug it off, don't worry about it, and keep up the good work.

18
  • 2
    Maybe it would be nice if the user(s) marking the answer as spam left a comment in these complex cases. Commented Mar 17, 2016 at 11:33
  • "If you'd have received a review ban on this - we'd have lifted it immediately after it was brought to our attention as audits shouldn't require you to do research" - but reviewing should require you to go chasing mods every so often because the system is broken?
    – Dan Hulme
    Commented Mar 17, 2016 at 11:40
  • 2
    @DanHulme The merits of audits have been discussed at lenght dozens of times on here. This is not the place for that.
    – Magisch
    Commented Mar 17, 2016 at 14:32
  • "audits shouldn't require you to do research to put the jigsaw together before making your choice." sadly, spam cannot be identified as such since they are now more sneaky, human written spam.
    – Braiam
    Commented Mar 17, 2016 at 14:40
  • @Magisch You're right, they've been discussed at great length. That's why I don't want to start a fresh discussion, I just want to highlight the evidence when it comes up.
    – Dan Hulme
    Commented Mar 17, 2016 at 14:43
  • 1
    @DanHulme: You know, if SO had a perfect system for identifying good review-audits, they would just use it to automatically handle all the bad posts instead, and retire the flags and reviews completely. Alas, they couldn't quite get their AI-project done on time, meaning they have to balance false-positives against false-negatives, and going too far to one side makes the other utterly inacceptable. Commented Mar 17, 2016 at 14:59
  • 3
    @Deduplicator I don't think the people who implemented the system realised they were balancing false-positives against false-negatives. If it had entered their mind that they would be putting good-faith, effortful reviewers through the "failed audit" process, they wouldn't have made it such a negative experience. I'm not arguing for a process that's always right, I'm arguing for a process that doesn't make the reviewer feel cheated and unvalued.
    – Dan Hulme
    Commented Mar 17, 2016 at 15:59
  • @DanHulme Well, there's also lots of bad input for the system. Anyway, have you seen this post? Commented Mar 17, 2016 at 20:57
  • @Deduplicator I hadn't, but thanks for the link. Upvoted, but even if it were implemented, it's too little, too late for me. From the point of view of a human reviewer, having to "appeal" against the decision of a machine that's known to make wrong judgements is already a failure in the process.
    – Dan Hulme
    Commented Mar 17, 2016 at 23:34
  • 1
    On another note, why exactly is the audit system using spam in the LQP queue? Its not like thats its intended purpose, to identify and close spam.
    – Magisch
    Commented Mar 18, 2016 at 5:06
  • @DanHulme So, you want a perfect ai for independently evaluating posts, or a squad of humans picking a hundred review-audits per queue or so daily? AI is too hard, as is getting enough perfect selecters for review-audits. Anyway, that's getting a bit far, and there are already other posts about that... Commented Mar 18, 2016 at 17:42
  • @Deduplicator No, I want the experience of review audits to be less unpleasant and onerous for reviewers.
    – Dan Hulme
    Commented Mar 18, 2016 at 23:56
  • If 'audits shouldn't require you to do research' why are the review audits full of spam traps that require you to do exactly that?
    – user207421
    Commented Mar 20, 2016 at 6:23
  • @Braiam we still get enough markov chain spam even today. It's fun to write automated scripts to catch these. Just visit the Tavern once in a morning GMT. Commented Mar 20, 2016 at 6:57
  • @EJP because at first "spam" seems like a good choice for "obviously bad content". You are welcome to post a feature request to change the audit algorithm. Commented Mar 20, 2016 at 6:59
7

The wrinkle here is that this was a new answer (Feb 2016) to an old question (May 2011) that had already been deleted through spam flags.

This means that the system will automatically pick it as an audit candidate. You may not agree with spam flags, but the system thinks that it is and that's all that the audit system looks at. All you can do is raise the issue here and perhaps (though it's not certain) the team will remove this answer from the audit system.

The whole purpose of the audit system, as we keep saying, is to make sure you are paying attention. You should have noticed that it was an new answer to an old question and as such you should be on the look out for it being a possible spam answer to a recommendation question.

However, having said that, don't worry about the occasional failed audit. Everyone fails them from time to time (including moderators - I speak from personal experience here :( ). You should only start to worry if you fail several in a row.

3
  • 3
    I get and agree with the "don't worry too much" part, but as for the rest, I'm not sure what your message is here. Are you lecturing the reviewer for not paying enough attention? That would be a bit crass. The reviewer accurately marked this as not spam. Even with infinite attention paid, that would still be the right outcome. Commented Mar 17, 2016 at 10:40
  • 2
    @Jean-FrançoisCorbett - I didn't mean to lecture, just point out where the system "thinks" differently to the average user.
    – ChrisF Mod
    Commented Mar 17, 2016 at 10:46
  • 1
    Smiley in parentheses? xkcd.com/541
    – DJMcMayhem
    Commented Mar 18, 2016 at 17:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .