-2

While reviewing I came across a well formatted question, which had:

Our system has identified this post as possible spam; please review carefully.

Being a well formatted 0 score question I thought it was a test, so I clicked the link provided in the review, which linked to a +6 score question, before I actually complete the review. Then I completed the review with Looks OK.

By "link" I refer to this:


Enter image description here


Seeing most audits are concealed appropriately, and a simple click to the link can ruin this one, I started wondering how effective some audits are. Even a review bot (removed that since it caused confusion, my point was that it's too easy). Anyone could "click" the link and mindlessly pass this test.

Is it possibly a bug in the audit creation system, or is it a situation that wasn't predicted?

Note: The "Our system..." message is irrelevant. It matters not if it is present or not.


FURTHER TESTING

I do not know if this happens on every single audit or if you have set any "traps", but I have managed to pass the following review audits with 100% accuracy and without reading the question content:


26
  • 1
    @rene The suggested duplicate question sums up to "Where is the challenge when the expected review audit result is displayed?" (that is, he considers "Our system.....possible spam" a hint). My question sums up to "a bot can simply check the upvotes in a link and decide 100% accurately.".
    – user
    Commented Mar 9, 2015 at 14:09
  • A bot can always decide 100% accurately. Although the review out-come might be different there is not much else to set this question apar from the proposed duplicate and that answer will also hold on your question. I'm not conviced, yet, that I've choosen the wrong dup.
    – rene
    Commented Mar 9, 2015 at 14:19
  • Could also be that the post has an spam flag as explained in this answer: meta.stackoverflow.com/questions/253770/…
    – rene
    Commented Mar 9, 2015 at 14:23
  • 4
    @rene "A bot can always decide 100% accurately. " If this was really the case, there would be no need for human reviews.
    – user
    Commented Mar 9, 2015 at 14:24
  • @rene Perhaps i should have stated that it was a 0 score question in the audit, linking to a +6 question, before i pass the audit. Is that what all those proposed duplicates are about? Edited the question to include it.
    – user
    Commented Mar 9, 2015 at 14:26
  • 6
    I meant to say that a bot can always decide the correct review outcome, in other words it will never fail an audit. I'm not saying bots are curently capable of indetifying spam 100% correctly.
    – rene
    Commented Mar 9, 2015 at 14:27
  • 21
    Yeah, and the point is that robo-reviewers don't do that so in this case they might get lured into clicking Recommend deletion and thus fail the audit. Anyone who visits the actual question when presented an audit is not the kind of user the audit systems tries to catch.
    – rene
    Commented Mar 9, 2015 at 14:35
  • 3
    If you follow the link to investigate the question further, the audit has already accomplished its goal. It isn't there to trick you or to try to get you banned, it's there to make you have to pay attention and to catch those who aren't.
    – Kevin B
    Commented Mar 11, 2015 at 21:25
  • 6
    If they "exploit it", they are paying attention.
    – Kevin B
    Commented Mar 11, 2015 at 21:30
  • 3
    I'm sure it's been a bit frustrating getting the same response over and over about this being completely fine and intended behavior.... But I think you're ever-so-slightly missing what people have been trying to tell you: Robo-reviewers don't click through to the question, and audits are there to catch robo-reviewers. I know you understand what audits are, but I think you're slightly missing what robo-reviewers are. Robo-reviewers don't click through to the question. Instead, they blindly hit whatever button gets them through the reviews fastest hoping to game badges. (cont.)
    – Kendra
    Commented Mar 11, 2015 at 21:31
  • 3
    It's certainly possible that someone goes through the review following every link to make sure it isn't an audit, and then clicking any of the review buttons but skip. Is that the case you are talking about?
    – Kevin B
    Commented Mar 11, 2015 at 21:33
  • 2
    If the user is trying to game badges, but is clicking through on questions to check if it's an audit, then the audit has at least made that potential robo-reviewer pay more attention, meaning that they no longer classify as a robo-reviewer. A bad reviewer, possibly, but not a robo-reviewer. I think that's where you're getting stuck on what's been said so far. You're thinking only of badge-gamers that are at least smart enough to watch for audits, but there are still reviewers going for the badges that don't click through and check for audits. Those people are the ones audits are for.
    – Kendra
    Commented Mar 11, 2015 at 21:33
  • 2
    And honestly, these bad reviewers you seem to be thinking of that are just going for the badges are going to end up letting spam or other horrible posts get through and are going to end up manually banned by a moderator, anyway. So even if they pass the audits and are only checking for those then hitting a button to increase their review count, they are still going to eventually hit a post that should be stopped, review incorrectly, and be manually banned.
    – Kendra
    Commented Mar 11, 2015 at 21:35
  • 5
    I don't think an audit can reliably catch users who are doing that without removing important functionality from the review process. We don't want to prevent people from going to the question being reviewed, and that's really the only thing that would prevent someone from getting around the audit in this way. Users that abuse the system in this way can likely be caught by statistics instead.
    – Kevin B
    Commented Mar 11, 2015 at 21:44
  • 1
    @user5061 I can at least confirm that manual bans do happen, as I have witnessed it when someone took a closer look at edits that a suspicious user had got confirmed here: meta.stackoverflow.com/questions/265169/… and that happened on a totally unrelated question I posted when I didn't understand the concept of binding votes from post owners. So "manual audits" do happen, usually when other irregularities are discovered.
    – ivarni
    Commented Mar 15, 2015 at 21:26

3 Answers 3

24

Yes, you are correct, the audit system is exploitable. You can indeed visit the question by clicking the link (or even clicking the question title). And by doing that you'll never fail any audit, ever.

It is not a problem at all because the audit system was not designed to catch the reviewers that are prepared to visit every question from an review to verify the correct action required.
Worst what will happen, I think, is that those users will gain badge while being poor reviewers.

Let's take a step back and try to solve the issue. We want to design an audit system that is unexploitable. I see some options:

  • We could disable the links on the page that let you visit the post
    This would indeed make it harder to exploit the 2 audits on every 50 reviews but it would also make the life of the reviewers during legit reviewing extremely misserable
  • We use fake questions for audits This would make it harder to exploit but if the search enginge doesn't find the question it probably is an audit so still exploitable. Creating such audit might be more difficult than the automated proccess it currently is so we might have scaling issues with this as well.
  • We could keep track if a user visits a question currently locked for them in the queue. This could be an option with a price for the hardware and resources needed but still not failsafe. With a second account, second IP still an exploit is possible. And legit users that want to take extra action on a question (voting, editing, answering, commenting, delete voting etc) are directly impacted by this.

What really remains is the question if the audit system was and is designed to be non exploitable? And the answer to that is no. There are users and by defintion that is not you nor me, nor all users that participated in this question, that don't care about audits and simple click looks good / no action needed. Mindless reviewers. For those users the audit system does its job. They are stopped and now and then one is brave (or not smart) enough to come to meta and complain about a failed audit. And now and then a high-rep, involved user gets tricked by an audit.

As long as reviewing doesn't give you more than a few badges it is not a big deal. As soon as reputation can be gained this way we are in a different ball-park.

And to be clear: We need people who want to review and moderate those queues. If you make the life miserable of the sheer amount of users that take on this task day after day because you implement features to catch a few robots that get a badge by cheating I would say development resources are focused on the wrong group.

Related posts:
- Easy to bypass “Are You paying Attention” test by viewing question in detail
- Bring a “human factor” into review audit composition/selection
- Are the suggested edit audits too easy?
- Showing votes on review audit questions
- Audits bug in the filtered review queue

4
  • I am speechless. When i started reading your answer i was thinking "finally someone that fully understands my question". And then i saw your name. Out of all people, you were the only one i didnt expect. After the duplicate lock, then negatively predisposing viewers in my MSE post, your comments where i could see you totally missed the point... Dont get me wrong, i have no hard feelings. (I was on the verge on turning to the ultimate SO troll though, especially after seeing 21 upvotes on the offtopic answer; reminded me of youtube chat and user quality). Probably will accept it as answer.
    – user
    Commented Mar 11, 2015 at 22:39
  • I think this answer is eligible for the Community.
    – unbindall
    Commented Mar 16, 2015 at 16:48
  • 1
    I prefer the original wording... just saying.
    – user4639281
    Commented Aug 15, 2018 at 17:10
  • 1
    Forget about visiting the question. There's literally a (correctly set) isAudit property in the websocket that pulls the audits from the server... Commented Aug 15, 2018 at 17:47
29

Congratulations! You passed the audit.

No, seriously. The point of the audits is to make sure people are paying attention in review. The probable spam warning is the sort of thing that might cause some people to turn off their brains and blindly spam-flag stuff. The system grabbed a known-good question as an audit, and it having the warning specifically made sure you were doing what the warning says; reviewing carefully.

15
  • 2
    The problem with this audit, is that a bot can pass it as easily, making it ineffective.
    – user
    Commented Mar 9, 2015 at 13:15
  • 9
    Eh, the same is true for basically every audit. They're really just there to stop people who're clicking blindly without even looking. If you're paying the tiniest bit of attention, you should pass most any audit. Commented Mar 9, 2015 at 13:17
  • 5
    Not sure why this answer gets downvotes but it is the correct answer, maybe not the one you like or want it to be ....
    – rene
    Commented Mar 9, 2015 at 13:44
  • @rene This answer is based on opinion, not facts. If references prove that opinion to be a fact, then it will be a correct answer. The answer suggests that this warning is targeted at human users that don't click the link to the actual post. Until then, my point that presented audit can be passed by a review bot or anyone that clicks that link, still stands.
    – user
    Commented Mar 9, 2015 at 14:31
  • Still feel it is backed by this fact but we'll see what happens
    – rene
    Commented Mar 9, 2015 at 14:44
  • @rene This is completely irrelevant. Even without the "Our system.." message, the link in the presented audit makes it totally ineffective.
    – user
    Commented Mar 9, 2015 at 15:00
  • @user, you realize the audits are created and graded automatically, right? Of course they can be passed automatically, given the right information. Commented Mar 9, 2015 at 15:22
  • 9
    @user5061 This answer is perfectly correct and these are the same sentiments we've been passing along to users for a couple years now since the audits first started being used. They exist to stop mindless clicking, nothing else.
    – animuson StaffMod
    Commented Mar 9, 2015 at 15:27
  • "Therefore, the system sometimes grabs a known-good question as an audit, and adds the warning specifically to make sure you're doing what the warning says" - I'm not sure this is correct. I think this warning only appears when there is some indication in the system that there's a higher possibility of a question / answer being spam (a history of spam from the poster's IP, for example). I don't think the audit system artificially injects this on good posts, just that occasionally a good post trips enough of the heuristics to pop up that warning.
    – Brad Larson Mod
    Commented Mar 9, 2015 at 15:32
  • @animuson I often click the "link" to either upvote or downvote before i review. If reviewers generally do not click the link when they see the message, then the link doesnt render the audit ineffective and this answer is correct. Is that the case? If not, i ll never fail an audit ever again.
    – user
    Commented Mar 9, 2015 at 15:35
  • @BradLarson is it possible that the post has a spam flag as explained here by ChrisF?
    – rene
    Commented Mar 9, 2015 at 15:37
  • @rene - Not in this case, but my example of a location with a large number of spammers may be the case here.
    – Brad Larson Mod
    Commented Mar 9, 2015 at 15:39
  • 1
    @BradLarson: I'm sure there is a post about introducing possible-SPAM review-audits somewhere... because (at the beginning?) there were no such audits. I might be mistaken about the resolution though... Commented Mar 9, 2015 at 16:11
  • 1
    @Deduplicator It was probably this post which you, and I actually, were thinking of. While it does indeed say this, Servy is the one who says this and I have not found an employee agreeing with this. On MSE, this post does at least suggest that this is the case.
    – Kendra
    Commented Mar 9, 2015 at 16:29
  • @Deduplicator - I think Tim's answer here: meta.stackexchange.com/a/216787/135615 is expressing that the "this may be spam" warning and audits come from different places.
    – Brad Larson Mod
    Commented Mar 9, 2015 at 20:01
1

The fact that you even thought to click on the link to look at the question strongly suggests that you were paying attention and not just "robo-reviewing," so it seems like the audit worked. It was checking to see if you were paying attention, and you clearly were.

Incidentally, you can click on the title to view the question (not just the link). Sometimes I'll do that for questions I'm not sure about (or ones that look like they may be an audit).

Not the answer you're looking for? Browse other questions tagged .