36

I don't remember ever getting a review test on the Security Stack Exchange, but now that I am trying to help to work down the queue here as well, they seem very common. I keep wasting time trying to figure out the intention, if maybe a translator was used, if "globalktable" refers to something else in the thread (this is where I open the question in a new tab to investigate answers and comments)...

Eventually, I give up and reject it with a custom message (people put their honest time into edits, I can at least help them understand why I think it's not a good edit), and then it goes "haha just a prank bro! Good job tho." As part of this, I also noticed that the user that supposedly submitted the audit is real (I was wondering if they had more such edits and checked that before hitting reject; that's how I found out that they never submitted this).

Hence, a few remarks/questions:

  • Since I got each of them correct so far, this makes me wonder: how often are these tasks supposed to appear?

  • I noticed it pushes me towards rejecting hard-to-understand edits without properly reading them since one can't fail audits if you just reject everything and I don't want to waste more time than necessary. Rejecting a legit edit because I didn't understand a post would be harmful, so I try not to do that.
    If there were a selectable reject reason "I think this is an audit", at least the site could tell you when it's not.

  • If the site wants to test people continuously, it could instead be comparing how frequently my decision (accept/reject) deviates from the other reviewer's decision.

  • Does the test rate depend on any variables related to my account, or is it a static percentage or chance factor for everyone?

  • How are decoy users picked? Could my name appear below a garbage edit submission?

If it is normal that they appear every couple of tasks regardless of whether you get them right, they are not worth my time and I will stop reviewing on this network site. I am hoping this was just bad luck.

This is about the suggested edits queue, in case it is different per review type.

16
  • 18
    "Since I got each of them correct so far, this makes me wonder: how often are these tasks supposed to appear?" - obnoxiously often. Welcome to the review queues :') Commented Nov 10, 2021 at 15:18
  • 3
    Somewhat related: Why aren't there any correct suggested edit audits?
    – 41686d6564
    Commented Nov 10, 2021 at 15:27
  • 7
    "since one can't fail audits if you just reject everything". So the audits can push you into robo reviewing. Something that it's supposed to prevent.
    – Suraj Rao
    Commented Nov 10, 2021 at 15:33
  • 8
    @SurajRao not so fast, there are known good audits that you fail if you reject. The only absolutely safe option is Skip. Commented Nov 10, 2021 at 15:38
  • @RobertLongson I have not seen any of those, nor seen it mentioned in other meta questions related to the audits. See also the link u/41686d6564 posted. Do you have a reference for that there are positive suggested edit audits?
    – Luc
    Commented Nov 10, 2021 at 15:40
  • @RobertLongson not according to the link above my comment.
    – Suraj Rao
    Commented Nov 10, 2021 at 15:43
  • 1
    @SurajRao There are known good tests in other queues. Commented Nov 10, 2021 at 15:59
  • 2
    Review audits are only enabled on some sites: "Stack Overflow, Server Fault, Super User, Software Engineering, Ask Ubuntu, Mathematics, Stack Overflow em Português, Stack Overflow en Español, and Stack Overflow на русском."
    – Andrew T.
    Commented Nov 10, 2021 at 16:27
  • 5
    "I noticed it pushes me towards rejecting hard-to-understand edits...I don't want to waste more time than necessary." Nothing is forcing you to make a specific decision on a review item. If you feel like it's taking you too long to understand what's going on with an edit, you should Skip. "Does the test rate depend on any variables related to my account, or is it a static percentage or chance factor for everyone?" The point of review audits is to make people slow down and pay attention so it might have something to do with how fast you review in general, but I don't know that for sure.
    – BSMP
    Commented Nov 10, 2021 at 17:28
  • @BSMP I would guess I review rather slowly compared to others based on how much the audits seem to annoy me compared to others :p, but certainly there are also obvious good edits (fix a clear typo in title, no other changes) that can be done in under 3 seconds. That could be it, yes.
    – Luc
    Commented Nov 10, 2021 at 17:30
  • 3
    Eh, I don't think I'm fast. I just find them easy to spot since they just add nonsense to (or occasionally delete important information from) a post. It will probably get easier for you to spot them too with more experience. I have noticed that they show up more frequently sometimes.
    – BSMP
    Commented Nov 10, 2021 at 17:45
  • 2
    The suggested edit queue audits are especially easy to spot with a little bit of experience. They tend to insert nonsense into the title and every paragraph of the post. At this point it takes me less than 5 seconds to realize that it's an audit and reject with "spam or vandalism." I sometimes waste time on the audit tasks in the other queues, but I find the suggested edit audits to be ludicrously clear. Commented Nov 10, 2021 at 20:17
  • They should appear that often, so that some measure combining the gains in review accuracy by reviewer paying more attention (maybe not measured) and the loss by wasting time reviewing audits is maximized. Whatever that means. Probably up to discussion. My gut feeling says that audits shouldn't be failed in more than 5-20% of cases. Commented Nov 11, 2021 at 9:00
  • 7
    I clicked the review queue once, it was really hard to parse the text I was reviewing, took me over half an hour to figure out what was going on and how to proceed with the review. Turned out it was fake and just trying to see if I would review correctly. I never clicked the review queue again, it's by far the most insulting "feature" that Stack Exchange has.
    – Kevin
    Commented Nov 11, 2021 at 9:15
  • 2
    @Kevin "...took me over half an hour to figure out what was going on..." Really? At that rate it would be difficult to run into your daily review limit. Sure, it's your right to be disappointed by the review audits but then I think it's mostly designed to catch robo reviewers (people who need only splits of seconds in order to know what's going on) not people who do proper reviewing. So maybe, if somebody takes his time, he/she really should be exempt from audits (or get less of them). Commented Nov 11, 2021 at 12:02

1 Answer 1

30

Since I got each of them correct so far, this makes me wonder: how often are these tasks supposed to appear?

There is no information about that in the MSE's FAQ post on review audits as of yet, but the bits and pieces of the algorithm are scattered across various network posts. Here is a small compilation of those that either ask for the same/similar stats or contain glimpses into how the frequency of audits is determined:

Year Post Useful Info
2021 Frequency of appearing of audits in review queues - WHEN do they appear? no official response
2015 Do failed audits affect the frequency of audits? same info as in "Stop bothering me with suggested edit review audits"
2013 How many failed audits to be put on quarantine? "the logic is supposed to give you an audit every 20 reviews or so; it looks at reviews in any queue though <...> We're tweaking the logic here: 5% chance of being audited starting 5 reviews after your last audit."
2013 Stop bothering me with suggested edit review audits "100% success rate for, say, the past 20 audits gets you at least 30 reviews before the next one, Anything under 50% rolls the dice every review." from Shog9's answer
2013 Reviews Audits: Are Repeat Failers Audited More Often? no official response
2013 Increasing frequency of review audits "random chance whether any given review will be an audit or not, but there's a much lower "no audits" period after you've been audited" from Shog9's answer

Does the test rate depend on any variables related to my account, or is it a static percentage or chance factor for everyone?

Yes and no, see the table above. It is supposed be a fixed percentage with a hard switch between 1/30 chance on 100% success rate and 1/2 under 50%.

I noticed it pushes me towards rejecting hard-to-understand edits without properly reading them since one can't fail audits if you just reject everything and I don't want to waste more time than necessary.

Do not overthink this. A rule of thumb when reviewing is: when in doubt, skip. Reviewers are not supposed to brood over review tasks for ages - if you feel like you cannot make a clear-cut decision right off the bat, skip the task and leave it to someone who can.

As for pushing you to reject edits, I have a hard time believing the system does that. I am yet to fail a single suggested edits audit without using any filters - as others mentioned, edit audits are extremely easy to recognize. The golden rule here is: if the edit looks like complete gibberish, it is most likely an audit.

If the site wants to test people continuously, it could instead be comparing how frequently my decision (accept/reject) deviates from the other reviewer's decision.

It kind of does apart from the suggested edits audits where the only correct action with audits is to reject. That said, checking for consensus in review audits is not such a great idea as it might seem at first glance. It suffers from the well-known fallacy of survivor bias.

Imagine the majority of reviewers reviews incorrectly (this is a hypothetical situation) while only the minority chooses the correct action. The latter consistently fails audits under the system that only takes into account consensus because they consistently review against it.

How are decoy users picked? Could my name appear below a garbage edit submission?

Yes, you can be picked, with a random chance of 1/<number of users on the site> (the pool is likely constrained to users who need to suggest edits, i.e. with less than 2K reputation as hinted at in animuson's answer):

an unsuspecting user is randomly selected and displayed as the proposer of the edit

As Brad Larson clarified back in 2013, the user choice is randomized per user, so there is no guarantee that any two reviewers will see the same user as the suggested edit author.

This approach caused problems before (the quote above is from Emmett's answer to this question), but it only lead to the real user being substituted with the Community user after the audit is passed/failed.

8
  • 4
    survivor's biais link is wrong, should be xkcd.com/1827
    – Pac0
    Commented Nov 11, 2021 at 0:57
  • 4
    Thanks for the elaborate answer! "Reviewers are not supposed to brood over review tasks for ages" Well sure, if there is some edit regarding an esoteric language where I need to understand whether the code change is truly good and not in conflict with the author's intent, then I'll be quick to hit that skip button. But when an edit appears to insert nonsense, I should be able to decide on it as well as anyone. Someone's gotta do it, might as well be me to do that task properly, so then it takes me a minute (could have done multiple other tasks in that time...).
    – Luc
    Commented Nov 11, 2021 at 11:23
  • 1
    @Luc NP, about time we got all this information in one place. Regarding "brooding over" - yeah, it is more about that we, as reviewers, are not really intended to spend much time on individual review items - thus "skip" is considered the best option by both mods and fellow reviewers. I can pretty much guarantee that suggested edit review audits never look like valid audits, even in esoteric technologies, so don't worry about them too much. It is usually best to start with checking if the review is utter unintelligible gibberish, and, in lieu of that, just proceeding with the normal review.
    – 0Valt
    Commented Nov 11, 2021 at 12:44
  • 1
    "As for pushing you to reject edits, I have a hard time believing the system does that." I know for me for low quality posts, I'm so terrified of failing an audit that I can no longer use Looks OK almost at all ever. the only thing I can do is delete or skip. though yeah that doesn't mean that I can always delete. Commented Nov 12, 2021 at 5:00
  • I just got audit 2 times out of 7 consecutive reviews (Triage). This is ridiculous.
    – holydragon
    Commented Nov 12, 2021 at 7:44
  • @Dave That part you quoted only refers to the suggested edits queue only. Other queues, unfortunately, can and do have difficult (and sometimes outright incorrect) audits, and are heavily skewed towards "looks ok" being never chosen, yeah
    – 0Valt
    Commented Nov 12, 2021 at 9:13
  • 1
    @holydragon "This is ridiculous." Not sure. What is your audit history? How many audits did you fail in the past? Commented Nov 12, 2021 at 11:47
  • Seconding @Trilarion, with a 5% chance one normally gets (some failed audits, some passed, > 50% passed), 2/7 does not look that ridiculous - statistically, this is not representative enough to make any conclusions. It could've just as easy been 7/7 with the same 5% chance.
    – 0Valt
    Commented Nov 12, 2021 at 12:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .