21

I've been thinking about this quite a bit, and it seems like that there are multiple easy ways to automatically detect whether or not a review is an audit (mismatching timestamps when you click the question link, mismatching vote counts, etc). It seems like if someone cared enough you could easily write a userscript that can detect whether or not a review is an audit, and even detect what the correct action to take would be (in most cases).

Are there measures in place to prevent this and if not is using such a script considered severe abuse of the system? Because this looks like a massive oversight in the auditing process.

To expand on this: It would only take one person to write such a script and share it around and immediately, people could bypass the audit process entirely and continue robo-reviewing their way through the queues.

Edit: User Normal Human has provided such a script, for the close vote queue. Will be interesting to find out whether or not Mods find it objectionable that this is out there on stackapps.

10
  • 2
    "mismatching *X* when you click the question link" isn't that easy to implement in a userscript. For humans, those audits are easy to recognize, but for a computer? Not so much. I'd say, give it a try, then share your success with SO's employees / mods, so it can be fixed. (Assuming there is something to fix)
    – Cerbrus
    Nov 23, 2015 at 8:09
  • @Cerbrus Userscripts are not really within my expertise. The point isn't even that it has to be a userscript. A browser extension or any similar programming thing works just as well.
    – Magisch
    Nov 23, 2015 at 8:11
  • Assuming you can reliably get good positives. I think it's harder than you may think.
    – Cerbrus
    Nov 23, 2015 at 8:15
  • @Cerbrus Assume we can compare the timestamps and relevant data (votes/etc) to the direct link of the question/answer then there would be guaranteed success because only in audits do both or either of these mismatch grossly.
    – Magisch
    Nov 23, 2015 at 8:16
  • But they don't always mismatch. It's exceptions like that that'll get ya. I'm just saying: If it's that easy to circumvent, why hasn't anyone written a script to do that yet?
    – Cerbrus
    Nov 23, 2015 at 8:22
  • 1
    There were a field in the JSON indicating whether it's an audit or not, but it was at least a year or two ago when I read about it. I did write my own for LQ queue and Close queue by loading the actual page and check for the status of the question/answer, but I haven't used it much, since reviewing Close vote is a headache, and LQ audits are easy to detect.
    – nhahtdh
    Nov 23, 2015 at 8:23
  • @Cerbrus Give me an example of an audit where there is no timestamp or vote mismatch. Because I haven't seen one and I don't think one even exists, since to qualify as an audit the action in question needs to already have been taken on the post itself. Also alot of people probably already have, using various methods.
    – Magisch
    Nov 23, 2015 at 8:24
  • Yea, I'm not going to dig through audits for that.
    – Cerbrus
    Nov 23, 2015 at 8:25
  • @Cerbrus How could an Audit not have mismatching timestamps or votes? The action outlined in the audit needs to per necessity have taken place already, creating a mismatch in at least one of a few easily detectable criteria.
    – Magisch
    Nov 23, 2015 at 8:26
  • It's been a known issue for a long time. isAudit is the easy way to detect all audits, and it still works that way. Nov 24, 2015 at 22:57

2 Answers 2

30

Let's try it out! Here is Review Audit Detector, source on GitHub.

If you never hear from me again, then this was probably "illegal".

10
  • 2
    You dont see a problem with something automating away audits? If thats legal and not an issue and someone could write and distribute such a script, would that not make audits entirely pointless?
    – Magisch
    Nov 23, 2015 at 8:30
  • 1
    They are, for people using your script. If your script fetched all audits, these people could go on to robo-review entirely, completly evading the intended effect of audits.
    – Magisch
    Nov 23, 2015 at 8:32
  • Don't forget that (afaik), a large amount of incorrect reviews also raises a flag.
    – Cerbrus
    Nov 23, 2015 at 8:34
  • 2
    @Cerbrus This works only as long as a critical mass of people are still doing correct, non-robo reviews, which is something audits were designed to achieve.
    – Magisch
    Nov 23, 2015 at 8:40
  • Will be interesting to hear wether or not this is actually allowed. Its clearly not in the best interest of SO itself, but if mods actually forbid it is another question.
    – Magisch
    Nov 24, 2015 at 8:55
  • @Magisch: The (implementation of) audits is why I don't do reviews anymore. I've posted a meta answer with a bit of userscript to hide the review link, for others who share my view on the review system. What the audit proponents fail to understand is that reviewing is volunteering. The bar for not volunteering is obviously very low. How high do you raise the bar for volunteering? That's why I don't think this script matters much - it's too much work when it's easier not to review at all.
    – MSalters
    Nov 24, 2015 at 10:48
  • @MSalters I partially agree, and it stings a little to eat a review ban after 2 failed audits (both of which I think were bs but oh well), but consider the alternative: Do you want reviewers who don't review at all but just blindly click stuff for badges? Some control is necessary.
    – Magisch
    Nov 24, 2015 at 10:51
  • 4
    @5gon12eder You forgot the relevant link: xkcd.com/810 Nov 25, 2015 at 9:47
  • 1
    I wrote something like this a while back just to see if I could. As far as I'm concerned, if you put information out there for me to find, you can't get mad at me for finding (or using) that information.
    – user4639281
    Nov 26, 2015 at 19:51
  • 2
    His account has been deleted, so its probably illegal
    – Ferrybig
    Aug 22, 2018 at 8:42
5

If this issue really does become a problem, then StackOverflow can just create audit questions/answers that are artificial examples of good questions/answers (not just artificial examples of bad questions/answers).

In fact, creating such samples would be a way to check whether or not such a tool was being used in the first place.

1
  • 6
    And lets face it - if someone's robo-reviewing, they probably are.
    – Sobrique
    Nov 24, 2015 at 13:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .