Timeline for Is there any hard evidence that audits improve review quality?
Current License: CC BY-SA 3.0
11 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Feb 10, 2016 at 8:21 | history | edited | lisa p. | CC BY-SA 3.0 |
deleted 39 characters in body
|
Feb 9, 2016 at 22:17 | comment | added | Jim Garrison | I agree that some form of audit is necessary given the robo-reviewers (i'd call them zombie reviewers myself). However, an appeal mechanism is needed. The current system is equivalent to letting a poorly implemented AI act as judge, jury and sentencer with no possibility of appeal. There's lots of science fiction horror stories about just such a scenario, and it mirrors the current ethical debate about AI in the sphere of autonomous weapons. I'm getting off topic but there's a common thread here. | |
Feb 9, 2016 at 10:26 | comment | added | lisa p. | I just have the impression that you already think the system has to be changed, which is most likely necessary, and want data to back it up. It wouldn't help you if audits turned out to actually improve overall effectiveness except for a few individuals. I can remove the answer if it doesn't resolve your problem. | |
Feb 9, 2016 at 10:05 | comment | added | Magisch | My number of failed audits dropped drastically (currently 0 after the ban), but only because I now look for audits before voting on my best judgement. It has happened several times before to me that I examined a post and then failed an audit on it, and afterwards I still would have voted the same on that post. | |
Feb 9, 2016 at 9:33 | comment | added | Jim Garrison | My question has nothing to do with one specific individual, and is not a "rant" as you seem to think. I see a deeply flawed mechanism that seems to choose audits based on upvotes alone. I believe the bad audits reduce the effectiveness of the review queues by discouraging active participation from experienced users. I want to know if the audit process is being measured to determine if it improves the overall effectiveness for the entire site. I don't believe it does. | |
Feb 9, 2016 at 9:26 | comment | added | lisa p. | @JimGarrison this already answers your question: The audits - in your case - failed their goal. | |
Feb 9, 2016 at 9:25 | comment | added | lisa p. | @BoltClock do these stats only include the number of failed audits or do they give insight about the quality of audits? How do you measure that? | |
Feb 9, 2016 at 9:18 | comment | added | Jim Garrison | "Just process tasks you are absolutely sure about" I already do that. I skip LOTS of reviews in areas I don't know. But the bad audits are a regular fixture. Do you really want to discourage experienced users from participating in reviews? OK, I'll just "review" questions as I read them in the normal flow outside the review queues. I don't need to do reviews, I just thought it was a way the system made it easier to spread the moderation task among experienced users. Maybe I misunderstood the purpose of the review queues? | |
Feb 9, 2016 at 9:08 | comment | added | Jim Garrison | The number of failed "good" audits has certainly decreased over time. However, that number is insignificant (<10) compared to the number of reviews I've completed since reviews were introduced. The number of failed bad audits is higher (10-20). I can't give accurate numbers since I'm currently banned and can't get to the review history tabs. | |
Feb 9, 2016 at 8:58 | comment | added | BoltClock Mod | There are stats all right. However these are only accessible to moderators and employees. | |
Feb 9, 2016 at 8:47 | history | answered | lisa p. | CC BY-SA 3.0 |