Skip to main content
replaced http://stackoverflow.com/ with https://stackoverflow.com/
Source Link

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the low S/N ratio and the users willing to upvote anythingusers willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the low S/N ratio and the users willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the low S/N ratio and the users willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

added 57 characters in body
Source Link
Jim Garrison
  • 86.7k
  • 2
  • 30
  • 39

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the low S/N ratio and the users willing to upvote anythingusers willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the S/N ratio and the users willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

Has anyone produced any hard data showing that the current audit system is actually improving review quality?

I'd love to see data about number of audit failures that were actually merited vs failures on bad audits. Since most audit disputes posted on Meta get mercilessly downvoted and closed, there appears to be a strong consensus that audits are generally not open to appeal. This makes any actual evaluation of audit effectiveness impossible since bad audits are not tracked.

I take the review queues seriously and try to tackle them regularly. I occasionally fail audits because really bad questions get upvoted and fed into the audit system. I have learned that neither appealing on Meta nor flagging the question are seen as positive actions, leaving no recourse.

Consider that when reading a question outside the review queues I am free to downvote and VTC if I feel it is appropriate, regardless of the number of upvotes. Taking exactly the same action in a review task can get me temporarily banned with a nasty message (everybody says "be nice" but the audit failure messages are deliberately trying to elicit a shame reaction). This is rather inconsistent and demeaning to exactly the set of users you don't want to alienate.

When failing 2 bad audits in 2 months results in being banned from doing ANY review (even approving edits) I feel strongly demotivated, and I know many others have the same reaction.

SO, is there evidence that audits, including the bad ones, improve things? My take is that, given the low S/N ratio and the users willing to upvote anything, any good done by well chosen audits is outweighed by the harm caused by banning users based on bad audits.

edited tags
Link
Braiam
  • 4.5k
  • 13
  • 154
  • 265
Source Link
Jim Garrison
  • 86.7k
  • 2
  • 30
  • 39
Loading