11

I think that the current review system offers an awesome opportunity to learn about how people do in their reviews via crowd sourcing. For example, I'd like to be able to see how often my reviews disagree with those of others:

I was inspired by looking at this review task

In case it gets deleted, here is the question that someone reviewed as "Looks OK" (it's in nested quotes because the original was in a single quote)

i was developing javaSE application using netbeans, and as we know when we design the frame using netbeans and, once we double click the button, it will generate action performed for me bt i want to develop my application under mvc pattern. i can setup the controller and model in different classes bt that is time consuming.is there any framework which generate my SE application under mvc pattern? any help would be appreciate..

Why would someone review this question as "Looks OK"? But that's besides the point. The point is: It should not be that difficult to add a query to look back on our reviews and see "15% of your Looks OK reviews matched the crowd. Maybe you need to be less forgiving?" or "only 71% of your reviews as "close" actually got closed by the community. Maybe you need to leave open more?"

This is related to the honeypot idea but we could be using these stats as a feedback statistic, displayed right in the review queue itself, to try to train people to be better reviewers.

Obviously, this problem would not help with robo reviewers; this problem is for people who are genuinely trying to do the right thing but just need more training.

This is not a duplicate of Show cases when my review differs from the community consensus because this is more about showing statistics when actually in the review queue, not about showing individual disputed reviews when looking at the review history.

14
  • I feel like someone developed a stack app that was like this.
    – ryanyuyu
    Commented Jul 20, 2015 at 16:20
  • 2
    There is a userscript here that does partly what you want from yours truly ... and I recall answered this before on MSO so let me find the dupe.
    – rene
    Commented Jul 20, 2015 at 16:20
  • @NathanOliver Although that dupe is what I had in mind, this post has a slight twist to it because it doesn't focus on individual reviews but more an overall statistic, similar with declined/disputed flags.
    – rene
    Commented Jul 20, 2015 at 16:27
  • Related: meta.stackoverflow.com/questions/294705/… Commented Jul 20, 2015 at 16:36
  • @rene That should take care of that. Should have read it a little more closely to see about getting actual statistics. Commented Jul 20, 2015 at 16:37
  • 5
    I'm wondering if there wouldn't be a significant skew in some cases that could possibly promote undesirable behaviour. If you're reviewing appropriately but you happen to review an amount of posts that the robo-reviewers get their "Let's say everything Looks OK/Could be improved" hands on - you might well look at the stats and conclude you're applying far too much scrutiny - while, it'd be better quality wise if the others started applying more scrutiny (instead of you helping let more rubbish through) - plus - they're the ones less likely to bother checking such information... Commented Jul 20, 2015 at 16:38
  • @JonClements (and others) I feel like this could definitely be wrong in the short term, maybe only start showing statistics after a minimum sample of say, 200 reviews.
    – durron597
    Commented Jul 20, 2015 at 16:57
  • " I'm going to start an overhaul of the review history page soon. It was something we wanted to do as part of the quality project, when we introduced the helper queue - it just turned out to be complex enough to become its own larger project. People need to be able to see "That thing you spent time editing? Yeah, look how well it went on to do a month later", and it's also very useful to see how your review actions pair with the ultimate fate of the post."Tim Post
    – Braiam
    Commented Jul 20, 2015 at 17:04
  • @Braiam When was that, 6 to 8 weeks ago?
    – durron597
    Commented Jul 20, 2015 at 17:06
  • 4
    " "15% of your Looks OK reviews matched the crowd. Maybe you need to be less forgiving?" or "only 71% of your reviews as "close" actually got closed by the community. Maybe you need to leave open more?" " - This sounds very dangerous because it would lead to groupthink/mob mentality.
    – TylerH
    Commented Jul 20, 2015 at 18:37
  • @TylerH The wisdom of the crowd
    – durron597
    Commented Jul 20, 2015 at 18:46
  • 1
    @durron597 Whoever came up with that sure was a glass-half-full kind of guy. Or they work in PR. I don't want a machine to tell me how to act.
    – TylerH
    Commented Jul 20, 2015 at 18:48
  • 1
    I had the same wish, so I clicked around in my review history to see the outcomes of many reviews. I noticed that I was far more strict about declining insignificant typo edits, those were always my reject vs 3 approves. Having seen that I still don't know anything about me being right or wrong, all I know is that people tend to approve those regardless of the state of question in general. Its a useless statistic to me.
    – Gimby
    Commented Jul 21, 2015 at 7:12
  • I actually wrote a scraper that retrieves my reviews. I thought about generating some statistics on top of it, but haven't come around to completing it, because I noticed that the result may be skewed as Jon said. There is no authority that can say which reviews where good and which not. What I could do is for example assume that my reviews have remarkable accuracy. So I could determine those reviewers that deviate the most from my reviews (there are many recurring reviewers considering thousands of reviews) and investigate and talk to them about their possibly bad reviews.
    – Artjom B.
    Commented Jul 21, 2015 at 16:14

0

You must log in to answer this question.

Browse other questions tagged .