10

Background

For this SO reviewer test item https://stackoverflow.com/review/first-posts/14153669 (SO Question: is there anything wrong with this Singleton class)

The direct question at the very bottom was

"What are the possible problems i can face with this implementation?"


The accepted, and 12x up-voted, answer was under review as a supposed first answer.

As of this post, it provides some code while saying only:

"Simply use Meyers' singleton:" [code omitted for brevity sake]

That's all there is to the answer. It didn't address the question directly ask by the OP. It just functionally says "use this code" and provides the code. It lacks context, and explanation.

In my estimation that kind of an answer deserves a comment indicating an explanation will greatly improve its quality. So I made said comment, and promptly failed the test.

I find this troubling.

My Questions

  1. Was I expected to approve the answer under consideration because a fraction of a percent of SO users have up-voted it? (in this case 12)

  2. If not, while considering that the answer I was expected to approve, in no way gave a direct response to the direct question that was asked, please explain why that answer was supposed to be approved?

Thanks!

4
  • 2
    Related: meta.stackoverflow.com/q/260803/2675154
    – honk
    Nov 1, 2016 at 7:41
  • I thought the failing audits for making a comment thing was going to be adressed soon, but apparently not.
    – Magisch
    Nov 1, 2016 at 7:46
  • Given the rudeness of the reply to my comment about the lack of validity of their heuristic (a conflation of the word guess if I've ever seen one) I'm no longer going to contribute. Lots of SMEs have also voiced this as their course of action...
    – Jason D
    Nov 1, 2016 at 14:28
  • The members of SE seems to have little concern in addressing disputable audits and handles the discussions on them by a) ACCEPTING THE ANSWER for you. And b) being rude to you when you point out flaws in reasoning. This is not worth my time.
    – Jason D
    Nov 1, 2016 at 14:29

2 Answers 2

4

Was I expected to approve the answer under consideration because a fraction of a percent of SO users have up-voted it? (in this case 12)

Yes. The audit system using its algorithm determined that that answer was a good answer. We do not know exactly what it uses for this determination but votes and flags are most likely part of it. If we look at the answers timeline we can see that there were LQPQ reviews against it so most likely it was not flagged. Since it was also highly upvoted and had no down votes this is a good indication that it is a good answer.

Now was this a good audit, not really. It does not have a lot of content so adding a comment asking for elaboration would be a normal thing to do but unfortunetly that will cause you to fail an audit. Fortunetly bad audits are more the excpetion then the norm. They are going to happen occasionly and when they do you should bring them up so they can removed from consideration. It's not perfect but it does a pretty good job.

Also note that an ocassionl failed review is not going to hurt you. If you do get banned from review for a bad audit normally the mods will reverse it unless they see something that concerns them.

Your comment should not have failed you but unless/until Stack Overflow decides that is a good idea we are just going to have to deal with these corner cases. Do note you can always open the post in other tab/window to double check it if you do not like what you see on the review page.

2
  • The criteria for audits isn't hidden information. If memory serves it's a score of 10+, no downvotes, and no flags.
    – Servy
    Nov 3, 2016 at 13:30
  • @Servy If you could link me to where they have that it would be appreciated. I have never seen the audit criteria actually spelled out. Nov 3, 2016 at 13:33
1

1. Yes, you were. This tends to be a reasonably accurate heuristic for quality answers, but as you've seen, it is not a perfect heuristic. Every once in a while you're simply going to run across a poor quality answer that lots of people have upvoted. These will happen occasionally, but they'll tend to be rather rare. You can simply go to the post in question and downvote it and it will become ineligible for audits.

It's also worth noting that the problems with the answer are also a symptom of the problems with the question. The question is functionally just asking for a code review of something, with no specific problem mentioned nor any specific aspect of the code to be looked at in particular. Such a question is way too broad for SO, so it'd be helpful for you to toss a close vote on the question while you're there. The fact that the question is simply so broad is what has inevitably lead to the fact that no answer can realistically ever provide a complete answer.

5
  • 1
    in this case the question was direct, simple, and included all relevant source code. It was the answer I took issues with. And popularity is not a valid heuristic. Just talk to any statitician about that.
    – Jason D
    Nov 1, 2016 at 14:20
  • I do know what it is. Please don't presume you know what's in my head. The validity of a heuristic is determined by statistical reliability. Popularity is a highly unreliable statistic. So please don't be argumentative. Please, as I suggested, speak to statisticians about why this is the case.
    – Jason D
    Nov 1, 2016 at 14:24
  • The appeal to ignorance in your prior comment doesn't change the inferiority of the SEs audit selection criteria. The selection criteria ignores the verifiable accuracy (not if it's been verified, but the potential) of the review audit. Furthermore the lack of providing a dispute feature shows that SE isn't particularly concerned with much beyond popularity, their golden boy of heuristic. This is subject to much discussion because of its invalidity, inaccuracy, and perverse nature. Put simply these complaints you see about it, are a symptom of the real problem -- bad heuristics.
    – Jason D
    Nov 1, 2016 at 14:40
  • 2
    @JasonD You're once again making assertions without substantiating them. The percentage of audits that are bad is very small, combined with the fact that it takes a number of failures in a short period of time for there to actually be an effect, means that as long as the rate of bad audits is low enough, there is no problem. You've provided no evidence to the contrary.
    – Servy
    Nov 1, 2016 at 14:46
  • @JasonD If you have an issue with popularity being the heuristic of choice, perhaps your quarrel is with the concept of democracy. The system is effective in a large majority of cases here on SE. It can be very frustrating when it doesn't work properly.
    – 4castle
    Nov 3, 2016 at 2:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .