Skip to main content
replaced http://stackoverflow.com/ with https://stackoverflow.com/
Source Link

Whether the right answer in these cases is also the most upvoted is also quite debatable (for example, let's just look at this questionthis question, the most upvoted answer of which recommends a solution that indeed gets rid of the error message... by introducing a security vulnerability, but hey "error goes away" -> "upvote").

To provide a few examples to contradict this paper, here are a few answers I've written that I think required a bit more in-depth knowledge to answer: Properly closing SSLSocketProperly closing SSLSocket, In an SSL handshake, is it possible to have reversed roles?In an SSL handshake, is it possible to have reversed roles?, How to find current truststore on disk programatically?How to find current truststore on disk programatically?, How are SSL certificate server names resolved/Can I add alternative names using keytool?How are SSL certificate server names resolved/Can I add alternative names using keytool? (it's hard to judge how much expertise they required, but at least the first one required a bit more thought). They have few competing answers, if any at all (although some of these other answers were indeed written by people who know what they're talking about, which is a good thing - I'm generally quite happy to upvote other answers on the same question when appropriate).

In contrast, this is currently my highest scoring answerthis is currently my highest scoring answer, which is in 3rd position out of 9, so should provide a reasonably good MEC, yet it only took 10 minutes to write (admittedly having some notion of where to look in the official Java documentation, but also with quite a bit of help from a Google search with the right keywords in the question at the time). (Of course, this particular also illustrates the shortcoming of the existing reputation problem, but we all knew about that already.) This other answerThis other answer was also, to my surprise, my top-voted answer for months at the beginning. Yet it requires very little Python expertise. I just happened to notice it in the question list at the time, by chance.

Whether the right answer in these cases is also the most upvoted is also quite debatable (for example, let's just look at this question, the most upvoted answer of which recommends a solution that indeed gets rid of the error message... by introducing a security vulnerability, but hey "error goes away" -> "upvote").

To provide a few examples to contradict this paper, here are a few answers I've written that I think required a bit more in-depth knowledge to answer: Properly closing SSLSocket, In an SSL handshake, is it possible to have reversed roles?, How to find current truststore on disk programatically?, How are SSL certificate server names resolved/Can I add alternative names using keytool? (it's hard to judge how much expertise they required, but at least the first one required a bit more thought). They have few competing answers, if any at all (although some of these other answers were indeed written by people who know what they're talking about, which is a good thing - I'm generally quite happy to upvote other answers on the same question when appropriate).

In contrast, this is currently my highest scoring answer, which is in 3rd position out of 9, so should provide a reasonably good MEC, yet it only took 10 minutes to write (admittedly having some notion of where to look in the official Java documentation, but also with quite a bit of help from a Google search with the right keywords in the question at the time). (Of course, this particular also illustrates the shortcoming of the existing reputation problem, but we all knew about that already.) This other answer was also, to my surprise, my top-voted answer for months at the beginning. Yet it requires very little Python expertise. I just happened to notice it in the question list at the time, by chance.

Whether the right answer in these cases is also the most upvoted is also quite debatable (for example, let's just look at this question, the most upvoted answer of which recommends a solution that indeed gets rid of the error message... by introducing a security vulnerability, but hey "error goes away" -> "upvote").

To provide a few examples to contradict this paper, here are a few answers I've written that I think required a bit more in-depth knowledge to answer: Properly closing SSLSocket, In an SSL handshake, is it possible to have reversed roles?, How to find current truststore on disk programatically?, How are SSL certificate server names resolved/Can I add alternative names using keytool? (it's hard to judge how much expertise they required, but at least the first one required a bit more thought). They have few competing answers, if any at all (although some of these other answers were indeed written by people who know what they're talking about, which is a good thing - I'm generally quite happy to upvote other answers on the same question when appropriate).

In contrast, this is currently my highest scoring answer, which is in 3rd position out of 9, so should provide a reasonably good MEC, yet it only took 10 minutes to write (admittedly having some notion of where to look in the official Java documentation, but also with quite a bit of help from a Google search with the right keywords in the question at the time). (Of course, this particular also illustrates the shortcoming of the existing reputation problem, but we all knew about that already.) This other answer was also, to my surprise, my top-voted answer for months at the beginning. Yet it requires very little Python expertise. I just happened to notice it in the question list at the time, by chance.

Source Link
Bruno
  • 122.3k
  • 3
  • 32
  • 38

I think most of us who've been on SO for a while would acknowledge that an answer's score doesn't always reflect "true merit", so there are indeed issues with the current reputation model.

However, this paper's assumption to try to correct the flaws of the SO reputation model are plainly wrong. Here is, I think, the quote that underpins the reasoning of its authors, yet they seem to show a misunderstanding of how SO works:

Intuitively, difficult questions generate a lot of discussions, and several answers [...]

I think it falls apart from then onwards. The SO model does not encourage questions that have a large number of competing answers (which they call "debatableness"). Competing answers are welcome, yes, but above a certain number (let's just say 5 or 6), this also reflects an equally large number of different opinions on that question, which in turn is often the symptom of a question that is too broad or subjective, i.e. it doesn't adhere to he guidelines at all.

I'm not saying that all questions with 5 or more answers are necessarily bad (there might be 5 different ways of expressing facts related to the question), but the few questions I've seen with such a large number of answers are not always the most directly answerable (nor do they necessarily attract quality).

Whether the right answer in these cases is also the most upvoted is also quite debatable (for example, let's just look at this question, the most upvoted answer of which recommends a solution that indeed gets rid of the error message... by introducing a security vulnerability, but hey "error goes away" -> "upvote").

Another typical case of questions with a large number of answers I've occasionally seen are trivial questions asked by newcomers, where a number of concurrent answers are provided within minutes.

More simply, if one assumes that experts are, by nature, rarer than non-experts in a given field, you would generally tend to see fewer answers by experts on a given topic. What this paper seems to expect, and what's thankfully often missing in practice, is the large set of accompanying bad answers on the same question, only there to make the expert look good. Instead of having "Intuitively, difficult questions generate a lot of discussions, and several answers [...]", it tends to be "Intuitively, difficult questions generate fewer answers because fewer people may know how to answer them".

In addition, once someone sufficiently knowledgeable in the field has provided an answer, there is no reason for others to answer and repeat similar things. This clearly reduces the answer count too.


To provide a few examples to contradict this paper, here are a few answers I've written that I think required a bit more in-depth knowledge to answer: Properly closing SSLSocket, In an SSL handshake, is it possible to have reversed roles?, How to find current truststore on disk programatically?, How are SSL certificate server names resolved/Can I add alternative names using keytool? (it's hard to judge how much expertise they required, but at least the first one required a bit more thought). They have few competing answers, if any at all (although some of these other answers were indeed written by people who know what they're talking about, which is a good thing - I'm generally quite happy to upvote other answers on the same question when appropriate).

In contrast, this is currently my highest scoring answer, which is in 3rd position out of 9, so should provide a reasonably good MEC, yet it only took 10 minutes to write (admittedly having some notion of where to look in the official Java documentation, but also with quite a bit of help from a Google search with the right keywords in the question at the time). (Of course, this particular also illustrates the shortcoming of the existing reputation problem, but we all knew about that already.) This other answer was also, to my surprise, my top-voted answer for months at the beginning. Yet it requires very little Python expertise. I just happened to notice it in the question list at the time, by chance.


Another problem with this paper is that I'm not sure it makes sense to assess tags separately, in particular language tags (, , , ...). Being an expert in a language only makes sense with a limited scope. In my opinion, people tend to be experts in a language for a particular purpose. There is some overlap, of course, but there are few "pure" language question. Questions tagged with a language are often associated with a library or part of an API. This is particularly true for C# or Java which are effectively more than languages, and encompass runtime environments with a public API. People who answer on Java+Desktop (e.g. and ) might not be the same as those who answer on Java+Server (e.g. and ). I would suspect the answering patterns may differ when you start taking into account tags together.

Overall, this paper presents interesting ideas, but they don't seem applicable to the SO model unfortunately.