60

Tim Post pointed me to a fascinating paper: "Sparrows and Owls: Characterisation of Expert Behaviour in StackOverflow" (free version). It immediately caught my attention because (like foxes and hedgehogs) it placed a clear picture in my mind even before I'd read the descriptions:

  • Sparrows answer a lot of questions in order to collect a lot of reputation.

  • Owls answer a few difficult or important questions in order to increase the total sum of knowledge.

It's important to note going in that the paper does not assign these behaviors to completely separate species, but acknowledges overlap between the groups. The authors define sparrows as the top decile of active users and owls as users who score 1 or better on their Mean Expertise Contribution (MEC) metric. They calculate that ~30% of sparrows are also owls by this definition. Since roughly 10% of the total set of users in the studied tag () are owls, that's substantially greater overlap than the authors suggest.

The MEC calculation averages the product of inverse of answer rank by the relative "debatableness" of the question answered. I've attempted to duplicate the calculation on our public data. The inverse of rank is an interesting measure since the first answer is twice as weighted as the second. The further down you go, the less useful an answer is. Adding a 10th answer is really not worth very much. Actually, in my experience, after the first three or so answers, I tend to stop reading unless I'm incredibly engaged in a question. Answers after the 15th (where the first page of answers end) are unlikely to be ranked correctly (or even read).

"Debatableness" is directly proportional to the number of answers a question has. Unless I misunderstand the description of the equation, that more or less negates the inverse rank penalty. If the average number of answers in a tag is 2 (about what the tag currently gets), a solo answer is worth 1/1*1/2 and the 10th answer is worth 1/10*10/2 = 1/2. (With 10 other answers, the top answer would then be worth 5.) Similarly, adding a second answer to a question is seen as equally valuable as the first, since the value of the second answer is... 1/2.

An examination of my own MEC in several of my tags shows the metric is somewhat arbitrary:

Tag   MEC
---   ---
unix  1.66
shell 1.52
ksh   1.13
bash  1.03
perl  0.96
regex 0.89

There was no difference in my approach in these tags; I answered these questions with the same motivations. The difference largely comes from the number of other answers to the questions I answered.

Summary

The idea of a "sparrow" versus "owl" approach to answering questions is arresting and plausible. But weighting by number of competing answers doesn't do a good job of distinguishing between experts and non-experts. Questions with many answers tend to be popular and relatively easy to answer. Having the top answer to such a question does imply expertise. Having the 10th, does not.


The paper was intriguing and with luck the authors will refine their methods. I've sent them an email linking to this post so they (or anyone else interested) can respond in the answers below. In particular, corrections are welcome if this calculation doesn't match the paper's equation. I hope they will take this criticism as complementary; the paper is flawed in interesting ways.

18
  • I haven't read the paper, but is there a typo here in your question "the first answer is worth 1/1*1/2 and the 10th answer is worth 1/10*10/2 = 1/2"? 1/1*1/2 is also 1/2, isn't it? How would that be worth as much as the first answer?
    – Bruno
    Commented Jul 9, 2014 at 16:35
  • 3
    @Bruno: You read that correctly. One of the reasons I posted here and emailed the authors is that I'm not sure I read the paper correctly. The way I calculated things seems wrong, but I think matches the equation they used.
    – Jon Ericson Staff
    Commented Jul 9, 2014 at 16:43
  • @Bruno: I clarified that section a bit. The top answer becomes worth more as each additional competing answer is added by this system. Which is odd.
    – Jon Ericson Staff
    Commented Jul 9, 2014 at 17:18
  • 3
    Oh I see, essentially, if you're the only expert to answer (or the first expert to answer sufficiently well that no one feels the need to add a competing answer), you're basically worth as much a anyone else being the sole answerer on that question. That looks like a normalisation effort gone wrong. (It's possibly biased towards "Necromancers": you're the expert, you wait for everyone to poor in with mediocre answers, and then you add your own a couple of month later, so good it's upvoted on top... Not the SO I know...)
    – Bruno
    Commented Jul 9, 2014 at 17:23
  • For sake of conveniance I've made an altered version of Jon's query so it only returns your stats, here's the query.
    – Sam
    Commented Jul 9, 2014 at 18:09
  • I think it is a plausible model. I started off as a FGITW (sparrow) and then began adding a greater proportion of detailed owl like answers later on. The criteria they use to identify "expert" questions doesn't ring true to me though. Looks more like it will find bike shed questions. Commented Jul 9, 2014 at 19:28
  • 1
    The link in footnote 3 should have been stackoverflow.com/q/2147572 Commented Jul 9, 2014 at 21:12
  • 3
    I don't think the MEC metric is accounting very well for the difficulty of the questions being answered. A quick look at the top of the iOS rankings shows high MEC occurring when a user answers relatively few questions, but has a very large number of up-votes on one or two answers. That's basically working as designed, I think, but some of those very highly up voted answers are for rather easy questions---any "sparrow" could have answered. They just happen to be questions that lots of beginners run in to when they're starting out. Commented Jul 9, 2014 at 22:12
  • Their example question is terrible for a demonstration of their "debatableness" criterion; it's a straight-up list question, where many answers are to be expected. The difference in scores between the answers there is notable, but the "owl's" answer is far from complete (even by its own admission), and there are two other answers: stackoverflow.com/a/2149022 and especially stackoverflow.com/a/2148246 (with its links to further reading) that appear to be more thorough. I'm very skeptical of this categorization.
    – jscs
    Commented Jul 9, 2014 at 22:14
  • 12
    Please stop pigeon-holing SO users. It is demeaning and unproductive. Commented Jul 9, 2014 at 23:51
  • 18
    @HansPassant, no pigeons in that paper, just owls and sparrows :-)
    – Bruno
    Commented Jul 10, 2014 at 0:02
  • 4
    Pigeons pick up the stuff the sparrows won't touch. Commented Jul 10, 2014 at 0:57
  • 5
    Having chosen C# to develop their hypotheses, maybe they should have tested them on PHP, Java, C++, and a few other suitable tags, to see how things stack up. Commented Jul 11, 2014 at 1:09
  • 2
    According to this metric the most accomplished Haskell expert we have (MEC of 4.3) hasn't actually written any answers about Haskell.
    – sth
    Commented Jul 11, 2014 at 16:02
  • 2
    The link to the paper is broken, here's a link to the archived paper: Archived PDF - Sparrows and Owls: Characterisation of Expert Behaviour in StackOverflow Commented Mar 26, 2015 at 12:49

3 Answers 3

49

I think the idea of debatableness of a question is flawed to begin with.

Definition: "The debatableness of a question, measured according to the number of answers it generated."

Intuitively, difficult questions generate a lot of discussions, and several answers.

We know that this isn't true. In fact, I think the opposite is true. Difficult question often generate no answers whatsoever. The questions that tend to generate a lot of answers are usually one of

  • bike shed questions that everyone has an opinion on
  • easy questions that anyone can answer
  • "list of x" questions

We try to close bike shed and "list of x" questions, but C# to C++ 'Gotchas' seems to fall into both categories, and it was used as an example of a good question to identify owls. I think this question gave an unfair boost in MEC to the user with the top answer, which was good, but not significantly better than at least two other answers. (And if you take away that one answer, their score would drop down to nothing.)

9
  • 12
    I knew there was something wrong with the premise, but couldn't quite put my finger on it. I didn't read the paper, but if the math is correct, the MEC calculation appears to have no correlation whatsoever with "level of expertise" or answer quality.
    – Robert Harvey Mod
    Commented Jul 9, 2014 at 23:48
  • Difficult questions tend to generate a lot of followers of the question; perhaps number-of-followers could be used as an indication of what's a difficult question or what's an important question.
    – M.M
    Commented Jul 10, 2014 at 1:23
  • 1
    @MattMcNabb define "followers". Do you mean favorites/stars? People star/favorite questions for all sorts of reasons, not necessarily because they're difficult. You seem to be arguing that "difficult" questions are somehow popular, but in fact popularity does not imply difficulty, and vice-versa.
    – user456814
    Commented Jul 10, 2014 at 2:20
  • @Cupcake Yes, I meant favourites/stars. I guess the term should be "interesting" or "important". I'm assuming that by "difficult questions" here we are excluding cases such as someone posting ten pages of horrible code asking "why does this crash". Difficult and time-consuming to write a good answer, but not very interesting :)
    – M.M
    Commented Jul 10, 2014 at 2:25
  • 4
    @RobertHarvey I only spot checked a few users, and I can see that the calculation definitely seems to be over-penalizing answering a lot of questions. Eric Lippert will frequently swoop in and answer a seemingly normal question with an incredibly detailed answer, and he only has an MEC of about 1.8 in C#. Another user with one good answer has an MEC over 4.
    – Bill the Lizard Mod
    Commented Jul 10, 2014 at 3:04
  • @MattMcNabb I think you have to combine favorites with other parameters in order to draw a conclusion. For example, a question that gets answered immediately 30 times might get a lot of stars, but it's probably still not a great question. Another question that spends months on the unanswered list and gets a lot of stars is probably a difficult, high-quality question.
    – Bill the Lizard Mod
    Commented Jul 10, 2014 at 3:06
  • Shog seems to think otherwise: "Hard, subtle or thought-provoking questions: Everyone has an answer here too, and most of them are demonstrably wrong. Answer scores vary wildly between top and average..." (he also lists bikeshed and trivial kinds of questions as having multiple answers, but seems to think these are not much of a problem)
    – gnat
    Commented Jul 10, 2014 at 6:53
  • @gnat "Hard, subtle or thought-provoking questions..." That looks like three different categories. I'm only talking about difficult questions that often get no answers, or very few.
    – Bill the Lizard Mod
    Commented Jul 10, 2014 at 11:07
  • 1
    I agree with your observation on "difficult questions generate a lot of discussions, and several answers." That's not a truth, and the contra-positive teases it out (the statement and the contra-positive are equivalent): "If there are not a lot of discussions or not several answers, then the question is not difficult. The contra-positive is not a true statement, therefore the statement is not true. Determining the metric to use for "difficult" or "debatable" could be difficult itself.
    – jww
    Commented Dec 25, 2014 at 10:07
15

I think most of us who've been on SO for a while would acknowledge that an answer's score doesn't always reflect "true merit", so there are indeed issues with the current reputation model.

However, this paper's assumption to try to correct the flaws of the SO reputation model are plainly wrong. Here is, I think, the quote that underpins the reasoning of its authors, yet they seem to show a misunderstanding of how SO works:

Intuitively, difficult questions generate a lot of discussions, and several answers [...]

I think it falls apart from then onwards. The SO model does not encourage questions that have a large number of competing answers (which they call "debatableness"). Competing answers are welcome, yes, but above a certain number (let's just say 5 or 6), this also reflects an equally large number of different opinions on that question, which in turn is often the symptom of a question that is too broad or subjective, i.e. it doesn't adhere to he guidelines at all.

I'm not saying that all questions with 5 or more answers are necessarily bad (there might be 5 different ways of expressing facts related to the question), but the few questions I've seen with such a large number of answers are not always the most directly answerable (nor do they necessarily attract quality).

Whether the right answer in these cases is also the most upvoted is also quite debatable (for example, let's just look at this question, the most upvoted answer of which recommends a solution that indeed gets rid of the error message... by introducing a security vulnerability, but hey "error goes away" -> "upvote").

Another typical case of questions with a large number of answers I've occasionally seen are trivial questions asked by newcomers, where a number of concurrent answers are provided within minutes.

More simply, if one assumes that experts are, by nature, rarer than non-experts in a given field, you would generally tend to see fewer answers by experts on a given topic. What this paper seems to expect, and what's thankfully often missing in practice, is the large set of accompanying bad answers on the same question, only there to make the expert look good. Instead of having "Intuitively, difficult questions generate a lot of discussions, and several answers [...]", it tends to be "Intuitively, difficult questions generate fewer answers because fewer people may know how to answer them".

In addition, once someone sufficiently knowledgeable in the field has provided an answer, there is no reason for others to answer and repeat similar things. This clearly reduces the answer count too.


To provide a few examples to contradict this paper, here are a few answers I've written that I think required a bit more in-depth knowledge to answer: Properly closing SSLSocket, In an SSL handshake, is it possible to have reversed roles?, How to find current truststore on disk programatically?, How are SSL certificate server names resolved/Can I add alternative names using keytool? (it's hard to judge how much expertise they required, but at least the first one required a bit more thought). They have few competing answers, if any at all (although some of these other answers were indeed written by people who know what they're talking about, which is a good thing - I'm generally quite happy to upvote other answers on the same question when appropriate).

In contrast, this is currently my highest scoring answer, which is in 3rd position out of 9, so should provide a reasonably good MEC, yet it only took 10 minutes to write (admittedly having some notion of where to look in the official Java documentation, but also with quite a bit of help from a Google search with the right keywords in the question at the time). (Of course, this particular also illustrates the shortcoming of the existing reputation problem, but we all knew about that already.) This other answer was also, to my surprise, my top-voted answer for months at the beginning. Yet it requires very little Python expertise. I just happened to notice it in the question list at the time, by chance.


Another problem with this paper is that I'm not sure it makes sense to assess tags separately, in particular language tags (, , , ...). Being an expert in a language only makes sense with a limited scope. In my opinion, people tend to be experts in a language for a particular purpose. There is some overlap, of course, but there are few "pure" language question. Questions tagged with a language are often associated with a library or part of an API. This is particularly true for C# or Java which are effectively more than languages, and encompass runtime environments with a public API. People who answer on Java+Desktop (e.g. and ) might not be the same as those who answer on Java+Server (e.g. and ). I would suspect the answering patterns may differ when you start taking into account tags together.

Overall, this paper presents interesting ideas, but they don't seem applicable to the SO model unfortunately.

3
  • 3
    Even among non-broad/opinionated questions as well, a large number of answers (more than maybe five) usually indicates not "debatableness" but that there's only one way to do it and everyone has answered with that way -- i.e., what we here call a FGITW situation. Older factual questions will also often accrete duplicate answers. There may be no good way for the researchers to measure answer content duplication, but the "debatableness" measurement is, I think, lacking foundation.
    – jscs
    Commented Jul 10, 2014 at 0:17
  • @JoshCaswell, ah yes, "FGITW", that's what I meant by "trivial questions asked by newcomers". I couldn't remember the expression.
    – Bruno
    Commented Jul 10, 2014 at 2:00
  • OT: the misguided SSL answer reminded me of stackoverflow.com/questions/9079298/…. The research paper is also linked to from LWN: lwn.net/Articles/522111
    – ninjalj
    Commented Jul 28, 2014 at 11:09
1

The further down you go, the less useful an answer is. Adding a 10th answer is really not worth very much. Actually, in my experience, after the first three or so answers, I tend to stop reading unless I'm incredibly engaged in a question. Answers after the 15th (where the first page of answers end) are unlikely to be ranked correctly (or even read).

Vote count / submission time correlates with quality, but it is not a causal relationship. An example

JavaScript equivalent to printf

To me, this answer with a small edit perfectly suited my needs. However at the time it was on the second page with one vote. It is now on the first page with three votes. Granted that is not stellar, but again for my situation this is the best answer.

You should only stop with the first three answers if you were completely satisfied with one of them. Otherwise you are doing yourself a disservice.

2
  • 9
    Actually, you should not stop at the first few top-voted answers at all. It's extremely foolish to do so. Technology changes and becomes obsolete all the time. Old questions are updated with new answers all the time, but those new answers have to start at the bottom, with a score of 0, until enough people vote them to the top. So yes, look at the top voted answers...but you also need to consider whether or not any new answers are better than the old ones, even if they're not as highly upvoted yet.
    – user456814
    Commented Jul 10, 2014 at 0:24
  • 2
    @Cupcake, that's a good point. I also wish everyone was able to see the distinction between up and down votes, since it's often a good indication whether an answer may be incorrect, or at least contested. (That would also be very useful for opinions on Meta too, since by nature an answer can easily gather a number of both types of votes here.)
    – Bruno
    Commented Jul 10, 2014 at 2:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .