Tim Post pointed me to a fascinating paper: "Sparrows and Owls: Characterisation of Expert Behaviour in StackOverflow" (free version). It immediately caught my attention because (like foxes and hedgehogs) it placed a clear picture in my mind even before I'd read the descriptions:
Sparrows answer a lot of questions in order to collect a lot of reputation.
Owls answer a few difficult or important questions in order to increase the total sum of knowledge.
It's important to note going in that the paper does not assign these behaviors to completely separate species, but acknowledges overlap between the groups. The authors define sparrows as the top decile of active users and owls as users who score 1 or better on their Mean Expertise Contribution (MEC) metric. They calculate that ~30% of sparrows are also owls by this definition. Since roughly 10% of the total set of users in the studied tag (c#) are owls, that's substantially greater overlap than the authors suggest.
The MEC calculation averages the product of inverse of answer rank by the relative "debatableness" of the question answered. I've attempted to duplicate the calculation on our public data. The inverse of rank is an interesting measure since the first answer is twice as weighted as the second. The further down you go, the less useful an answer is. Adding a 10th answer is really not worth very much. Actually, in my experience, after the first three or so answers, I tend to stop reading unless I'm incredibly engaged in a question. Answers after the 15th (where the first page of answers end) are unlikely to be ranked correctly (or even read).
"Debatableness" is directly proportional to the number of answers a question has. Unless I misunderstand the description of the equation, that more or less negates the inverse rank penalty. If the average number of answers in a tag is 2 (about what the c# tag currently gets), a solo answer is worth 1/1*1/2
and the 10th answer is worth 1/10*10/2 = 1/2
. (With 10 other answers, the top answer would then be worth 5.) Similarly, adding a second answer to a question is seen as equally valuable as the first, since the value of the second answer is... 1/2.
An examination of my own MEC in several of my tags shows the metric is somewhat arbitrary:
Tag MEC
--- ---
unix 1.66
shell 1.52
ksh 1.13
bash 1.03
perl 0.96
regex 0.89
There was no difference in my approach in these tags; I answered these questions with the same motivations. The difference largely comes from the number of other answers to the questions I answered.
Summary
The idea of a "sparrow" versus "owl" approach to answering questions is arresting and plausible. But weighting by number of competing answers doesn't do a good job of distinguishing between experts and non-experts. Questions with many answers tend to be popular and relatively easy to answer. Having the top answer to such a question does imply expertise. Having the 10th, does not.
The paper was intriguing and with luck the authors will refine their methods. I've sent them an email linking to this post so they (or anyone else interested) can respond in the answers below. In particular, corrections are welcome if this calculation doesn't match the paper's equation. I hope they will take this criticism as complementary; the paper is flawed in interesting ways.