54

This is what the top of the list of Top openssl Answerers currently looks like: Top of the list of 'openssl' Top Users

For each user, the first column indicates the total score received for answers and the second column is the total number of answers posted for the tag in question. In this particular example, five out of the eight Top Answerers happened to answer a question that turned out to be very popular.

Although the phrase "Top Answerer" can be interpreted in different ways, I think people expect to see a list of experts for the associated tag. The current algorithm of simply counting the score does not provide that; at least not for tags with a low amount of questions being asked. Is it possible to use a somewhat more sophisticated algorithm that better reflects the expertise of answerers especially for low(er) traffic tags? And the list of askers could use a similar improvement.


One comment below suggests discussing what makes somebody a Top Answerer first, what does that mean and what does SO try to achieve with this list? Only after answering these fundamental questions it makes sense to select an associated algorithm.

That is a fair point, although I would like to argue that SO has not given a whole lot of thought to the Top Users list, otherwise its implementation would not be this flawed. A better question might be: what do SO users expect to see in such a list to provide them value, if any at all.

In my opinion, getting a feel for who are expert users for a subject does have value. As an example, I have spent quite a bit of time on the openssl tag recently because I am learning about it, and answering other people's questions is an efficient way for me to do so. After several weeks of dwelling around, I start to recognize names of high quality contributors. Consequently, I start to pay more attention to what they have to say. This is just a natural filtering process but it takes quite a while before that filter is properly initialized. A well-designed list to Top Answerers would instantly help me with that.

Summarizing, I think a Top Answerer is somebody who has consistently displayed a deep knowledge about the subject. Being a Top Answerer should be a reliable indication of that user having a good reputation.


Since there appears to be a bit of interest, I have created a few queries that show alternative lists of Top Answerers, based on different algorithms proposed by commenters:

26
  • 20
    Could you propose such a "somewhat more sophisticated algorithm"?
    – Yunnosch
    Commented Sep 12, 2018 at 5:02
  • 2
    @Yunnosch: the approach would have to mitigate the "single answer with many upvotes" effect. For example attach more value to acceptance of answers and less to upvotes. Something like score = a * nof_accepted_answers + b * nof_upvotes, where a and b would have to be chosen properly. Probably a would have to be significantly larger than b. Commented Sep 12, 2018 at 5:15
  • 6
    @Yunnosch: paradoxically, accepted answers with little upvotes might be a better indication of expertise than accepted answers with many upvotes. The latter typically indicates that the question was a very common one and therefore did not require a lot of in-depth knowledge to answer. Commented Sep 12, 2018 at 5:21
  • 20
    Accepted answers should not be used as part of a "top" answer list anywhere. Votes are what is supposed to indicate community consensus in terms of quality. The accepted answer check mark is simply the OP's opinion (and since they asked the question, it is not likely they would be an expert and a good judge of highest quality). But the idea of a single highly voted answer putting someone near the top of the "top answerers" list is probably not ideal. Commented Sep 12, 2018 at 11:18
  • 4
    Perhaps the best way would be to simply introduce a cap per answer. Of lets say 10. So that having multiple good upvoted answers will land you in the top more easily. The same should go for top askers in that tag, as all but 4 have just 1 question.
    – Luuklag
    Commented Sep 12, 2018 at 12:36
  • 1
    what about average upvotes per answer? Commented Sep 12, 2018 at 12:38
  • 5
    @WhatsThePoint, that would only benefit those with a single highly upvoted answer.
    – Luuklag
    Commented Sep 12, 2018 at 12:38
  • 4
    average upvotes probably wouldn't work. I am arguably one of the biggest experts on stackoverflow for a couple of specific tags, and I get very few upvotes mainly due to the low traffic (I assume). My average upvote is probably at best 2, and probably closer to 1. Someone could answer one question and get 4 or 5 votes who could instantly be ranked higher than me. Commented Sep 12, 2018 at 12:41
  • 1
    @Luuklag ah yeah true, massive brain fart on my behalf :D Commented Sep 12, 2018 at 12:42
  • 2
    What about sorting by number of answers with a score of 1 or more?
    – Cœur
    Commented Sep 12, 2018 at 12:45
  • That would only benefit users with many (and probably mid-quallity) answers @Cœur. I think we need some more intelligent algorythm, which takes the score and the answer count into account. Probably the same style, like we do in the "select next tag"-dialog Commented Sep 12, 2018 at 12:55
  • 1
    @psubsee2003: I do not think that votes alone are a good way to measure community consensus in terms of quality. In many cases, that is more a way to measure question popularity. Maybe "votes per view" would be a better indication, so divide the score by the number of views the question has had. Commented Sep 12, 2018 at 13:32
  • 1
    @ReinierTorenbeek Note that I said "supposed to". But yes, because of age and view differences, it is not always a perfect measure of quality when comparing relative quality of answers between 2 different questions, or of multiple answers on the same question that were posted at very different times Commented Sep 12, 2018 at 14:09
  • 2
    Just to set the cat amongst the pigeons but.... Who cares? This list of top users is specifically meant to show who gained the most rep for answers in these tags, that's all. What is the benefit is having a different algorithm? Will it change how you interact with the site or how you perceive these users? I doubt it. So, what benefit do you think there is?
    – DavidG
    Commented Sep 12, 2018 at 15:51
  • 1
    @DavidG: Who cares? Is this a real question, or did you mean to say Nobody is interested in this anyway. If you are saying the list of Top Users is basically noise on the website, then you should recommend removing the feature altogether. And then, by extension, removing any kind of ranking list on SO, no? I agree somewhat with your remark and am actually not so much interested in the ranking itself. However, lacking any better mechanisms, I think the list of Top Answerers could be useful as an indicator of a user's reputation but from a different angle, for a certain subject only. Commented Sep 12, 2018 at 16:18

3 Answers 3

20

Put users with less than 20 answers at the bottom (or don't show them at all).

Note: 20 answers is similar to the cut-off for a bronze badge.

For less than 20 answers, the ordering probably won't be particularly meaningful anyway.

The main advantage here (above more complex orderings) is that it will be fairly easy for someone looking at it to figure out how it's ordered (it will just be by score, for the most part).

Possible variant: Only show users with less than 20 answers if there aren't any users with 20 or more answers.

9
  • 1
    Good idea: Simple and effective
    – kjhughes
    Commented Sep 14, 2018 at 14:40
  • 2
    the only issue is if there aren't enough bronze badge holders. Do we not have a top 10? Commented Sep 14, 2018 at 14:48
  • 1
    I favor this answer over the other.
    – user5451396
    Commented Sep 14, 2018 at 15:19
  • 1
    @psubsee2003 Yeah, we'll need to decide what the desired behaviour is when we have 0-19 top users with 20 or more answers. Well, actually the staff will probably make their own decision about this regardless of what we decide - I didn't really want to get caught up on arguing specifics at this stage. Commented Sep 14, 2018 at 15:52
  • 1
    Thanks, I like this idea as well. How about the following variation: first order by tag badge (descending), then by vote score (descending). That would be intuitive and would not introduce any new concepts, just reusing concepts that have already been defined. Commented Sep 14, 2018 at 17:16
  • @ReinierTorenbeek Could work. Then you can also show the badge next to the user, to make it clearer how you're sorting. You might want to post a separate answer about that. Commented Sep 14, 2018 at 19:37
  • I do not agree with this, since this means, you need at least 20 answers in order to have a chance to be considerded for the highscore. This benifit only person, which are posting much answers (igonoring the quality of answers). My answer still considers persons below 20 answers, but makes it more likely, persons with an answer count > 20 are be shown. Remember: We are talking about tags which have not much traffic. Commented Sep 14, 2018 at 19:51
  • @ChristianGollhardt Your ordering may or may not be a better measure of actual contributions, but it requires an explanation to be understood, which isn't great from a user experience point of view. An explanation of "why aren't I on the list" might also be needed for my suggestion, but at least just looking at the list would make it fairly clear why one user is above another (especially if you're just not showing <20 score at all). Commented Sep 14, 2018 at 20:03
  • Regardless of the algorythm, the sorting needs to be explained, hence this question. It is pretty obvious, it is sorted by the answer score, still we have this question. That said, sure your answer is more easy to explain. @Dukeling Commented Sep 14, 2018 at 20:06
9

A single answer doesn't make one a top expert in a specific tag. The answerer might be lucky. Therefore, yes, I agree it needs some improvement.

For example, I definitely wouldn't call me an expert on , but the system thinks I am.

On the other hand, I see some danger in giving the count of answers too much impact on the ranking. This would only reward persons who are posting many answers. But posting many answers which manage to get an upvote is not the same as posting many answers which are scoring very high.

So my suggestion?

I think the algorithm of the Select your next badge dialog is reasonably sorted. Maybe we could copy it in someway.

One possible way: Create a Maximum Answer count, e.g. 20 and calculate the percentage of answers. Then we can multiply the overall score by the percentage.

Example:

50 Answers, Total Score 100: Rank: (20/20)*100 = 100,00 (note, we reached the maximum 20/20)

20 Answers, Total Score 100: Rank: (20/20)*100 = 100,00

19 Answers, Total Score 100: Rank: (19/20)*100 = 95,00

5 Answers, Total Score 100: Rank: (5/20)*100 = 25,00

4 Answers, Total Score 120: Rank: (4/20)*120 = 24,00

1 Answers, Total Score 100: Rank: (1/20)*100 = 5,00

This way low traffic tags have the benefit that the answer count is considered until we reach n (20) answers. If a user has enough answers, only the score (the quality) will be counted.

I composed a query, how it could be look like (sorry for my sql skills, I hope there is no error...)

select c.*
, ((c.accounted_post_quantity / 20.0) * c.post_repuation) as better_score   
from
(select p.OwnerUserId as [User Link] -- is you
     , sum(p.score) as post_repuation  -- answer score
     , count(p.parentid) as post_quantity
     , case when count(p.parentid) > 20 then 20 else count(p.parentid) end as accounted_post_quantity
from posts p  -- answers
inner join posts q on q.id = p.parentid -- link answer to question
and p.posttypeid = 2 -- answers
and q.tags like '%<##tag##>%' -- tags are enclosed in brackets
                                 -- and are only filled for questions

group by p.OwnerUserId) c
order by better_score DESC

Here are the current results for :

SEDE Result

10
  • The /20 in your formula is unneeded to sort the users: your sorting algorithm is equivalent to an easier concept: number of upvotes * number of answers. Try to emphasis on such formula.
    – Cœur
    Commented Sep 12, 2018 at 14:18
  • The main point I am trying to do with this formula, is rewarding the first n (20) answers, but after that is reached, only the score of the answer is relevent. @Cœur Commented Sep 12, 2018 at 14:55
  • 1
    Thanks for the suggestion. Done @Cœur Commented Sep 12, 2018 at 15:30
  • From the results of the query, it sounds that it may be equivalent to only take into account people with at least 20 answers on a given tag.
    – Cœur
    Commented Sep 12, 2018 at 15:32
  • I have had an rounding error. Fixed now. Since Steam has less traffic, here we can see the difference. Probably this needs some tweaking, some pow or such, but I am not realy good in statistics @Cœur Commented Sep 12, 2018 at 15:46
  • 2
    It may not be about tweaking, but more about perception: what is a top answerer, how is it defined? Is it about votes, about number of solved questions, or an arbitrary mixed of different factors?
    – Cœur
    Commented Sep 12, 2018 at 16:16
  • I think an top answerer is someone how gives the best (highest voting) answers. Since one can get some higher voted answers by accident, we shouldn't count the votes from one answer as high as if the person has 20+ answers. That is what I am trying to reflect with the calculation. For High traffic tags, there will be no difference, but for low traffic tags it will be a difference. Commented Sep 12, 2018 at 16:27
  • Is it really possible to devise an algorithm that provides a genuinely meaningful metric? The "Impact" feature on the user's activity page suggests it may not be. I think the simplest improvement would be to allow sorting by number of answers as well as by rep. In low traffic tags, the latter may be a much more significant indicator of overall contribution, because most users don't have the knowledge to even attempt an answer. There is little danger of users "posting many answers which manage to get an upvote" - even the genuine experts often get nothing more than an accept for good answers.
    – ekhumoro
    Commented Sep 12, 2018 at 17:18
  • Thanks, in particular for making me aware of queries, I did not know of their existence. Your approach does seem to weed out the low frequency answerers for openssl. Commented Sep 13, 2018 at 5:21
  • Good to see the community wiki is so active in the tag ;)
    – Luuklag
    Commented Sep 13, 2018 at 6:31
4

Sort by tag badge first, then by vote score

... with a gold / silver / bronze tag badge icon for each user, where applicable.

This is the outcome of a joint idea.

The advantage of this approach is that it should be easy to understand out-of-the-box for those familiar with tag badges, since the visual badge indicator should make it trivial to figure out the ordering. The icon could even be clickable, like it is on your profile, to allow those who aren't familiar with tag badges to under figure it out as well (and learn about tag badges).

A mockup list by @Dukeling looks as follows (although in reality, there are more bronze badge owners):

For a real-time calculation for a given tag, see the query Top Answerers: order by badge, then by score descending. Currently, the result looks like this:

Outcome of query

4
  • Thanks! I have created a query that calculates the top 20 according to this rule. Can I add it, with a screenshot of the result, to this answer? (I do not know how 'community wiki' works.) Commented Sep 14, 2018 at 21:42
  • @ReinierTorenbeek Sure you can. Did you intentionally INNER JOIN on Badges a second time, to multiply the results? (as opposed to this). Commented Sep 14, 2018 at 21:56
  • I did intentionally add it because I was struggling to get it to work otherwise :-) (I am new to SQL) Thanks for fixing that, I will add the results to the post now. Commented Sep 14, 2018 at 21:58
  • Sorry, my attention was drawn to the gratuitous freehand red circle. What were you saying?
    – kjhughes
    Commented Sep 15, 2018 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .