219

Happy New Year, all! This is the first 2019 installment of our regular, bite-size, data-focused updates for Meta. You can check out previous posts if you like.

This month, let's look at how question views and answers are related. I expect most of us intuit that questions with more views have more answers, but how true is this? And what is the relationship like? To explore this, let's build a dataset of undeleted questions, with their number of answers, tags, and number of views. This is the number of answers as of today, so does not take into account deleted answers. How are views and answers related?

enter image description here

This box plot shows that overall, questions that have more answers do have more views, but notice that there is a large amount of individual variation. Also notice that the y-axis is logarithmic.

Linear models

We can learn more about this relationship by fitting some simple models, modeling the number of answers as a function of the number of views. Let's talk about two approaches for this model:

  • a simple linear model
  • a logarithmic model where we transform the views to log(Views)

The two models are similar in that both give us statistically significant coefficients, but the R2 of the logarithmic model is higher (0.16 instead of 0.13), indicating that the logarithmic model is a better fit. This makes sense, given what the first graph looks like. Also, notice that the R2 of neither model is very high; this is because there is so much question-to-question variation and our simple model can only account for a bit of this relationship between answers and views.

Tag differences

We can also train a separate (logarithmic) model for each large tag, instead of for all questions at once. When we do this, we find that all the slopes are positive and almost all the fits have low p-values. The number of answers per question increases logarithmically with views for all technologies. That being said, the rate of increase is not the same for all technologies.

enter image description here

The dashed line in this plot shows the median slope, i.e. the median increase in answers per 10x increase in views for this group of tags. Some technologies are near the line, but some are farther away. Technologies like .NET, iOS, C, and C++ get more answers per view, while technologies like R and Angular get fewer answers per view. The number of answers per question increases logarithmically with views for all technologies, but around this main effect, some technologies are more "answerer-y" while some technologies are less, at the same level of views. A few smaller technologies that exhibit even more extreme relationships are COBOL and Smalltalk (more answers per view) vs. dplyr and D3.js (fewer answers per view).

What are your thoughts? This analysis uses only data that is publicly available so you can reproduce this for yourself if you like, or dig into further details. Do you have topic ideas for future data science explorations?

38
  • 42
    This is really neat to drill into because we've known that people active in tags like, say, COBOL really mean business when they view a question; it's almost always to answer. These are essentially micro-towns on the outskirts of the outskirts that, statistically, thrive as well or better than the big city itself if you go by ratios. We half look at this any time we think about changes to tag maintenance tooling, but never really just to explore. This is super cool :)
    – user50049
    Jan 7, 2019 at 19:39
  • 25
    Yay, it's my monthly "how can I be so dense?" when looking at graphs time :) Jan 7, 2019 at 19:40
  • 24
    @FélixGagnon-Grenier It takes me usually 15 minutes to really understand this kind of graph with the context of the relationships being described. I know some people have this gift of just instantly having that all come together, I admire them. But you are certainly not dense, even "small sized" sets like in these posts can take repeated effort to digest, I really enjoy these because they take a bit of work to assimilate.
    – user50049
    Jan 7, 2019 at 20:04
  • 8
    What are the actual numbers for COBOL and Smalltalk (the slopes)? Jan 8, 2019 at 1:32
  • 20
    Ideas: To what extent does the age of the question affect (a) number of views and (b) the number of answers? There are old questions with massive numbers of views which don't accumulate new answers very often because the existing answers already cover what's needed, and there are new questions which quickly collect some number of answers and then only slowly garner more answers even as they are viewed more often. There's also the infamous FGITW (fastest gun in the west) syndrome; to what extent do questions in the popular tags gain lots of answers quickly because of FGITW vs unpopular tags? Jan 8, 2019 at 2:11
  • 16
    Ah, VBA... that one statistic alone encapsulates the entire plot, I think. I mean... how many times have you stumbled across a VB or VBA question, known the answer, but have just been too depressed by the poor quality of question that you just move on.
    – J...
    Jan 8, 2019 at 13:13
  • 39
    this may be just my opinion, but please, use logarithmic grids on logarithmic axes!! Jan 8, 2019 at 13:14
  • 11
    Is it views per registered SO user who is able to immediately post an answer? Or is it all views, including random Google hits by non-registered users who won't answer, accumulated over the years? Because I don't see any direct relation between the latter and the number of answers, other than post age. As in, what are you even measuring here: views as a metric of post age or views as a metric of SO users reading the question? In addition, wouldn't a question with more answers accumulate more Google hits? So it got a lot of views because it had a lot of answers.
    – Lundin
    Jan 8, 2019 at 15:03
  • 2
    An issue with using box plots is that their center mean is really just a midpoint and not reflective of a weighted average. It would be nice to see the weighted average point included in the boxplot (for example: i.imgur.com/TGRTCpc.png).
    – Travis J
    Jan 8, 2019 at 22:47
  • 6
    @PeterMortensen The slope for COBOL is about 1.6, and the slope for Smalltalk is about 2.4, using the same modeling approach used to make the graph shown in this post. Jan 9, 2019 at 2:38
  • 1
    @JonathanLeffler Older questions do have on average more views and more answers, and including time to create a slightly more complicated model results in a ever-so-slightly lower R^2, for example. It is a small effect compared to views-to-answers, though, and including time in the model doesn't affect the slopes I plotted very much. Jan 9, 2019 at 2:46
  • 2
    @llrs I built a tidy dataset with one row per question-tag combination, and looked at distinct PostIds when analyzing all questions. I put the code I used (except for the database query, which used the internal version of our tables) here: gist.github.com/juliasilge/cd046eaaa98417f32e05ada87b3fd1bc Jan 9, 2019 at 2:51
  • 1
    @Sinatr The plots in this post show all questions from all time, but I did look at the past year to see if there were differences, because this is something that interested me too. The dynamic range of this effect is smaller in the recent past but still directionally the same. Jan 9, 2019 at 2:53
  • 2
    @Lundin You bring up some interesting questions around causation and which direction this relationship goes. The views being counted here are all views, including unregistered users. I did look at post age as well (check out my comment above) and that can't account for these relationships that we see. The effect from time (post age) is small compared to the effect from views. Jan 9, 2019 at 3:02
  • 5
    I, for one, didn't intuit that number of views would correlate strongly with number of answers, and it doesn't look like a stretch to define "strongly" in a way that confirms my intuition. I've never thought that the factors driving one had very much to do with the factors driving the other. It might be illuminating in this regard to look at when answers tend to be posted relative to questions. I think we would find that most questions get most of their answers very quickly, when their view counts are still low. Jan 9, 2019 at 23:03

4 Answers 4

65

The following hypothesis seems plausible prima facie:

For a given technology, the Answer/View ratio (AV) is largely a reflection of an underlying Expert/Consumer ratio.

Where

  • "Experts" (or superusers) have high knowledge of the technology and have a greater tendency to answer questions; while
  • "Consumers" (or casual users) have low knowledge, and mostly view questions as they search for answers to the issues they're experiencing.

Under this hypothesis,

  • Low AV implies that the population using a given technology is, on average, skewed towards a low level of expertise (low density of experts, long tail of casual users). This is definitely the case for the VBA and Excel outliers, two technologies which are installed on many, many, many machines (with relatively few of those users caring to become experts). Perhaps we could characterize these as "means to an end" or "gimme results" technologies. In my experience, scientific languages like R are also like this ("I want this statistical analysis done now"). Maybe this is common to all domain-specific technologies that don't mostly attract pure software types. Even Python is a bit like that, what with the recent data science buzz about it.
  • Conversely, high AV implies that a high proportion of users of a given technology are advanced users. Let's face it: git, Xcode, Eclipse, multithreading etc. don't exactly attract casual users. C#, C++, Java, and iOS are also established technologies on which people build careers and in which they develop expertise (and for some, love). These technologies have more of a "must learn to use" character. COBOL is perhaps a special case, where we have, relatively speaking, a lot of experts as a legacy of COBOL's prior greatness, but few new users getting into COBOL.

This makes sense in my head. Not that this means much. I can see some counter-examples as well in the graph above.

12
  • 35
    You seem to be making the implicit assumption that more than one answer (or very few answers) is actually necessary or at least desirable. Maybe the R tag community is simply promoting a culture preventing superfluous answers and/or better at moderation? Also, I believe the slope presented in the graph should only be judged in conjunction with the proportion of unanswered questions.
    – Roland
    Jan 8, 2019 at 13:42
  • @Roland I see your point. Note, though, that I'm not making any claims about desirability, but rather about experts' comparatively higher tendency to post answers. Jan 8, 2019 at 16:00
  • 3
    @Roland Another data point: R has other thriving forums and communities (SO competition). The answers are being "hashed out" elsewhere and single answer posted here eventually.
    – yzorg
    Jan 8, 2019 at 18:02
  • @Roland That's not to downplay SO's role with respect to Google ranking and reputation. For R if SO is the "google index", or even a "ghost town", even then those forums might be replaced over time. So 2017 vs 2019 SO answers might point to different forum sites. SO still plays the role of "best search reputation" and even "huge search reputation" over years and decades, even if bulk of user activity is elsewhere any given year, which I believe provides massive value even to those communities.
    – yzorg
    Jan 8, 2019 at 18:28
  • 1
    @yzorg That's not at all what happens. Answers to R questions are not "hashed out" elsewhere.
    – Roland
    Jan 8, 2019 at 19:15
  • 2
    I think the languages you pick are interesting (and mostly point to scientific programming vs. software engineering to me). Where the "expert" in these field certainly think more along - well, this worked for me, for my research so why would I care if I went through the SDLC and really improved it - so "expert" vs. "asker" here can be misleading (and indeed one and the same as they know what they know and improve when they need to know more). Also, "Java" and "C++" don't attract casual users? That is the basic programming classes around me so its the basic of the basic student Qs as I see it
    – LinkBerest
    Jan 8, 2019 at 23:49
  • 7
    I disagree with the hypothesis. The number of answers does not correlate with the number of experts, it corresponds with the number of people posting answers. There's no indication, here, of the quality of answers; and I would expect that there's rarely more than 1 or 2 good answers on a given question, all others being "Me too" answers, not "Expert" answers. Both your and my hypothesis are equally backed by the data we have; despite being nigh opposite. TL;DR: we need more data. Jan 9, 2019 at 9:32
  • @JGreenwell Two things: 1. ""Java" and "C++" don't attract casual users?" I didn't write that. But I will say that they attract comparatively fewer casual users than something like Excel. 2. A student is most definitely not a casual user i.e. someone just trying to get something done ("means to an end"). A student is (in principle...) trying to learn the thing itself ("must learn to use" is the opposite category I made up). Jan 9, 2019 at 22:00
  • 1
    @Roland, yzorg maybe, maybe not? A quick glance at community.rstudio.com/search?q=stackoverflow suggests that a number of questions are answered by pointing to SO, but some others are asked because SO hasn't had an answer given yet. Anecdotally, many (newer) R users I've met find SO off-putting, and prefer the rstudio community site.
    – DaveRGP
    Jan 11, 2019 at 16:48
  • 1
    @DaveRGP It's ironic that you cite anecdotal evidence in a discussion under a post about statistics. I suggest you compare the number of answers by top 20 answerers with the number of questions in the tag. I can assure you these experts don't need to ask around to be able to answer the type of questions we get. Unfortunately, the questions usually don't even pose a challenge and have become quite tedious (and expert activity has decreased as a result).
    – Roland
    Jan 12, 2019 at 16:56
  • @Roland, I guess so. Unfortunately I didn't have a dataset of peoples opinions of their experience of SO available that wasn't biased towards the experts that are finding helping people tedious. Maybe that's why alternate sites also have their place in the R community for the 'non-experts'? :)
    – DaveRGP
    Jan 14, 2019 at 13:05
  • I would like to know statistics around what makes an answer become a verified answer. I would like to know the average timeline at which an SO answer gets "marked as an answer", whether certain qualities like code snippets increase answer acceptance likelihood, and how much asker response rates come into play. I would also like to know which tags receive the highest and lowest verified answers. Jan 17, 2019 at 20:24
12

It looks to me that the A/log(V) ratio might be to do with how narrow the language, and the community around it is.

Narrower tags have one/fewer right ways to tackle a problem. Broader tags have more scope for differences or different approaches.

Narrow Tags

  • Python-3
  • ReactJs
  • Excel
  • VBA

From experience of the top two, and cursory knowledge of the bottom two the community either takes the approach of 'one way of solving the problem' or the languages themselves are so esoteric, that there is only one working solution. This sort of matches the answer by Jean-François Corbett, in that you require experts to determine if there is a correct way (either because they're authoritive, or because most users just dabble and don't become experts).

Broad Tags

  • Python
  • Windows
  • Bash
  • iPhone
  • Java

These tags do not gravitate towards one-stop solutions. Although the zen of python espouses having one solution per answer, compared to python-3 it has had several years to tweak which answer is the 'best' (see string formatting). Likewise the other tags have 'history', where the correct soltution may differ by version or when the person started using that technology (see array maniputlation in java)

1
  • 3
    It's funny how VBA is so different from VB.net when they're similar languages. I understand they are different in many ways but still many similarities run between them and many solutions in VBA can be handled in a similar fashion in VB.net and vise-versa. Jan 18, 2019 at 0:44
8

I find the location of the .net tag very interesting. The c#, asp.net, asp-mvc.net, and vb.net tags are all more or less related and all follow a pretty nice line in the graph, but the .net tag is way out there. I wonder if this tells us more about the question authors who include the .net tag than the answering community, or if there is a difference in those people who follow and answer c# as opposed to .net questions? It feels to me that it is more than a statistical oddity and there is something useful to be learned there.

2
  • 1
    c#, asp.net, asp-mvc.net, and vb.net can all get the .net tag and just sum up the results, so the oddity is based on tagging even it is reasonable.
    – David
    Jan 14, 2019 at 13:48
  • @David if it just summed up the results, it would be in line with the others. As it is, it has a much higher slope (50% higher) than the others for the number of questions. That makes it an oddity, and I was wondering why and if it would tell us anything useful that could provide useful guidance. It could be based on tagging (is the rep higher on average for those who use the .net tag), or perhaps those who follow .net have a higher rep, or I don't know what. But it may be interesting to find out. Jan 15, 2019 at 14:06
4

I'd be interested to see how this would look with a 3rd axis: Score of the Answers

I suspect, in line with the thoughts about expert/consumer ratios, that this will highlight technologies with high-volume/low-quality answers, vs high-quality/low-volume answers.

I'd suspect the iPhone crowd will fall into the category of high-volume/low-quality. iPhone has a high degree of younger fledgling developers trying to make the next flappybird. Not saying that's the whole crowd, but self-taught people tend to gravitate toward that which they are most familiar. This actually suggests an interesting survey question: what phone OS do you have? Cross-referencing that with formal education in development/CS would be interesting, and would also help to verify this hypothesis.

This also seems to be supported by the proximity of android to Java and C#. Both languages are similar enough that I would expect them to behave the same way with regard to answer variability. There arent too many ways to solve a particular problem in them. The android questions tend to be framework related, which also has very few options to solve a particular problem. But the variation from C# to .NET seems to indicate an unaccounted factor. Score may help to shed some light on this as well.

Comparing the Answer/View/Score metric of android to iPhone would be interesting, just for the sake of being interesting.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .