-4

We are software engineering researchers. We recently conducted a study on Stack Overflow question comments (Using the Dec. 2021 datadump with more than 40 million comments for questions). Below are some of our major findings. Please share your thoughts on our findings and share your own experience of using the comment section of questions on Stack Overflow.

We studied when and how much people comment on questions

  • The proportion of questions with comments have increased by year; for example, ~30% of questions posted in 2008 have comments, and ~60% of questions posted in 2013 have comments. Of course, a question posted in 2008 may receive a comment many years later, but the data shows that comments are most commonly posted very soon after the original posting of the question.
  • Comments can occur even after a question is solved! In commented and solved questions, 19.4% of questions received their first comment after the accepted answer was posted. We think these comments may highlight important information in the question such as false information, improper assumptions, or obsolescence.

Based on our findings, it seems like commenting on questions is actually pretty common and can happen at any time during the question and answering process. Currently, comments have several functions as suggested by their prevalence; people use them to discuss with each other and share additional information. We think there exists an opportunity for a better design of the system to separate the functionalities. For example, we can add buttons to report false and/or obsolete information. Then reported content can be reviewed by other members of the community. More interestingly, there could be automated summary tools to summarize the comment content so people won't miss what's in the comments if they are really crowded (hey, by default SO only shows 5 comments).

We looked at the users who comment on questions

  • Lots of users comment on questions! The number of users who comment on questions is of similar size to the number of users who answer questions.
  • Askers and answerers have a high appearance in the question's comment section. (When more than 5 comments are present in an answered question, 90.3% of them have askers in the question's comment section and 51.9% of them have one of the answerers in the question's comment section.) We think this suggests that people are leveraging the comments to communicate with each other when the question is hard to answer in the first place.

Given that many people are using comments essentially as an interactive way to answer questions, can another way of Q&A happen on Stack Overflow, where people ask and answer questions in places like chat rooms and later they can organize the Q&A section into dedicated questions?

We also discovered some relationships between question comments and other activities

  • Questions with more comments are edited more
  • Questions with more comments before answers are answered more slowly.

Please let us know what you think of our findings and share any experience whether positive or negative regarding your experience of posting comments on questions.

36
  • 18
    "For example, we can add buttons to report false and/or obsolete information. Then reported content can be reviewed by other members of the community" - Too late, false is covered by downvotes, obsolete content is covered by new answers (because nothing is obsolete, people support old software you know). In addition to new answers, and obsolete content will potentially be being covered in the upcoming (tm) outdated answers project. Feb 16, 2022 at 3:27
  • 3
    @Joundill a question deemed interesting might get people to comment on it. I don't think "more comments" correlates with "bad question" strongly enough.
    – VLAZ
    Feb 16, 2022 at 5:36
  • 9
    @Joundill I'd challenge that. The vote count to close a question is 3. If there are multiple comments, it means multiple people saw it. What they need to do is close vote in addition to their comment. Which often happens. I've seen questions with blatantly missing information stay open but they are usually seen by very few people. I've looked at my own expired close votes and it's often on questions with, say, 20 views or so. And maybe 0-2 comments. Yes, there are questions with multiple comments which is a 1-2 close votes but that's rarer and the votes expiring rarer still.
    – VLAZ
    Feb 16, 2022 at 5:46
  • 10
    "people ask and answer questions in places like chat rooms and later they can organize the Q&A section into dedicated question threads" This is already very much possible. And it happens rarely because most questions just aren't worth throwing more effort after them. Honestly, the comment statistics you cite seem like a bad sign for Q&A, since they hint at lots of help-desk style questions that need the back and forth. Feb 16, 2022 at 6:54
  • 3
    I think a lot of comments result from the fact that questions are so poorly phrased, researched or formatted that folk can't be bothered to answer. Feb 16, 2022 at 8:42
  • 6
    Also, you can invest a lot of effort in providing a well explained, well researched and tested answer and so often the asker never logs in again, or never acknowledges the answer in any way so it is not a bad strategy by answerers to request a quick clarification via comment first (before full answer) to be sure that the asker is going to hang around and stand by their question. Feb 16, 2022 at 9:41
  • 6
    I do not trust any study that does not disclose their method of data collection or the method used to arrive at their results Feb 16, 2022 at 10:06
  • 13
    You're drawing some conclusions from these statistics, but correlation isn't causation... Of course question askers and answerers are likely to be the ones commenting... Of course comments can be made at any time. How are you drawing your conclusions from these statistics?
    – Cerbrus
    Feb 16, 2022 at 10:16
  • 12
    "A significant and increasing proportion of questions receives comments" How did you get access to this information. To my knowledge, Stack Overflow doesn't share deleted comments
    – Dharman Mod
    Feb 16, 2022 at 10:23
  • 6
    “A significant and increasing proportion of questions receives comments (increases from around 30% since 2008” — did you account for the fact that comments have only existed since some time in 2009? At the beginning of the site, there was no comment system yet. The initial increase you’re seeing is likely due to that (and in fact all comments you’ll see on content before that would have been added later). Feb 16, 2022 at 11:01
  • 5
    "...it seems like commenting on questions is actually pretty common and can happen at any time during the question and answering process..." Yes, I can confirm that. But then it's kind of like a very general observation.
    – Trilarion
    Feb 16, 2022 at 11:45
  • 4
    "...Dec. 2021 datadump with more than 40 million comments for questions..." One important thing to know is that comments are considered second class citizens on SO, i.e. even if all comments would be gone, the content should still make sense. That's why, even though there are 40 million comments existing, 20 millions or more of them could probably go in an instant without a big impact on the operation of the site.
    – Trilarion
    Feb 16, 2022 at 11:48
  • 3
    @CosmosZhu I'm not questioning the data points you gathered. I'm questioning the conclusions you draw from them. This question contains some arbitrary statistics... Now what? "Discuss!" isn't an effective way to get a discussion started.
    – Cerbrus
    Feb 16, 2022 at 15:04
  • 3
    @CosmosZhu: Who are the "we" you are representing? A research institution? Students? Professors? What drives your research?
    – Cerbrus
    Feb 16, 2022 at 15:26
  • 5
    What were the research question(s) you set out to answer? I don't see any mention of research questions or hypotheses in your post. The findings are mostly either trivial or it's not clear how you inferred them, but more importantly, what were you trying to find an answer to? It strikes me that you acquired the data and then tried to figure out what you could do with it, rather than formulating a hypothesis and then deciding what the best way to test it would be.
    – kaya3
    Feb 20, 2022 at 6:51

5 Answers 5

31

What I understand is that you found out that comments on Stack Overflow are used according to their purpose. What exactly was the purpose of this research? Did you find supporting evidence for the assumptions you were trying to prove? What metrics did you use?

It looks like you ignored the fact that the majority of the comments ever posted on Stack Overflow are probably deleted. There's definitely huge survivorship bias when it comes to comments, because we delete all the obsolete ones. Many of the longer comment conversations were moved to chat. How did you factor all of this into your findings?

I don't find this research useful at all, because I don't even know what you were trying to research. Yes, we have comments and we use them to clarify questions and answers when needed. We already knew that.

9
  • 1
    We started this research trying to understand the dynamics of commenting behavior on questions on Stack Overflow. As an empirical research, studies like this more focuses on the results and maybe confirming/denying "common sense" regarding the topic.
    – Cosmos Zhu
    Feb 16, 2022 at 14:25
  • 1
    We are aware of commenting could get deleted over time. But unfortunately, Stack Overflow doesn't share this data publically. This is definitely a potential threat to the validity of our study. Since Stack Overflow do have snapshots of the datadump from the past, we compared the datadumps between 2019 and 2021. The percentage of comments deleted is quite low (i.e., ~2%). While comments that are deleted immediately or within a short amount of time wouldn't be captured by the datadump, the low percentage does suggests that if a comment "survived" initially, it'll likely stay.
    – Cosmos Zhu
    Feb 16, 2022 at 14:28
  • We also noticed that in very long comment conversations, moderators are likely to move the conversation into a chat room to avoid clutter. We initially looked at this data, and fun fact, they are done very actively during a period by a few mods.
    – Cosmos Zhu
    Feb 16, 2022 at 14:30
  • Sorry for the multiple comments, a char limit exists : (. The chat room data is also very challenging to collect. SO deletes inactive chat rooms (rooms with <15 messages in 7 days) and the information are also not publically available. Traces in comments can be find (i.e., "let us continue this conversation in chat"), using this suggests that about half of chat rooms induced by comments are deleted.
    – Cosmos Zhu
    Feb 16, 2022 at 14:32
  • 3
    @CosmosZhu "we compared the datadumps between 2019 and 2021" So you have one snapshot per year only, meaning that on average if comments are deleted within 6 months they are completely absent from the statistics? Feb 16, 2022 at 14:33
  • While I understand that you don't find this research useful, the goal of our empirical research is to understand the exact numbers and details. For example, if we observed more percentage of comments after answers, it would imply very different meanings and if we observed significant less percentage, than these comments are likely just noise.
    – Cosmos Zhu
    Feb 16, 2022 at 14:34
  • 2
    @MisterMiyagi Unfortunately yes. Stack Overflow datadumps only capture what is a "current" copy of the website. Stack Overflow datadump are posted quarterly. So any comment that has been posted and deleted within a quarter won't be noticed.
    – Cosmos Zhu
    Feb 16, 2022 at 14:36
  • 2
    @CosmosZhu it's actually good that deleted comments are not public and Stack Overflow data dumps only capture the snapshot at the time it was taken. We don't want this kind of comments to linger for more than a quarter...
    – Andrew T.
    Feb 17, 2022 at 3:20
  • @AndrewT. For us it's good, for science it's bad. Science will then never know the true picture and all inferences may be biased and only conditional on the subset of undeleted comments. This should be described and discussed in any related scientific work and makes the interpretation of results more difficult. I guess that SO keeps a record of deleted comments privately (see unfriendly comments detection project) and one could ask them for that data for a scientific study. The authors here probably didn't or couldn't obtain permission.
    – Trilarion
    Feb 17, 2022 at 7:53
20

I just want to state the obvious: easy questions are often answered in comments.

Many questions are very localized (i.e. help only the OP) and very easy to answer (i.e. obvious to anyone who has any experience). For these, writing a proper answer, with explanations, seems like a waste of time. So people answer in comments.

Here is the thought process from a perspective of such answerer.

I would waste 5 minutes of my time by writing an answer to help one person. While writing a comment would waste 1 minute of my time, and probably still help that one person. Of course, doing nothing would waste 0 of my time, but I already invested 2 minutes of my time into reading the question, so doing nothing makes me think that I wasted those 2 minutes.

I don't want to suggest anything to improve this situation; I guess this has already been discussed, and possibly some solutions were found and suggested.

2
  • 4
    Very true - and it's often easier and more helpful to the OP to give a quick hint in a comment than to try and remember, or look up, a related answer that refers and/or declare it a duplicate and get an argument about whether it's a true duplicate or the circumstances differ marginally. Feb 16, 2022 at 10:22
  • 8
    @MarkSetchell On the other hand, answering basic questions that are likely duplicates in comments may help the OP but would also increase the noise level of the whole set of existing questions unless this questions also end up closed and deleted. Not sure it's a good strategy in the long run.
    – Trilarion
    Feb 16, 2022 at 12:41
20

One thing I've noticed is that new users don't seem to realise they can edit their questions. When asked for clarity, they often (attempt to) dump great chunks of information in a comment.

Visibility of the edit feature should be higher for the question author, especially for new users.

This would greatly cut down on redundant comments, in particular those that attempt some form of code formatting.

6
  • 2
    Probably worth posting a separate question for the FR.
    – VLAZ
    Feb 16, 2022 at 8:10
  • @VLAZ fair enough, I changed the tag to discussion for this one
    – Phil
    Feb 16, 2022 at 8:15
  • 12
    I always add a suggestion to [edit] the question unless I am sure the user knows how to do it. Such a suggestion is not offensive, and provides the user a convenient button to edit.
    – anatolyg
    Feb 16, 2022 at 10:19
  • @anatolyg same here, it does get a bit tedious at times though
    – Phil
    Feb 16, 2022 at 13:44
  • 1
    If I see updates in the comments I use Rather than adding details in comments, please [edit] your question to add requested information. You can then comment saying that you have done so. Comments here may get hidden by "show more comments" or could be deleted without notice. Editing updates into your question ensures that they don't get lost and that everybody who wants to answer the question sees them. Feb 16, 2022 at 15:49
  • To be fair, this is hardly exclusive to SO. I often see the same behavior on other sites with both editing and comment replies, like Facebook or Reddit. Apr 19, 2022 at 18:15
11

I think your analysis is pretty much useless as your post is low on substance outside of basic numbers and trivial findings. It seems you went on a fishing expedition in the data dump, calculated various metrics, and now report some "interesting" (not really) findings back. While the basic numbers (e.g. comment counts and percentages) might have some use when debating comment use on SO, the rest of your post is lacking.

For example, you say:

We think these comments may highlight important information in the question such as false information, improper assumptions, or obsoleteness.

That's nice, but seems to be utterly unsubstantiated, or at least your post does not indicate if that is pulled out of thin air or based on any further analysis of the comments. Even if we ignore that, it's very vague and as such not exactly useful - obsoleteness alone would be a broad topic requiring a detailled analysis before any conclusions could be drawn (for example: what parts of the Q/A pair are perceived to be obsolete; how often is that debated/rejected by other comments; if something's "obsolete", is the old version still useful or should it not be used in any context; etc).

The TL;DR for the rest of your findings is "yawn" - comments were less used when SO was in its infancy, people who interact most with a question (askers and answerers) are also writing a lot of the comments, and some vague correlations of comment numbers with edits and answer speed which are not even detailled enough to form an opinion about.

All in all, if you call yourself "researchers" I would expect a more rigorous scientific approach. As I said earlier, this does not look like you used the SO data to verify or falsify some hypotheses - instead, you just fished for anything that looked like it could be significant.

3
  • While not considered useful, we are glad to hear that our findings align with your impression of commenting on SO as you don't find anything too surprising. Our research is the first step to understanding commenting behaviors on Stack Overflow questions.
    – Cosmos Zhu
    Feb 16, 2022 at 14:49
  • As for your raised points on obsolete on the platform. There is actually a research on this topic, if that interests you here is the link. To quick summarise, many content are obsolete when they are posted on SO; over half of the obsolete information does not get updated even if people point out its obsolete.
    – Cosmos Zhu
    Feb 16, 2022 at 14:51
  • 1
    "To quick summarise, many content are obsolete when they are posted on SO; over half of the obsolete information does not get updated even if people point out its obsolete." It might be a good idea to correlate that with answer/question score or number of visitors in further research. The more important and useful pieces of content may have a higher update ratio. Also there is quite some tendency against changing existing content too much, often new answers are created instead. The obsolete content is kept as reference but only marked as such.
    – Trilarion
    Feb 16, 2022 at 20:21
-12

This is very interesting research, which may answer why quality of questions is so low now on SO.

Does number of comments correlate with question score?

Does number of comments with question mark (?) correlate with new users questions?


I think some very common issues with questions - and following comments - can be automated to prevent such questions from posting.

  1. pictures of code
  2. "why this isn't working" phrases
  3. questions without question marks (?) and without any sight of formatting at all
  4. questions with poor grammar
9
  • 7
    What evidence do you have that “quality of question is … low now on SO”? Compared to what? Did the average quality drop? I strongly doubt this claim. Feb 16, 2022 at 11:02
  • 1
    @KonradRudolph based on a common sense that more questions on SO - more information available for everyone - less new questions asked. From my perspective SO drives into dev forum - instead of answering for a specific problem
    – Kos
    Feb 16, 2022 at 11:22
  • 10
    "Common sense" isn't exactly a credible source / evidence...
    – Cerbrus
    Feb 16, 2022 at 11:39
  • 4
    From your list, only pictures of code are a sign of a low quality posts. If asked properly, why something isn't working can be excellent questions. Also a good post does not need a question mark -- I'd always prefer a clearly stated problem with a nice succinct description than unnecessary fluff just to sprinkle punctuation marks around the post. And about the grammar: one needs to keep in mind, that most of us are not native speakers. As long as the posts are understandable and clear, the grammar can easily be fixed, that's not what makes a post good or bad. Feb 16, 2022 at 11:45
  • 1
    @samcarter_is_at_topanswers.xyz great, so any improvement of quality of SO questions is counter-productive in your opinion? When you open main questions page and read first questions you are happy with their quality?
    – Kos
    Feb 16, 2022 at 12:23
  • 2
    @Kos No, the question quality on SO is abysmal (compared to other sites of the network), but reasons 2.-4. are not indicators of this. Feb 16, 2022 at 12:31
  • 3
    IMHO the main cause of the quality problem is the ratio of curators vs. OPs. For years SO would do anything in their power to drive curators away while desperately trying to attract more users without making it clear to them what the platform is actually meant for. Feb 16, 2022 at 12:36
  • @samcarter_is_at_topanswers.xyz I'm glad that you value my input so much, it were examples from top of my head, I'm glad that quality of SO content is valued high in others opinions, maybe it's just my tiredness, I can't deal with it
    – Kos
    Feb 16, 2022 at 12:38
  • 5
    Comments cannot be posted until the post they are commenting on are posted. So how would research on comments help to ensure low quality posts are not posted in the first place? I don't have a problem with your opinion of the research (even if it is a bit myopic), but the conclusions past the horizontal rule seem to belong to a different conversation. Feb 16, 2022 at 14:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .