81

The blog post Welcome Wagon: Classifying Comments on Stack Overflow has brought out both sides of the "too welcoming"/"not welcoming enough" debate. I've noticed every time this is brought up, there's some group of existing users who decide (or as least posture) that they'll simply walk away from difficult users to avoid being unwelcoming.

The most recent example I've seen is from Is this really what we should consider "unwelcoming"?:

Please tell me what we're supposed to be doing instead then. Walk away and don't provide any help at all rather than call a spade a spade?

This was from a >100k user; not a user I'd want to opt out of comments. Personally, I would find this far more harmful to the site than being unwelcoming. However, this all depends of whether or not there is data which suggests high-rep users have been commenting less frequently on new posts in the recent few months as Stack Exchange has fleshing out a better idea on how welcoming to be.

If there's been no observable change, my point is moot.

If there has, however, then it may be worth a discussion on where a balance lies between being welcoming while still encouraging our most helpful users to contribute. I'll table that discussion for now.

As it stands, I was wondering if we have been able to observe detrimental changes in the commenting behavior of high rep users (or any helpful user, I'm just not sure a better metric to use. Users who comment most often, maybe?)

NOTE: Please don't rehash this debate in the comments. I'm only after objective data here.

42
  • 92
    "I've noticed every time this is brought up, there's some group of existing users who decide (or as least posture) that they'll simply walk away from difficult users to avoid being unwelcoming." That's SO's whole goal here. They're trying to get people to comment less with the problems a post has and how they can be improved because new users don't like being told what they've done wrong. The experienced users are complaining because they want to be able to provide that useful feedback that SO is trying to prevent them from posting.
    – Servy
    Jul 10, 2018 at 19:06
  • 36
    Comments are pretty hard to measure. But there has been an observable change, the number of downvotes instantly fell by 24%. Arguably the most institutionalized kind of unwelcoming. I expected it to recover, it slowly increased again. But it didn't, stuck at around 16% down today. Whether it is because users opt to no longer downvote or because a lot of users with vote privileges quit is not obvious to me. The previous time this happened (2012) a lot of users did quit. SO never recovered from that. Jul 10, 2018 at 19:16
  • 41
    @TravisJ It's a pretty fundamental principle of the site that quality questions are important. Questions that are unclear, are missing information, or that have other problems are problematic. It results in answers that are just guesses (which will sometimes be wrong), questions that can't be answered at all, makes it harder for others coming across the question later to see if it is applicable to their problem (or even to find it). There are lots of sites that don't have much in the way of question standards and where most questions are allowed. SO is one of the few that isn't like that.
    – Servy
    Jul 10, 2018 at 19:20
  • 4
    @Servy - Comments about missing information and clarifications are perfectly acceptable. It is when users start defining "other problems" as whatever their pet peeve might be and then enforcing those requirements on questions that issues arise. I agree with you, question quality is very important. Standards for closure are important and should remain. However, kibitzing on tangential material in questions instead of addressing the core situation is the main problem being addressed here. Is SO being successful with their approach to solve that? Perhaps, but that is the main issue.
    – Travis J
    Jul 10, 2018 at 19:23
  • 10
    @TravisJ Wait, so your problem is with people posting comments with information that's likely to cause problems for a question author but that isn't what they explicitly asked about? First off, that's not what SO appears to be primarily targeting (at least to my understanding), but second, why in the world do you not want people posting such information? Providing information that's tangential, but still useful, is entirely appropriate. In some cases, far more important than the actual answer (and, naturally, sometimes much less important).
    – Servy
    Jul 10, 2018 at 19:28
  • 10
    @TravisJ The examples of things that SO is saying they're trying to get rid of aren't people commenting on the author of the question, merely pointing out problems with the question that need to be fixed. That people so often consider advice on how their question can be improved to be unwelcome is, of course, highly problematic. Yes, lots of people take criticism of their post personally, and treat comments stating problems with a question as an attack on them. The solution isn't to stop people from posting comments explaining problems with a post.
    – Servy
    Jul 10, 2018 at 19:39
  • 12
    @TravisJ I don't consider it inappropriate at all to ask the author, specifically, to fix a problem with the question. It makes it clear that it's their responsibility to take a specific action to fix the problem, and that they can't just wait for someone else to fix it for them. It's still not a commentary on them as a person, merely the actions that they have (or have not) taken, and what they should (or shouldn't) do in the future. Acknowledging that a person is involved in authoring the question isn't inherently problematic.
    – Servy
    Jul 10, 2018 at 19:51
  • 12
    @TravisJ 92% according to a few dozen SO employees. An opinion not shared by many of the other thousands of members of the community.
    – Servy
    Jul 10, 2018 at 19:57
  • 10
    @Servy Don't be absurd. That is not SO's goal. Their goal is for people to stop being rude. Yes, they are stumbling like a spider wearing roller skates in their attempt to do it. But their goal is not seppuku.
    – TylerH
    Jul 11, 2018 at 5:09
  • 8
    @HansPassant Downvotes being down 16% actually really worries me. Is that a consistent trend since the blogpost (I assume obtained using SEDE?)? Do you have similar trends going for close votes and delte votes?
    – Magisch
    Jul 11, 2018 at 7:43
  • 4
    I have no real idea what you are objecting against. I merely noted that DVs are a welcoming problem, a lot of users take them as a personal attack and complain about it loudly. Not the only kind of institutionalized unwelcoming, questions bans are the harshest kind and nobody is ever happy about close and dup votes. Those have all been weakened quite a bit, but DVs are pretty hard to squelch. I assumed, but apparently a blog post was enough to put a dent in them. Jul 11, 2018 at 11:38
  • 12
    @TylerH Their goal is for more people to use the site. They think that they aren't getting as many users as they want because there is rudeness on the side, and they think that the problem is a result people posting rude comments. I don't believe that either of those are major factors that affect the userbase of the site, nor do I think that their attempts to make the site more friendly are likely to be successful (on a significant scale). But just because they're not actively trying to kill themselves doesn't mean that's not precisely the direction they're moving towards.
    – Servy
    Jul 11, 2018 at 13:11
  • 7
    @TylerH I've seen lots of SO staff specifically saying that they're trying to get people to comment less. It's not always easy to distinguish company policy from the policies that individual employees are advocating, but I'm also not sure if it matters, as the later will be treated by most as the former even if that's not the intent. Many people have made it clear that they want people downvoting and moving on way more often, and commenting way less.
    – Servy
    Jul 11, 2018 at 15:15
  • 11
    @Omnifarious And it's precisely your behavior that SO thinks new users tend to perceive as an attack and as being unwelcoming, and that SO is therefore trying to discourage. That said, I find it rather uncommon to see a post be downvoted much or closed without comments indicating its problems or how it could be improved. The community, generally speaking, spends a lot of time trying to help people improve their questions, despite the fact that very few of them actually do so in response to such feedback.
    – Servy
    Jul 11, 2018 at 19:15
  • 7
    I can speak from my personal perspective (as a 100K+ user). I've cut my commenting by probably 90%, opting instead to simply downvote and closevote rather than commenting. When you can't ask for a MCVE or suggest an edit or clarification without saying Pretty please, may I please (with no offense intended) ask you (again, not meaning to make you feel unwelcome), beg of you to provide some code (but it's OK if you don't, because asking for it might make you feel bad) that helps explain the problem (not that you don't write extremely eloquently, but because I'm not smart enough to understand)?
    – Ken White
    Jul 13, 2018 at 2:15

1 Answer 1

59

If you're talking about the amount of comments, I've constructed this SEDE query to measure that.

Graph of comments by >10k users

Eyeballing it, I see that the amount of comments by >10k users has been on a steady decline since 2014. You can enter your own cutoff for high-rep and see the numbers yourself.

If you're talking about the quality of comments, SE is just developing tools to measure that. I hope they make the data they gather publicly available, so we can draw our own conclusion. But there's no real data available.

Note that there are many sources of bias for drawing conclusions based on this query. The main ones are: the reputation cutoff is current reputation, not reputation when the comment was made (this less new comments), deleted users and comments aren't counted (this leads to more new comments), reputation may be harder to achieve nowadays (this leads to less new comments). So take it with a grain of salt.

17
  • 17
    Note that this isn't able to account for deleted comments, or comments from now deleted users, which skews the data.
    – Servy
    Jul 10, 2018 at 19:41
  • 6
    @Servy Indeed, and it doesn't correct for the fact that there are more high-rep users with passing time either. But I'd expect all those effects to skew it towards more recent and less older comments. Honestly, this wasn't at all the result I was expecting.
    – Erik A
    Jul 10, 2018 at 19:43
  • 5
    Trying to normalize for number of users with X reputation would be hard as, over time, users will stop being active. You'd really want to normalize it based on "active" users, but now you need to define the term somehow, which isn't easy.
    – Servy
    Jul 10, 2018 at 19:46
  • 3
    @Servy Yes, activity might be a large bias here. I wouldn't know how to reliably correct for that, besides calculating rep at the time the comment was posted, which will be way too complex to not cause a timeout.
    – Erik A
    Jul 10, 2018 at 19:49
  • 1
    @ErikvonAsmuth This is just the kind of stuff I'm looking for, thanks! Would there be a way to add a filter to run this query only for comments added with 24 hours (or some other amount) of a question/answer being posted? I'm giving it a shot to try joining it on Posts where the creation date of the post is within a day of the creation date of the comment, but I'm not very good with the SEDE. If you don't want to actually write it, you can just tell me it's possible and I'll work it out; I just don't know if there's a queriable association between posts and comments. Jul 10, 2018 at 19:51
  • 5
    @LordFarquaad I have the time, so here you go. The numbers are remarkably similar to the previous query: a very large fraction of all comments is posted within 24 hours, especially those by high-rep users. Tip: read the SEDE faq. There's a relationship between nearly all tables in SEDE.
    – Erik A
    Jul 10, 2018 at 20:02
  • @ErikvonAsmuth Thank you! For the query, but mostly for that faq link; that's a gold mine. I guess that makes sense the data is so similar; I suppose high-rep users aren't prone to going back to a week-old question Jul 10, 2018 at 20:06
  • 3
    2014 sounds right, that's about the time IIRC that my activity on SO started to taper off. Nowadays the only things I really do here are I ask a question when I need to, upvote good answers that I get to from Google, and peruse meta.
    – Ian Kemp
    Jul 11, 2018 at 11:59
  • The decline exists period. You can see it with the HighRepBorder at a mere 90 - not 900 or 9000, 90. It's just more pronounced at >2k, as around the 2k rep mark and below, there's an odd spike in 2015 before it starts dropping again.
    – Izkata
    Jul 11, 2018 at 14:07
  • It's worth comparing this to the overall number of comments for all users. There does seem to be an overall decrease in comments, but the trend is more pronounced for high rep users. (Query for 1 rep or higher)
    – Stevoisiak
    Jul 11, 2018 at 14:54
  • 4
    @StevenVascellaro Keep in mind that 10K users have more than 1 rep too, so you might still see the effect of the high-rep users commenting less, just diluted by more comments. If you only query for users with <1000 rep, it looks very different. Do note that comment by Servy can explain a lot of this: the query takes current rep into account, so 1-rep users in 2014 may have more than 1K rep by now.
    – Erik A
    Jul 11, 2018 at 14:58
  • 1
    This also doesn't account for past initiatives like the summer of love, better tooling for removing comments, pushes to delete older comments by the staff/moderators, etc.
    – TylerH
    Jul 11, 2018 at 15:24
  • 1
    This needs to be adjusted for 'rate of people achieving 10k'. There is going to be a lifetime for most people's engagement with any site. Over time, people will leave. Also, new people might not achieve the required cutoff level as easily as time goes on for a wide variety of reasons. If you adjust for the rate at which users achieve the cutoff, I think the measure will be more accurately reflect what you're trying to measure, which is high-rep users engagement with the site. Jul 11, 2018 at 19:14
  • 3
    @Omnifarious I've added a note at the bottom about some of the sources of bias. Unfortunately, with the data available and the timeouts and performance of SEDE being what it is, I couldn't do better (rep history is not available and would need to be calculated for things like rate of people reaching 10K or reputation at time of placing comments. If you feel like you can write a less biased query, I encourage you to do so. And you can add notes to that message if you feel I've missed something.
    – Erik A
    Jul 11, 2018 at 19:26
  • 2
    @ZevSpitz No. The more you answer question, the more you get really annoyed due to people that ask questions without doing a simple Googling. Too many poor questions causes high rep users to reduce commenting, ignore the question, flag to close it or down-vote it. The quality of questions has not improved.
    – Programmer
    Jul 12, 2018 at 14:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .