116

Julia Silge and Jason Punyon just posted Welcome Wagon: Classifying Comments on Stack Overflow. There doesn't seem to be a dedicated post for giving feedback on it, so I thought I'd post my own.

I take a series of issues with it:

1. It'd be far more useful if we could see the raw data.

There's a whole bunch of interesting analysis we could do if we had access to the full set of comments that were classified, and the (potentially anonymised) classifications given by each individual staff member. In particular, we could:

  • Classify the same comments ourselves, and see to what extent our personal sensibilities align with those of the staff.
  • Extract out the set of comments conveying some particular point - like, for instance, that the question is missing an MCVE - and see which ways of phrasing the point were most likely to be taken as unwelcoming (or indeed see whether the phrasing matters at all).
  • Analyse the extent to which respondents agreed on which comments were unwelcoming.

2. The word "unsure" in the sentence "I feel unsure or annoyed" has two plausible meanings

I could take "I feel unsure" to mean either:

  1. Reading this comment would make me feel insecure / unsure of myself, or
  2. I am unsure how to classify this comment.

I took it to mean the former, and only saw the latter interpretation when it was pointed out to me in chat. Which did you intend? Are you sure your survey respondents interpreted it in that way?

3. The notion that "abusive" comments are ones most likely to make readers "feel angry or upset" is dubious.

If someone were to engage in outright abusive behavior in comments - tell me to go fuck myself, spew racial slurs, threaten me with violence or whatever - they would certainly not succeed at upsetting me. At worst I would find them mildly irritating. I certainly can be personally upset by comments on the internet - heck, I have been recently - but the sort of stuff that would warrant a "rude or abusive" flag on Stack Exchange ain't gonna cut it.

I suspect most people are similar. The sort of social behavior we're most likely to find hurtful is frequently not the blatantly outrageous conduct that all our peers will agree is wrong, but instead something that is at the line where others will find it acceptable - that's a key part of what makes it hurtful. But this facet of human nature makes labeling the most upsetting or anger-inducing comments as the most "abusive" a dubious approach at best, and an outright contradiction at worst; if your survey respondents' feelings work anything like mine, and they followed the instructions literally, the majority of the most blatantly abusive comments won't've been classified as such.

4. The notion that a comment that makes me feel "unsure or annoyed" is bad is dubious

If somebody claims that I got something wrong in a post, I'm probably gonna feel unsure or annoyed. I don't like getting things wrong, and, like, at the very least I'm going to immediately feel unsure about whether I actually made a mistake or not. Tone is irrelevant, here; I don't think there's any way you could criticise a post of mine that wouldn't at least briefly induce this emotional response.

Despite this, I think criticism is valuable, and I greatly appreciate it when others point out errors in my posts. Such comments are the most useful comments I ever receive.

If I participated in your survey and responded honestly, I'd label critical comments as making me feel "unsure or annoyed" and then you'd label them as "unwelcoming", which is... nonsense. It doesn't follow from the feelings expressed at all, and there's something profoundly wrong with a survey methodology that would reliably conclude that the kind of comments I'd most like to receive make me feel unwelcome.

5. It would be interesting to know the nationalities of the survey takers, and how results varied by nationality

There are substantial differences in standards of politeness between the US and the rest of the world. US call center workers and service staff in shops and restaurants are cheerier than anywhere else I've traveled; this cheeriness famously makes foreigners uncomfortable. This difference in national culture - in which typical American politeness seems saccharine to non-Americans, and the typical politeness of some other cultures seems curt to Americans - seems highly relevant to a discussion about the politeness of comments.

As such, I'd be interested in the distribution of nationalities that participated in your survey, and in the ways you could detect that nationality affected a participant perceived the comments. I find it odd that you chose in the blog post to comment on the participants' sexual preferences and the color of their skin, but not about this characteristic that I'd intuitively expect to be much more predictive of their comment ratings.

6. The definition of "unwelcoming" established here isn't logically compatible with the rules on "unwelcoming" conduct in the draft Code of Conduct (unless you want to suspend most of your active users, which you don't)

The new proposed code of conduct spells out that:

  • "Unwelcoming language" is "unacceptable"
  • Intent is irrelevant to this; if you post an "unwelcoming" comment simply through ignorance of how your wording would be received, it's still misconduct
  • "Repetitive misconduct" will be addressed by an account suspension

Then these classifier results come along and establish that apparently, uh, this is unwelcoming:

The code you posted cannot yield this result. Please post the real code if you hope to get any help.

Or this:

Also, any time you have enumerated columns, you can be sure that something’s gone very, very wrong with your design. That said, you’re probably after LEAST(). But don’t do that. Fix your design.

Or this:

Please provide a full compilable sample if you want anyone to be able to help you: https://stackoverflow.com/help/mcve. I have already told how you can bind to the property. If you can’t make it work, you are doing something wrong.

These are comments that go out of their way to help the recipient and don't clearly contain any deliberate condescension. If they are "unwelcoming", I have certainly posted hundreds of "unwelcoming" comments. It's not obvious what number is needed to count as "repetitive" misconduct, but I've certainly passed it! I expect if I picked any user with hundreds of comments and looked at just their most recent page of comments, I'd find comments with a harsher tone than any of the above. Heck, let's try looking at some of Shog9's comments (usernames censored by me; otherwise copied verbatim):

Far harsher and more personal:

If you're gonna go through the last five years of history, polish up those editing skills @user1 - I'm doing this on a phone, I know you can do better.

Insinuating that someone hasn't read the answer:

Because append() doesn't operate on the jQuery object, @user2 - it operates on the elements contained in the jQuery object. By definition, an empty jQuery object cannot append()... Please read the answer above for why and how to do what the asker intends.

Curt, and giving instructions:

Put together a test case & ask a new question about that, @user3

Look, I don't personally think that any of those comments cross any lines of civility. But I do think they're at least as curt as some of the official examples of "unwelcoming" comments we now have. What exactly are we supposed to make of it when you on the one hand tell us that we will be suspended for "repetitive" posting of "unwelcoming" comments, and then flesh out a standard of "unwelcoming" so harsh that even staff members are clearly and regularly over the line?

I'm guessing the answer is that either the clause about "unwelcoming" comments in the CoC isn't going to be enforced, or that the sort-of-definition of "unwelcoming" established by this blog post isn't going to be used for interpreting the CoC. But I still find this objectionable. We shouldn't have to read between the lines in this way; it's worrying, and frustrating, to have standards articulated under which I and many other users should be suspended the instant the new CoC comes into force, and have no clarity on whether that's an outcome that anyone even remotely intends, and otherwise no idea precisely what part of what we've been told we're not supposed to take seriously.

26
  • 1
    Have you read through this post about the enforcement of the code of conduct? Commented Jul 10, 2018 at 22:44
  • 28
    Regarding using this tool to define "unwelcoming"... 57 people have used it thus far. That's a good proof of concept, but it is a tiny sample - over 15 thousand people got new comments in their respective inboxes on Stack Overflow just since yesterday. Also... Nothing against my dear coworkers, but we're almost certainly a skewed sample. The blog post mentions that we're gonna try to enlist more participants - I'd take any results with a huge grain of salt until we do.
    – Shog9
    Commented Jul 10, 2018 at 23:40
  • 1
    I think Shog9's comments come across as worse than they are due to the lack of context. Here are the links to actual comments in case you wish to see the context: first, second and third Commented Jul 11, 2018 at 5:29
  • 33
    @NisargShah unfortunately, Blog & Tutter Inc. don't care about context. The most brain-dead, abusive, deadbeat qustion ever is nothing - only the comment criticising it is blog-worthy:( Commented Jul 11, 2018 at 6:43
  • 62
    This: "if you post an ;unwelcoming' comment simply through ignorance of how your wording would be received, it's still misconduct" is, effectively, an instruction to "Never comment at all because you can not be sure that any comment, no matter how worded, will be received as 'welcoming". Commented Jul 11, 2018 at 6:49
  • 20
    I mean, the concept that it's automatically misconduct to suggest that OP's can be wrong and can have bad designs is unreasonable. All error-messages would be misconduct, and compilers/linkers would be suspended:( Commented Jul 11, 2018 at 6:54
  • 4
    Some might say that the 'unwelcoming' net is being cast so wide that almost anyone could get caught, and so the mods/CM can then choose as they like which fish to throw back. The worry might then be that new accounts will swim away, and curators will be frozen, filleted, canned and eaten. Commented Jul 11, 2018 at 8:42
  • 30
    @Shog9 If you're gonna use any user input to define "unwelcoming" y'all are going to be in for a rough time. The userbase of this site is diverse enough for there to be contradictory definitions of unwelcoming for different people. But I'm sure you know that and the data scientists know that too, so what is this exercise for?
    – Magisch
    Commented Jul 11, 2018 at 11:02
  • 6
    That's probably a question you should post separately, @Magisch. But... IMO, the biggest potential advantages lie in identifying areas where we can potentially improve the comment system: either via a classifier that prompts authors to reflect when their frustration level grows, or perhaps just a system that engenders less frustration to begin with. Consider that we currently prompt users to "continue in chat" when a back and forth thread emerges... Perhaps we'd be better off prompting participants to go for a walk, drink a cup of tea, do something that makes them happy...
    – Shog9
    Commented Jul 11, 2018 at 14:57
  • 3
    First thing I thought while reading the post - "I want to test the model. Where's the text box?". I bet we'll see funky results like php being more snarky than python. I also agree with Shog9 - I don't think you can extrapolate from 3,992 comments to "7% of comments on Stack Overflow are unwelcoming".
    – Kobi
    Commented Jul 12, 2018 at 7:45
  • 5
    @MartinJames “All error-messages would be misconduct, and compilers/linkers would be suspended”. Well, that’s the reason so many people don’t read them and go directly to Stackoverflow, just to get the comment “the error message is pretty self-explanatory”. Of course, that’s unwelcoming, at least as much as the error message…
    – Holger
    Commented Jul 12, 2018 at 8:27
  • 13
    @Kobi And yet, that's exactly what happened. We're not supposed to analyze without more data, but we get an official blog post doing exactly that. I'm not a fan of the double standard that seems to be happening here.
    – fbueckert
    Commented Jul 12, 2018 at 13:45
  • 2
    "If someone were to engage in outright abusive behavior in comments [...] they would certainly not succeed at upsetting me." Okay, well consider yourself very lucky to have such thick skin; for most people such insults would be very upsetting. Especially from strangers.
    – TylerH
    Commented Jul 12, 2018 at 16:12
  • 1
    @ryanyuyu I think you're missing nuances here; I haven't in fact mentioned sample size anywhere. I'm pointing out methodological problems that I think invalidate the entire data-gathering exercise, and would continue to invalidate it even if they, say, extended the survey to 10000 users; that's different to just arguing that the quantity of evidence is insufficient. As for point 6, I guess I left it to last because it's long and kind of tangential; it's arguably more about the CoC than it is about Julia and Jason's research, which this question is nominally feedback upon.
    – Mark Amery
    Commented Jul 12, 2018 at 19:45
  • 4
    @MarkAmery Hardly; I frequent both of those environments daily and it's easy to not see that because you control who you hear or talk to in those environments. Stack Overflow does not have a blocking feature. Someone says "f you man" to me, that would be upsetting because it is far outside the bounds of normal polite or professional discourse. I am not thin skinned, I was raised on the internet practically. I have trawled the depths of Reddit, Digg, 4Chan, UseNet, IRC, etc. Foul language there is part of the deal. Here it is not, nor should it be. Here, it's upsetting and unwelcoming.
    – TylerH
    Commented Jul 12, 2018 at 19:50

5 Answers 5

62

Personally, I think this trying to classify comments is entirely missing the main reason that people tend to feel unwelcome. I'm pretty sure that the main frustration people have is with their questions being downvoted, closed, or ignored. In my opinion, this focus on unwelcoming comments is trying to solve the Y in the XY Problem instead of getting to the true root of the problem X.

From the comments on the blog post:

In my personal experience, it’s the behavior encountered that is more unwelcoming than any specific statement or comment. One Stack Exchange site I was new to closed my first question with little or no explanation and then punished the one person who was trying to help me. I then went on meta to better understand what I did wrong, and no one replied to me. That’s far more unwelcoming than receiving a somewhat snarky comment after hours of help (I totally agree with the folks above who mentioned that context is very important).

Newbies don't understand all the etiquette and how to ask a question properly right away. No amount of policing of comments is going to help them feel more welcome if they never have positive interactions where they get the help they're looking for. On the other hand, that's often not possible because the help they are looking for is off-topic or poorly asked.

I think the Ask a Question Wizard is a great step toward solving this conundrum and approaches along that line are far more likely to yield significant positive results than trying to force commenters to be extra careful about people's feelings.

3
  • 9
    I think that this whole process has clearly illustrated that comments aren't the problem. There is a problem with being unwelcoming, but it's the wikipedia editor problem: Arcane codes of conduct. Selective enforcement. Jealously guarded domains. etc. Commented Jul 11, 2018 at 22:56
  • 10
    I also think there is too much emphasis on comments.There were always complaints about downvotes in general, downvotes without comments, close votes in general, and duplicate close votes. I'd like to believe the worst comments are already removed, and while I agree we would be better off without snarky comments, I doubt this will solve the problem or improve the experience by much.
    – Kobi
    Commented Jul 12, 2018 at 7:29
  • 1
    This is certainly the answer, though this is something the science and math stack exchanges have more of a problem with than SO, SO has an issue with a few power users that need to be tamed, but overall when I see some one go to meta, they are met with help pretty readily. Posting on Mathematics SEcan sometimes be a nightmare, and Physics is probably one of the worst offenders of this IMO. Can see this with downvotes and no feedback though. On SO people get so fatigued and it is hard discerning between bad faith and good faith, so its a structural problem here, not a community problem.
    – Krupip
    Commented Jul 13, 2018 at 20:54
54

I originally posted this over here, as I'd missed this meta topic entirely.

Whoa SO, slow down a moment.

Comment Evaluator 5000

Where was this? How was it advertised? WAS it advertised? This intro page sure looks like it was meant to pull in a large sample of people from Stack Overflow to get involved. I sure as heck have never seen this before.

We had 57 participants

Oh, no apparently it was not advertised or even openly available. Heck, this doesn't even cover a quarter of SO's own employees (much less the 120,000 or so people who've gained at least 100 rep this year).

I have a feeling that more people (and more comments) should have been involved in this ratings process. Because of the fact that tone, in text, is a matter of interpretation you need to get a wide spectrum of readers to participate. I would even then break down the results by age of the account to see if there's an interpretation difference between new users and "old foagies."

On top of that, identifying the comments that 99% of folks thinks are fine, but That One Guy rated as frustrated or angry. Because the problem isn't that people make rude comments (those people can be dealt with), it's the new user who sees a comment, interprets the worst, and leaves telling his two friends "Stack Overflow sucks, they're all elitist jerks" because someone asked him to provide a MCVE. Fixing that is your stated goal, isn't it? Rather than punishing the commenter: teach the reader. Find a way to inform this person that "no, that comment isn't rude, our site has guidelines for asking good questions, and this person is trying to inform to that fact."

Then we get this chart:

Ratings chart

This looks pretty typical, lots of abusive comments, some less than ideal comments, and some that are fiiiii--wait a minute! What was that vertical axis labeled again? "Number of raters." As in the number of people who rated the comment. Several of these have 1 or 2 ratings, most have less than ten!

How is this graph even statistically significant?

And now that I look at the name of the X axis, I'm not even sure what kind of data this chart is even supposed to be showing. "100% of comments with each rating" had 7 raters...huh? What does that even mean? How does a comment get a single rating that leaves it at "75%"? Even if we assume that the X axis is supposed to be "acceptability" or "average of all ratings" the choices are "outright hostile" "unsure" and "fine." A single entry of any one of those should not leave a comment at "75% fine."

This comment said that the chart is actually trying to say "6 people marked 100% of the comments as 'fine', 1 person marked 75% of the comments as 'fine', 51 people marked 0% of the comments as 'abusive', etc." Which took them 10 minutes to puzzle out. This is still not a great chart even using that reading, "So one observation is that the raters couldn't decide on how many posts were unwelcoming - the orange distribution is w-i-d-e."

Now you have me convinced that you're deliberately trying to deceive the community. For what purpose, I don't know, but I know that Stack Overflow knows how to analyze data and present good charts; you have been doing it for years with the developer survey. This on the other hand...this is really sloppy. I don't know what to say.

I'm confused

18
  • bmm6o's reading of the graph is how I initially read it, too.
    – TylerH
    Commented Jul 12, 2018 at 16:28
  • @TylerH In not-reading the graph axis, its very easy to see it as a "% nice : number of comments" bar graph, which is what I expect to see from a "we had people rate comments" type of data. Arranging things by "% reviewed as nice : number of people" the graph ends up misleading. E.g. the blue and orange areas are basically the same, but mirrored (the 5 people who rated a non zero number of comments as "very bad" alters the comparison slightly) which tells you nothing about the data gathered. Its equivalent to saying that "80% of votes on YT videos are positive, ergo YT has good content." Commented Jul 12, 2018 at 16:34
  • 2
    to read it that way you'd have to first ignore the blog post itself which means thousands of comments being rated by 57 people. Also, you shouldn't consider a viewpoint which only makes sense when you don't read the labels on the axes... just saying. I agree it is not the best or most sensible graph though.
    – TylerH
    Commented Jul 12, 2018 at 16:42
  • 5
    "thousands of comments being rated by 57 people" still makes me expect a chart that shows the average rating of the comments, not the number of comments rated a given way by person. Especially when prior to the chart we have "Rating: % of comments: Fine 92.3%, ..." which lines up nicely with the peak of the blue in the chart. Commented Jul 12, 2018 at 16:49
  • 1
    Like I already said, i agree the chart is not the best, I'm just pointing out how it doesn't make sense to interpret it they way you initially seem to have done.
    – TylerH
    Commented Jul 12, 2018 at 18:39
  • 4
    Good observations, thank you. I wonder how "a data engineer (Jason) and data scientist (Julia)" can conduct a 'study' like this and even think about the results having any significance. True, the blog talks about caveats. But still the results are presented in a way that sound like that since the analysis is done, the execution phase can begin. I am not convinced.
    – user9455968
    Commented Jul 12, 2018 at 20:04
  • @LutzHorn: "But still the results are presented in a way that sound like that since the analysis is done, the execution phase can begin." I'm curious as to how exactly you can draw that conclusion, when the blog specifically says that the next step is to get it in front of more people. Where is the wiggle room in "We will be fielding this comment classification task more broadly soon, in order to learn more about how our community understands interaction via comments."? Commented Jul 12, 2018 at 21:21
  • 13
    @NicolBolas Oh, probably the "We'd like your feedback by July 11, 2018" from the last blog post...They wanted our feedback on their new rules before releasing the data driving the new rules or--heck--gathering more data. Cart meet Horse. Commented Jul 12, 2018 at 21:26
  • 2
    I'm not saying that they're lying. I'm saying that the order in which they're doing things doesn't make any sense. That is: they're changing the rules before they have solid data on what the problem is. Commented Jul 12, 2018 at 21:48
  • 2
    @Dukeling That was written prior to knowing how the chart was supposed to be interpreted. I was completely baffled at what I was supposed to get from the chart. That said, the fact that the Code of Conduct changes have already been proposed and commented on (the deadline for comments was yesterday by the way) and that the unimaginably small dataset they used to craft those new CoC rules (even admited to by Shog9) still leaves me baffled. [1/2] Commented Jul 12, 2018 at 21:55
  • 5
    [2/2] I don't think that it's a deliberate attempt to obfuscate, but that I'm left feeling like it is. That's what the last paragraph is: My perceived feelings about what's going on; things being done backwards, a code of conduct update using incomplete data, a collection of "bad comments" that aren't...actually that bad being touted as "now against the rules." The end result is I feel convinced that someone is pulling wool over my eyes. Commented Jul 12, 2018 at 21:56
  • 1
    @Draco18s: "The end result is I feel convinced that someone is pulling wool over my eyes." OK, let's start from there: SE is trying to deceive you. To what end, exactly? They're trying to deceive you... into being nicer? Into posting comments that aren't so aggressive? What is the end-game here? Commented Jul 13, 2018 at 3:41
  • 3
    @NicolBolas If I knew that, there wouldn't be wool over my eyes and is be able to say, "Hey, you're lying." But I don't know that. Commented Jul 13, 2018 at 4:00
  • 10
    If I were to assume, arguendo, that the chart was deliberately deceptive, I'd say the end goal would be "to make you believe there's more of a problem than there is, so that you'll agree that the CoC changes are needed". I don't actually think the chart was deliberately deceptive. But I do think it was accidentally deceptive, that is, that the SO staff are deceiving themselves into believing there's more of a problem than there really is. And the examples they cited are another piece of evidence in that direction: they're seeing a problem, while I say, "Wait, what? THAT's unwelcoming?"
    – rmunn
    Commented Jul 13, 2018 at 6:47
  • 16
    @NicolBolas No, SO isn't trying to deceive people into being nicer (whether said deception is intentional or unintentional), it's trying to deceive people into not posting entirely appropriate and helpful comments because SO thinks that entirely polite and professional comments are "unwelcoming", and trying to get moderators to delete said helpful and appropriate comments for the same reason. Removing (or preventing the posting of) helpful content, under the false claim that it's not nice, is rather problematic.
    – Servy
    Commented Jul 13, 2018 at 21:35
35

The Comment Classifier post is a joke, right? It was intended to be published on April 1st, but somehow got delayed?

Let's see if I can sum up what happened:

  • A data scientist and data engineer teamed up to conduct a study.
  • They designed an input tool that provided three choices, two of which they consider negative.
  • They hand-picked a small sample of comments that they predetermined would probably be considered unwelcoming based on their biased opinion as SE employees (see next point).
  • They fed those comments to a handful of SE employees that were already biased based on the (unproven) belief that we're too negative (see next point).
  • They asked those already biased employees to pretend that they're new users here, when clearly that small group of employees is not representative of new users of the widely varying cultural groups that visit the site. (Hey. You know we're trying to prove that the site is unwelcoming to new users. Here are a bunch of comments that we've culled out that we think are unwelcoming. Pretend you're new users, look at these unwelcoming comments we've collected for you (that we already said are unwelcoming) and try to think of how those new users would interpret these (as we already told you) probably (almost certainly) unwelcoming comments, will you?)
  • They then interpreted that extremely small group of users rating a hand-picked selection of comments with mostly negatively biased options from which to choose to reflect some sort of (cough, cough) valid sampling that can judge the current state of the site, and produced a totally meaningless graph of those highly unscientific, inaccurate and skewed results to say we're all meanies.
  • They wrote a blog post to try to convince everyone that their conclusions are correct (See, we told you so! Here's proof!), and we're all a big bunch of rude bad people who drive away all of the poor little new users who can't be bothered to learn how the site works, post useless clutter and noise here, and add no value to the site, and we should all be suspended or banned if we've been here more than a couple of weeks or months and use any word that a single individual on the planet might think sounds unwelcoming or critical in a comment.

Have I got that about right?

23
  • 6
    "Have I got that about right?" Um, no. There's no indication that all the comments they were shown were intended to be unwelcoming. As evidenced by the fact that only 7% of them were ultimately deemed unwelcoming. If what you're accusing them of is true, wouldn't the percentage be much higher? And the people were not given just a comment: they were given the entire question/answer plus all of the comments. Commented Jul 13, 2018 at 3:33
  • 2
    "we're all a big bunch of rude bad people who drive away all of the poor little new users who can't be bothered to learn how the site works, post useless clutter and noise here, and add no value to the site, and we should all be suspended or banned if we've been here more than a couple of weeks or months and use any word that a single individual on the planet might sound unwelcoming in a comment." Hyperbole of this sort is not helpful. If you genuinely think that this post is such a scathing attack on the community, then you've never seen an actual scathing attack. Commented Jul 13, 2018 at 3:33
  • 9
    I respectfully disagree. We estimated how many comments we could rate, given the number of folks we have internally and the time we would be asking of them. Then we loaded up the right number of comment threads into the application and invited all of our community managers, designers, developers, executives, site reliability engineers, and product managers to spend an hour rating comments. We had 57 participants who made 13,742 ratings on 3,992 comments. Miniscule sample of the right number - What, 15 minutes worth of comments? - judged in an hour by our available staff?
    – Ken White
    Commented Jul 13, 2018 at 3:55
  • 18
    How is any sort of meaningful, unbiased scientific study conducted by a fraction of an hour's site content (the comments) based on the judgement of a minimal number of people who happened to be available and also employees of the company who were given an entire hour to rate something with three possible choices, only one of which is considered positive (and where those 57 participants made 13K ratings on 4K comments (in a whole hour)? Seriously? Do the math yourself. The hyperbole is in that blog post.
    – Ken White
    Commented Jul 13, 2018 at 3:59
  • 5
    The one part of your answer I disagree with is the third point, specifically "hand-picked". If they had hand-picked comments they thought would be considered unwelcoming, I believe they would have gotten a larger number of "unwelcome" ratings than they did. The post said that they decided how many comment threads to load into the tool, but doesn't say how they chose those threads. I would assume random selection unless I saw evidence otherwise, and the classification results would seem to bear out random selection given that 27 of the raters rated at least 95% of the comments as "fine".
    – rmunn
    Commented Jul 13, 2018 at 6:54
  • Apart from that quibble, I agree with you about the pitifully small data set available here. If they wanted good data, they should have picked a random selection of SO users (numbering in the tens of thousands) to ask for feedback, instead of just employees.
    – rmunn
    Commented Jul 13, 2018 at 6:55
  • 1
    @KenWhite "Seriously? Do the math yourself." - alright, it works out at 60/(13000/57/60) ≈ 15 seconds per comment, on average. Seems reasonable for just reading a comment and judging its tone. Most comments on main are short.
    – Mark Amery
    Commented Jul 13, 2018 at 8:21
  • 1
    I'm puzzled by the multiple people bringing up the sample size as a criticism of this exercise. Leaving aside the ways that the survey was systematically biased, do you really think that if you got a sample of 57 randomly-selected users to rate 4000 randomly-selected comments, you'd get a significantly different proportion of comments rated "unwelcoming" to if you did it again with a different 57 random users and 4000 random comments? I haven't done any maths on the problem, but intuitively that seems... very unlikely to me. 57 users and 4000 comments are decent numbers.
    – Mark Amery
    Commented Jul 13, 2018 at 8:26
  • 12
    The key phrase in your answer is: "we're trying to prove that the site is unwelcoming to new users". To me this seems to be exactly what is going on. I am not as sure as you that the "study" was rigged that way. But your base assumption that the SO/SE staff is of the opinion that something has to be done to combat the asserted "unwelcomingness" of the site seems obvious to me. We never read a blog post that questioned if the site has such a problem. They always only talked about that there is a problem and that now it will be taken care of. No, thank you.
    – user9455968
    Commented Jul 13, 2018 at 8:27
  • 2
    @MarkAmery On a site the scale of SO, I'd expect 100.000 users normally distributed across rep levels and a couple million comments to be actionable as far as datasets go.
    – Magisch
    Commented Jul 13, 2018 at 9:32
  • 4
    @MarkAmery - As I already mentioned, I don't think 57 comment reviewers was a decent number given that they could have had literally tens of thousands of people instead of 57. Furthermore, all 57 were employees, which inherently biases the results. A company employee is more likely to agree with the corporate culture since any who vehemently disagree would probably have already resigned. Whereas picking reviewers from among SO users would have produced results from people holding a much broader swathe of opinions about what is or isn't "welcoming", and thus been less inherently biased.
    – rmunn
    Commented Jul 13, 2018 at 9:55
  • @MarkAmery - Oh, I see you already addressed the way the survey was systematically biased.
    – rmunn
    Commented Jul 13, 2018 at 9:56
  • 6
    @MarkAmery: The only way I see that this survey or study or whatever you want to call it could have been more poorly conducted would have been if they went to Twitter and asked the users who are whining there to do the rating of the comments. SE is painting the users who built (and continue to maintain) this site with a broad brush of being bullies and meanies and downright rude. They damn sure are forgetting the hundreds of thousands of us who brought them here in favor of a handful of users whining and complaining on Twitter.
    – Ken White
    Commented Jul 13, 2018 at 12:40
  • 4
    @AnsgarWiechers I wonder if we will ever know. Did anybody official from SE comment on the topic here on MSO in the last few days? Some community moderators have commented but where is the reaction by the people who triggered all this? Where are the answers by the "study" authors or those at SE that are responsible? Are they just waiting for us to get tired? Will they then present new "facts" and conclusions?
    – user9455968
    Commented Jul 13, 2018 at 20:38
  • 8
    I didn't see any official response. And I doubt we'll get one. People at SO (the company) seem to be hell-bent on getting this in place, no matter the cost for the quality of the site. Shame. Commented Jul 13, 2018 at 21:05
-4

I take issue with your third point. If I'm having a discussion with a person on the internet and they resort to personal attacks, I would find that upsetting. Not to a degree that it would ruin my day, of course; after all it's not like I know them personally. But to someone who is in a minority in terms of skin colour and gender in my industry, these comments ARE hurtful.

They're hurtful because, even though they're blatantly not true, they're a reflection of what some people think about people like me. It hurts to be put down personally based on who you are, with no reference to your intelligence or person-hood.

So please, before you make conjecture like that, perhaps you should think about your own perspective and how it differs from people in a minority. I would be willing to bet that you've never been on the other end of a racist tirade. I have. It was from a person I'd never met before and I never want to meet again, but it nonetheless had me in tears.

I want to stress that I'm not trying to pick a fight; I understand your perspective, however I wanted to bring my own forward and point out that not everyone will have a thick skin and certainly might find such comments hard to bear.

8
  • I'm not sure there's any incompatibility between what you're saying here and what I'm saying in point 3. My conjecture is that most people are not upset by blatantly abusive behaviour directed on them on the internet; your assertion is that members of racial minorities frequently are, and that belonging to a racial majority makes it easier to have a thick skin. I can easily imagine a world (whether or not it's the one we actually live in) in which are both right, and (for instance) a racist tirade would tend to make racial minorities "feel angry" or upset but not racial majorities.
    – Mark Amery
    Commented Jul 12, 2018 at 23:33
  • Fair point, however even if the racism wasn't directly pointed at me, I would still be angry on behalf of the person receiving the comment, so I would classify that as "angry or upset". I would definitely be angry about any sexist comment, whether it applied to me or not. But that's simply my perspective.
    – Lauraducky
    Commented Jul 12, 2018 at 23:36
  • Fair enough as well - and, in the same way, I imagine that there are plenty of straight white men who would be personally angered to see homophobic/racist/sexist abuse directed at someone, regardless of their individual characteristics. (Indeed, I'd expect bigotry to be more likely to incite such a response than just plain old cussing at or threatening someone, perhaps precisely because of a sense - right or wrong - that members of groups classically targeted by bigots are, as you say, more likely to find such abuse hurtful.)
    – Mark Amery
    Commented Jul 12, 2018 at 23:40
  • 17
    Comments that are described in that third item (racist, sexist, profane or intentionally offensive or abusive) are very quickly dealt with here by almost every single user who has the ability to flag or vote. They're very rare to encounter, and when they are they're quickly flagged and deleted.
    – Ken White
    Commented Jul 13, 2018 at 1:22
  • 1
    FWIW, I'm kind of irritated to see the unexplained downvotes here, on an answer that was relevant, brought a valid and currently little-visible perspective, and made a clear effort to be courteous in its dissent on a sensitive topic. I realise we have a long-established norm that explaining downvotes is non-mandatory, but when downvoting a post that is not obviously bad and in which the author has bared some degree of personal vulnerability, would it really have hurt to explain why you disagreed? I can make some guesses at what exactly the five downvotes here might mean... but only guesses.
    – Mark Amery
    Commented Jul 13, 2018 at 13:13
  • 8
    @MarkAmery I downvoted because I think this is not an answer to your question as I read it. Your question is is not about racist tirades or similar attacks. Those I detest and I welcome them being deleted quickly. My upvote for the comment by Ken White matches this downvote.
    – user9455968
    Commented Jul 13, 2018 at 13:35
  • 2
    @MarkAmery: FWIW, I did not downvote this post. I'm not sure of its relevance to the topic at hand, because the kind of comments that are described and discussed are not tolerated here ever, never have been, and are almost always immediately noticed and removed if they appear (as I mentioned in my previous comment to this post). The answer is well written and has a valid point of view. My comment was left just to point out the way that those sort of posts are rapidly removed.
    – Ken White
    Commented Jul 13, 2018 at 20:48
  • @MarkAmery I didn't downvote either, but I agree with Lutz that this answer doesn't really relate to your question. As Ken pointed out, rudeness or abusive behavior is not tolerated by the vast majority of SO users, and usually dealt with rather switfly. Commented Jul 15, 2018 at 11:05
-30

Those example comments are spot-on "unwelcoming" when I read them:

The code you posted cannot yield this result. Please post the real code if you hope to get any help.

Why can't the code yield the result? OP must have made a mistake composing a minimal example, but what is it? The commenter clearly knows more than the OP, but doesn't feel like sharing. Also, they question whether OP actually wants help with the problem they came here to get help with.

Also, any time you have enumerated columns, you can be sure that something’s gone very, very wrong with your design. That said, you’re probably after LEAST(). But don’t do that. Fix your design.

OP's design, which is presumably the best they could do, is "very very wrong". How could it be fixed? The commenter might know, but they're not telling. OP obviously should just know that already. OP is sad and feels bad at computers.

Please provide a full compilable sample if you want anyone to be able to help you: https://stackoverflow.com/help/mcve. I have already told how you can bind to the property. If you can’t make it work, you are doing something wrong.

Again, we start out by questioning whether OP is here for help or just to waste our valuable computer programmer time. Then we complain we already told OP how to solve their problem, and that it's their own dang fault if they didn't understand or didn't manage to apply the solution correctly. We certainly aren't going to keep helping them, since they've already proved their incompetence.

I'm a bit dubious of an SO-enforced policy of mandatory niceness, but if we do want to bring people who don't (yet) know what they're doing into the community, we may need it. We're trying to run a professional Q&A forum, and any written feedback we give to people should be constructive and friendly, even if we don't necessarily feel that way when reading their questions.

17
  • 14
    I read the first one as "The code that you posted cannot yield the problematic result that you also posted." i.e. there is a mismatch between the code and the results both of which were posted by the OP, so only the OP can know how to fix the question. Usually that means by posting the real code, not by posting some fake example that doesn't illustrate the problem.
    – shoover
    Commented Jul 12, 2018 at 22:12
  • The comment's not wrong; OP needs to come back with code and a result that match. They're being asked to do that in a way that comes off as brusque and uncaring.
    – interfect
    Commented Jul 12, 2018 at 22:15
  • I'm not taking issue with the comments about the tone of that comment; I'm taking issue with your claim that the commenter knows more than the OP about why the code and the result don't match. "The commenter clearly knows more than the OP, but doesn't feel like sharing."
    – shoover
    Commented Jul 12, 2018 at 22:23
  • 27
    I disagree. If a post claims that code posted produces an output that the code could not possibly produce, it's not rude of the commenter to point that out by saying The code you posted could not produce the output you claim. It's a flat, non-opinionated, non-rude statement of fact. If I posted code that clearly could not do what I said it did and asked people to help me fix it, I'd expect to be called out on it. What would you expect to be said instead? Dear poster, I hope to not offend you, but could you pretty please (with sugar on it) post different code to demonstrate the problem?
    – Ken White
    Commented Jul 12, 2018 at 22:30
  • 21
    I think all three of these analyses fall foul of a fallacy that is strangely common in this discourse: the presumption that the commenter is vastly cleverer than the asker, knows exactly what mistake the asker has made, and isn't revealing it purely out of contempt for the asker. Much more likely: commenter 1 doesn't know what the real code looks like or what the mistake in the sample's composition was; commenter 2 doesn't know what the best way to improve the design in the asker's particular case is; commenter 3 doesn't know what the asker is doing wrong because they haven't shown any code.
    – Mark Amery
    Commented Jul 12, 2018 at 22:31
  • 2
    OMG. I mistakenly could have offended someone with my alternative suggestion, which used the endearment Dear, which someone might find unwelcoming because it's overly personal. Give me a break.
    – Ken White
    Commented Jul 12, 2018 at 22:31
  • Regarding point 1, I often refrain from telling people why their fake code couldn't produce their claimed output if I think that they're likely to just re-fake the part I point out. These are often cases where the situation involves environmental factors no one but the questioner will be able to replicate, like mis-installed packages. If they re-fake the code to remove the signs, I may not be able to tell whether the problem is due to some unseen environmental factor or whether the code is a fake that doesn't accurately represent the real problem. Commented Jul 12, 2018 at 22:46
  • 3
    What Ken said. I have more than once had a situation where the code in a question couldn't possibly have produced the result the OP claimed it had. Which was often enough easily demonstrable by simply running the code from the question. But I fail to see how that would have given me any insight in the actual code that the OP decided to not disclose. Commented Jul 12, 2018 at 23:27
  • 1
    @KenWhite Yes, when you are directly contradicting someone, or telling them that they made a mistake, doing so without taking a deferential tone can be taken as rude and unwelcoming. Instead of saying something like "The code you posted doesn't compile", or, even worse, "You posted code that doesn't compile", it is more polite to say something like "I wasn't able to compile the code you posted". Many people gravitate towards software to avoid this kind of social nonsense, but it's quite hard to get away from.
    – interfect
    Commented Jul 12, 2018 at 23:33
  • 12
    But yes, if you intentionally give everything the worst possible interpretation with a biased attitude, then you can find anything offensive. I can find offense in the word Stack used as the site name, if I so choose. It would be ludicrous to do so, but is SE going to change it if I post on Twitter and get one person to agree that it's offensive? How about 100? How about 200? How far does the nonsense extend? Reading something with the intention to find it offensive or unacceptable is simply ridiculous. I can find 20 people in my office who would find offense in the tag brainfuck.
    – Ken White
    Commented Jul 12, 2018 at 23:58
  • 4
    So does SO ban the use of that tag, and questions about that language, because the creator decided to give it a profane name? No, they don't, even though the vast majority of professional offices in the US find the term unacceptable. Oh, wait - do I risk suspension for using the name of that language in my comment if it makes you uncomfortable, even though it's a publicly displayed tag on this site?
    – Ken White
    Commented Jul 13, 2018 at 0:00
  • 2
    FWIW, @KenWhite, while I'm broadly on your side here I still think that telling a user "You should be home letting mommy coddle you" pretty unambiguously is over the line into the sort of rudeness and personal insult that we should be flagging and mods should be deleting (and which, if repeated, should warrant a word from the mods). I'd flag the comment in this case, but thought - especially given we're in the context of a discussion about comment civility - it'd be more useful to simply point that out.
    – Mark Amery
    Commented Jul 13, 2018 at 14:19
  • 6
    @MarkAmery: I have never (nor would I ever) written that anywhere other than in this post. It was used here to stress a point, which was that if someone has such thin skin as to be made to feel offended or unwelcome for being asked to provide details about a technical problem, they shouldn't be interacting with strangers on the internet. If I wrote that in an actual comment in response to an SO question, I'd flag it personally and suspend myself. Read it in the context of the rest of my comment here, with the rest of what I wrote, rather than taking it out of context.
    – Ken White
    Commented Jul 13, 2018 at 14:23
  • 9
    There's nothing wrong with stating a fact as a fact "The code you posted doesn't compile". This is becoming ridiculous, people cannot be chained up in their speech to a point of not being able to state facts.
    – user3956566
    Commented Jul 14, 2018 at 1:11
  • 2
    To add a data point (b/c it just came up): right now there is a question where someone is trying to run a jar file and getting an error from the SQL Server JDBC driver. When I asked for the java commandline they were running (since they had obfuscated it to oblivion on their question) they responded with a literal copy/paste of an example commandline from the Liquibase documentation. For the ORACLE JDBC driver. Needless to say I askedt them to now post their real code. Commented Jul 14, 2018 at 16:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .