Julia Silge and Jason Punyon just posted Welcome Wagon: Classifying Comments on Stack Overflow. There doesn't seem to be a dedicated post for giving feedback on it, so I thought I'd post my own.
I take a series of issues with it:
1. It'd be far more useful if we could see the raw data.
There's a whole bunch of interesting analysis we could do if we had access to the full set of comments that were classified, and the (potentially anonymised) classifications given by each individual staff member. In particular, we could:
- Classify the same comments ourselves, and see to what extent our personal sensibilities align with those of the staff.
- Extract out the set of comments conveying some particular point - like, for instance, that the question is missing an MCVE - and see which ways of phrasing the point were most likely to be taken as unwelcoming (or indeed see whether the phrasing matters at all).
- Analyse the extent to which respondents agreed on which comments were unwelcoming.
2. The word "unsure" in the sentence "I feel unsure or annoyed" has two plausible meanings
I could take "I feel unsure" to mean either:
- Reading this comment would make me feel insecure / unsure of myself, or
- I am unsure how to classify this comment.
I took it to mean the former, and only saw the latter interpretation when it was pointed out to me in chat. Which did you intend? Are you sure your survey respondents interpreted it in that way?
3. The notion that "abusive" comments are ones most likely to make readers "feel angry or upset" is dubious.
If someone were to engage in outright abusive behavior in comments - tell me to go fuck myself, spew racial slurs, threaten me with violence or whatever - they would certainly not succeed at upsetting me. At worst I would find them mildly irritating. I certainly can be personally upset by comments on the internet - heck, I have been recently - but the sort of stuff that would warrant a "rude or abusive" flag on Stack Exchange ain't gonna cut it.
I suspect most people are similar. The sort of social behavior we're most likely to find hurtful is frequently not the blatantly outrageous conduct that all our peers will agree is wrong, but instead something that is at the line where others will find it acceptable - that's a key part of what makes it hurtful. But this facet of human nature makes labeling the most upsetting or anger-inducing comments as the most "abusive" a dubious approach at best, and an outright contradiction at worst; if your survey respondents' feelings work anything like mine, and they followed the instructions literally, the majority of the most blatantly abusive comments won't've been classified as such.
4. The notion that a comment that makes me feel "unsure or annoyed" is bad is dubious
If somebody claims that I got something wrong in a post, I'm probably gonna feel unsure or annoyed. I don't like getting things wrong, and, like, at the very least I'm going to immediately feel unsure about whether I actually made a mistake or not. Tone is irrelevant, here; I don't think there's any way you could criticise a post of mine that wouldn't at least briefly induce this emotional response.
Despite this, I think criticism is valuable, and I greatly appreciate it when others point out errors in my posts. Such comments are the most useful comments I ever receive.
If I participated in your survey and responded honestly, I'd label critical comments as making me feel "unsure or annoyed" and then you'd label them as "unwelcoming", which is... nonsense. It doesn't follow from the feelings expressed at all, and there's something profoundly wrong with a survey methodology that would reliably conclude that the kind of comments I'd most like to receive make me feel unwelcome.
5. It would be interesting to know the nationalities of the survey takers, and how results varied by nationality
There are substantial differences in standards of politeness between the US and the rest of the world. US call center workers and service staff in shops and restaurants are cheerier than anywhere else I've traveled; this cheeriness famously makes foreigners uncomfortable. This difference in national culture - in which typical American politeness seems saccharine to non-Americans, and the typical politeness of some other cultures seems curt to Americans - seems highly relevant to a discussion about the politeness of comments.
As such, I'd be interested in the distribution of nationalities that participated in your survey, and in the ways you could detect that nationality affected a participant perceived the comments. I find it odd that you chose in the blog post to comment on the participants' sexual preferences and the color of their skin, but not about this characteristic that I'd intuitively expect to be much more predictive of their comment ratings.
6. The definition of "unwelcoming" established here isn't logically compatible with the rules on "unwelcoming" conduct in the draft Code of Conduct (unless you want to suspend most of your active users, which you don't)
The new proposed code of conduct spells out that:
- "Unwelcoming language" is "unacceptable"
- Intent is irrelevant to this; if you post an "unwelcoming" comment simply through ignorance of how your wording would be received, it's still misconduct
- "Repetitive misconduct" will be addressed by an account suspension
Then these classifier results come along and establish that apparently, uh, this is unwelcoming:
The code you posted cannot yield this result. Please post the real code if you hope to get any help.
Or this:
Also, any time you have enumerated columns, you can be sure that something’s gone very, very wrong with your design. That said, you’re probably after LEAST(). But don’t do that. Fix your design.
Or this:
Please provide a full compilable sample if you want anyone to be able to help you: https://stackoverflow.com/help/mcve. I have already told how you can bind to the property. If you can’t make it work, you are doing something wrong.
These are comments that go out of their way to help the recipient and don't clearly contain any deliberate condescension. If they are "unwelcoming", I have certainly posted hundreds of "unwelcoming" comments. It's not obvious what number is needed to count as "repetitive" misconduct, but I've certainly passed it! I expect if I picked any user with hundreds of comments and looked at just their most recent page of comments, I'd find comments with a harsher tone than any of the above. Heck, let's try looking at some of Shog9's comments (usernames censored by me; otherwise copied verbatim):
Far harsher and more personal:
If you're gonna go through the last five years of history, polish up those editing skills @user1 - I'm doing this on a phone, I know you can do better.
Insinuating that someone hasn't read the answer:
Because
append()
doesn't operate on the jQuery object, @user2 - it operates on the elements contained in the jQuery object. By definition, an empty jQuery object cannotappend()
... Please read the answer above for why and how to do what the asker intends.
Curt, and giving instructions:
Put together a test case & ask a new question about that, @user3
Look, I don't personally think that any of those comments cross any lines of civility. But I do think they're at least as curt as some of the official examples of "unwelcoming" comments we now have. What exactly are we supposed to make of it when you on the one hand tell us that we will be suspended for "repetitive" posting of "unwelcoming" comments, and then flesh out a standard of "unwelcoming" so harsh that even staff members are clearly and regularly over the line?
I'm guessing the answer is that either the clause about "unwelcoming" comments in the CoC isn't going to be enforced, or that the sort-of-definition of "unwelcoming" established by this blog post isn't going to be used for interpreting the CoC. But I still find this objectionable. We shouldn't have to read between the lines in this way; it's worrying, and frustrating, to have standards articulated under which I and many other users should be suspended the instant the new CoC comes into force, and have no clarity on whether that's an outcome that anyone even remotely intends, and otherwise no idea precisely what part of what we've been told we're not supposed to take seriously.
php
being more snarky thanpython
. I also agree with Shog9 - I don't think you can extrapolate from 3,992 comments to "7% of comments on Stack Overflow are unwelcoming".