32

I just participated in a survey from Stack Overflow called the "comment-classification" and would like to share a couple of thoughts.

For those who were not invited:

  • See https://stackoverflow.blog/2018/07/10/welcome-wagon-classifying-comments-on-stack-overflow/
  • I was presented with the comments on the questions of new users and was asked to rate the comments as if I was the new user and was reading the comments on my question.

  • Ratings to choose from:

    • "The comment is fine",
    • "The comment is disrespectful, sarcastic, or otherwise unfriendly." or
    • "The comment is abusive or contains harassment."
  • The comments were batched per question and did not include the text of the question (Just bare, anonymised, comments).

Thoughts/findings:

  • I did not mark any comments as "The comment is abusive or contains harassment."
  • Of the comments that I marked as "The comment is disrespectful, sarcastic, or otherwise unfriendly." I would consider none as disrespectful nor sarcastic.
  • Nearly half of the comment were from the OP; this makes the question "Imagine that..." a bit strange: I am commenting on my own comments.
  • There are an awful lot of comments (over 50% easily) that should not be there (answers, discussion)
  • The survey did not allow me to skip a comment when classification would require the context of the question.

These findings make me second guess whether or not the effort should be put into "making comment friendly again".

That call is not up to me but I would argue that the quality of the Questions and Answers would greatly benefit from turning comments that should be answers into answers, and moving discussion comments to chat. Perhaps we should focus on that.

9
  • 3
    One could dispute if sarcasm (without belittleling the OP themselves) should be seen as unfriendly in general. Well depends on the wording and the humorous part of it. Commented Aug 28, 2018 at 18:12
  • 28
    Detecting sarcasm was really hard for me because the Question was not available in the survey. E.g., if a comment states "Use css" it could be sarcasm if the question is "how do I do this without using css?" but the question could also be "Should I use jQuery or css?"
    – Emond
    Commented Aug 28, 2018 at 18:20
  • 7
    "Detecting sarcasm is a really tough job, people don't mean how mean they mean something." -- Sheldon Cooper (source: www.fakecites.com) Commented Aug 28, 2018 at 18:22
  • @ErnodeWeerd Or let me try in other words: For me sarcasm somehow falls into the category of Deep Thought's answer to "Life, Universe and everything" is 42. What an awesome KI. Commented Aug 28, 2018 at 18:50
  • "I did not mark any comments as "The comment is abusive or contains harassment."" Makes me wonder how the sample takes deleted comments into account. As of recently flagged comments for rude/abbusive tend to be removed fairly swift. If they only took comments that are visible on the site, this may drastically skew results in favour of the wagon.
    – Luuklag
    Commented Aug 30, 2018 at 12:52
  • @Luuklag - this is one of the reasons for me to post here: I doubt the validity of this test.
    – Emond
    Commented Aug 30, 2018 at 13:35
  • 1
    Doctor Sheldon Cooper. Don't you agree that his credentials does make his words more trustworthy @πάνταῥεῖ ?
    – EMBarbosa
    Commented Aug 30, 2018 at 13:37
  • Little disappointed that I was not asked to be a part of the survey considering that I'm one of the top 20 users of the tag. shrug Oh well.
    – ouflak
    Commented Aug 31, 2018 at 8:10
  • @EMBarbosa He would probably write 'Sheldon Cooper, PhD' when writing it out, though.
    – TylerH
    Commented Aug 31, 2018 at 14:26

2 Answers 2

27

Truly abusive comments are rare, 0.3% of those identified when SE ran through this exercise internally, and the vast majority of comments are composed of discussion, clarification, or attempts to help the asker. It's therefore not surprising that you'd see few of these in a small sample size. Even the less-abusive "unwelcoming" comments accounted for only 7% of those in their sample, and there was quite bit of scatter from person to person on what fell into that category.

However, as anyone who has received one of these can tell you, these negative comments have a disproportionate impact. You can receive 100 comments of praise, and the one insult is what you'll dwell on. That's human nature.

On a site this size, even a low-percentage event can become a common occurrence. People cheating the voting system with sock puppets and voting rings are also a tiny fraction of the userbase, but we still can deal with dozens of them a day. The negative impact they have on the site makes it worthwhile to develop better tooling to deal with them.

The vast majority of comments don't need any action at all. Even the cases you bring up (discussion, attempts at answering) may not require any action. Both are usually positive actions taken in an attempt to help the asker or clarify an answer. You can look around on Meta to see many examples of people who were very upset about deleted discussions or otherwise helpful comments, so increasing removal of these may not be well-received by many.

Even with abusive or insulting comments being low-frequency events, I do think it is worth the effort to examine and deal with them. A survey like this is useful to quantify the scope of the problem and what the community as a whole perceives as unwelcoming or abusive. Community moderation does a very good job at present of identifying the worst comments for moderators to act on, but I'm all for finding ways to improve the process.

1
  • 5
    "and the one insult is what you'll dwell on." And it's the one you'll share to others... Commented Aug 30, 2018 at 8:10
18

Thanks for your participation in this project! We appreciate your time and feedback. All moderators on Stack Exchange will be invited to participate, as well as a sample of folks from our research list (you can opt in to this list via your email settings) and a sample of folks who are our users but don't consider themselves active participants here.

Your experience of finding rude/abusive comments to be quite rare (i.e. not seeing any during the time you spent classifying) lines up with what we are generally finding; this is good news about our community and moderation.

We are quite open to exploring options for how comments are handled on our network. I particularly want to use data to understand when/where/if they are more helpful than harmful on balance. Some of the specific issues you bring up are quite important, alongside the issues around inclusion, tone, and professionalism.

13
  • 4
    Perhaps I am too naive but the easiest way for me to prevent issues around inclusion, tone and professionalism is by never referring to the user but always to the question/answer. This does not prevent anyone from taking offense but it is easier to stay on topic and stay factual.
    – Emond
    Commented Aug 28, 2018 at 18:58
  • 1
    That absolutely sounds like a great approach, and we talk about focusing on the content not the person in the code of conduct: stackoverflow.com/conduct On our network we still sometimes see problems with comments that are directed at questions/answers, i.e. "this answer is lazy". In my opinion, your principle is necessary but not always sufficient for staying professional.
    – Julia Silge Staff
    Commented Aug 28, 2018 at 19:40
  • 3
    I had no idea the research option was under the email settings - that seems like something that should have been pointed out when this whole thing started Commented Aug 29, 2018 at 1:16
  • 1
    We have talked about the research list via multiple channels for a while now, but our site is definitely complex and it can be hard for a user to catch it all, and for us to communicate it all well! You can read more about email settings, including the research list opt-in, on this blog post by our UX design lead Donna from last November: medium.com/stack-overflow-design/…
    – Julia Silge Staff
    Commented Aug 29, 2018 at 1:23
  • 8
    @JuliaSilge Your example is directed at the user, in an incorrect form that "cleverly" hides this fact - an internet post obviously can not be "lazy" :)
    – BartoszKP
    Commented Aug 29, 2018 at 11:34
  • @JuliaSilge, could you have a look at my comment on the Q in this thread, about what your sample was exactly.
    – Luuklag
    Commented Aug 30, 2018 at 13:41
  • @Luuklag The comments in this classification task are live, non-deleted comments, selected from a recent month (but not but the current month, since many abusive comments are deleted quickly). We know which comments have been flagged and deleted, so the point of this exercise is to understand the characteristics of comments which have not been flagged. We can also compare the groups.
    – Julia Silge Staff
    Commented Aug 30, 2018 at 15:20
  • @JuliaSilge, fair explanation, however wouldn't it be interesting to see what percentage of people would choose to "flag" a now deleted comment. I feel we are playing "better safe then sorry" here, so as soon as a comment gets one, or maybe two rude/abusive flags it will be gone, whilst that might be a verry minority opinion?
    – Luuklag
    Commented Aug 31, 2018 at 7:53
  • Hi @JuliaSilge where can we participate in this survey?
    – TylerH
    Commented Aug 31, 2018 at 14:26
  • @TylerH There are three groups that are being invited to participate in this research task: a) SE mods, b) folks who responded to the survey at the end of Jay's blog post (this survey is now closed): stackoverflow.blog/2018/04/26/… c) a sample of folks on our general purpose research list. You can opt in to the general purpose research list via your SE account email settings. We are running this task on a rolling basis, analyzing results as we go and adjusting sample sizes, etc.
    – Julia Silge Staff
    Commented Aug 31, 2018 at 19:30
  • @Luuklag We are using the characteristics of both sets of labeled comments (flagged/deleted on the site, classified in this task) to compare. We've considered the balance of the value of time spent by classifiers and the ability to statistically compare the content of these groups, and landed on this option.
    – Julia Silge Staff
    Commented Aug 31, 2018 at 19:33
  • @JuliaSilge Thanks; I would love to participate in this because I think I could give highly useful and nuanced feedback. I am already in the general purpose research list via my SE account email, I believe, and I am pretty sure I signed up via the Blog's Google Docs form as well when it was open. I am a little worried about responses from SE mods; I have seen mods (a whole group of mods on one site) who are actually quite hostile/don't grok the concept of moderation that might skew the results a bit.
    – TylerH
    Commented Aug 31, 2018 at 19:33
  • 1
    @TylerH If you signed up on that blog post's survey and said it was OK to contact you, you will definitely get an invite as we are reaching out to that whole list. We will definitely look at differences in perception by group of raters.
    – Julia Silge Staff
    Commented Aug 31, 2018 at 19:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .