45

Kevin and I just posted to the blog about automatically flagging comments on Stack Overflow with The Unfriendly Robot. We also talked about the robot recently here.

After you give it a read we're interested in your thoughts/feedback and would like to answer any questions you might have. Thanks! 🙏

EDIT

Some questions are difficult to answer in the comments so I'm going to reproduce them here and answer them.


From jcsahnwaldt says GoFundMonica

From 2009 until mid-2018, comments could be flagged as "offensive". Happened for ~0.1% of all comments. Now we can flag comments as "unfriendly or unkind". When was the new flag introduced? (I guess in mid-2018?) How many comments are flagged as "unfriendly or unkind"? In your study with moderators and other users, the median person classified ~3.5% as "unfriendly". What was the average "unfriendly" flag percentage in that study? Did you try to measure how much of the difference between ~0.1% and ~3.5% is due to "unfriendly" being a broader criterion than "offensive"?

I agree with you that the definitions of offensive and unfriendly are different and that (by truthiness of gut) the frequency of unfriendliness should be higher than the frequency of offensiveness. We introduced the "unfriendly or unkind" flag in August 2018 in conjunction with the Code of Conduct change, and we kept the "offensive" flag around. What happened to the percentage of posts flagged before and after the change?

Percentage of Posts Flagged Each Day, Monthly Distributions

This a monthly box plot of the percentage of posts flagged each day, by flag type. In red is the offensive flag which corresponds to the box plot in the blog post. In blue is the unfriendly-or-unkind flag. When the unfriendly flag was released we see offensive flag usage drop way way down. There was copy change here too so "offensive" isn't what users see anymore, they see this...

enter image description here

Regardless, I think the graph shows the right thing, that what's truly offensive is way lower than what we used to think (if just going by the name of the flag). Unfriendly-or-unkind flag usage jumps up for a few months, and then settles back down to where the old offensive flag was. This is support for the hypothesis that the Stack Overflow system has an underflagging problem. Even when we expanded the definition of what you should be flagged, overall throughput wasn't meaningfully different. I don't think this result gives us any ability to answer "How much of the difference is from the broader definition?", though.

15
  • 6
    Sounds cool! One thing to fear when using NLP is that what's considered rude might vary over time, with changing culture and slang. Do you have plans to continuously/intermittently review and update the model?
    – Erik A
    Commented Apr 9, 2020 at 14:23
  • 7
    @ErikA Absolutely. We don't think our model "thinks". It's very sophisticated pattern matching. Our human flaggers are the only way the robot will ever know about new modes of unfriendliness, so if you see something unfriendly flag it as such. We last retrained in August and we're definitely due. We'll update on that when it happens. Commented Apr 9, 2020 at 14:40
  • 4
    This looks like really good work. Kudos. Commented Apr 9, 2020 at 14:48
  • 12
    I'm just happy you aren't claiming it's AI, like seemingly every other company in the world that uses machine learning. Inference is not intelligence, it's a component thereof.
    – Ian Kemp
    Commented Apr 9, 2020 at 15:38
  • 7
    @IanKemp Yep, you'll probably never hear the Data Team use that term. Dave Robinson, one of our former Data Scientists, has a great writeup on the subject. Commented Apr 9, 2020 at 15:54
  • 15
    the focus on unfriendly comments is fascinating given the influx of incredibly poorly worded questions from new users. i basically have given up answering on Data Science or Cross Validated and i guess i'll give up commenting as well. honestly, what's in it for volunteers at this point? Commented Apr 9, 2020 at 21:57
  • 2
    @DaveKielpinski Surely there are unfriendly comments, but I agree with that they aren't the only issue by far. Question quality doesn't seem to get comparable attention, one could say. Commented Apr 10, 2020 at 12:16
  • 3
    @DaveKielpinski Just because we're talking about comments today doesn't mean we haven't also talked about question asking. Commented Apr 10, 2020 at 12:36
  • 4
    @DaveKielpinski Or that we don't have direct plans for using the tech we've developed here to make question asking even better. Commented Apr 10, 2020 at 12:37
  • 4
    @Kayaman: I'm not gonna be harangued this morning. Your comments aren't being deleted by the robot, they're being deleted by me. Commented Apr 10, 2020 at 13:00
  • 1
    @JasonPunyon okey doke, but this volunteer currently sees significant downside and no upside to commenting / answering / reviewing overall. the influx of new users with poor questions means that it takes a lot more work to answer and new users don't upvote or accept answers. just one person's opinion Commented Apr 10, 2020 at 16:15
  • 1
    Cool, congratulations! I know you've worked on that one for a long time. So... can we test it pretty please? Not asking for much - just a textbox and a score.
    – Kobi
    Commented Apr 11, 2020 at 12:31
  • 1
    Wow! One of the best post I've read in a while. Also thank you for the tip about fast.ai. I'm currently working on an NLP project and I'll definitely take a look at this.
    – A.Game
    Commented Apr 16, 2020 at 20:44
  • 1
    @A.Game Glad you liked it! Hope fast.ai works out for you too. Commented Apr 16, 2020 at 21:43
  • 1
    Nice work! We've come a long way from "be nice" as a guideline, but it's also a little more clear than that -- "Be nice. Treat others with the same respect you'd want them to treat you. We're all here to learn together. Be tolerant of others who may not know everything you know. Bring your sense of humor." web.archive.org/web/20081225024656/stackoverflow.com/faq Commented May 2, 2020 at 18:27

8 Answers 8

30

Has culture been considered in any way as part of this research? In sociolinguistics, offensiveness is not an inherent property of a comment that can be calculated from analysis of vocabulary and grammar, but something that must be understood in the cultural context in which it was uttered.

As an example, in some Australian Aboriginal cultures, referring to dead relatives by name is per se considered highly offensive. If the majority of Stack Overflow participants were from such cultures, it is highly likely that "refers to dead relative by name" would be a valid reason to flag or even a specific named flag or suspension reason ("This account is temporarily suspended for referring to the dead by name"). It isn't, so what gives? Do we want to stick a finger in visitors' faces, telling them,

Declined - We know your deceased father invented that famous algorithm, but we aren't going to stop people from calling it the Roger Smith Machine! We don't follow that culture here and don't recognize its rules on offensiveness! Stop flagging!

Historically, I would have understood an adjudication related to culture-bound rules to revolve around intent or knowledge - e.g. when Mary posted a comment suggesting "Did you try the Roger Smith Machine?", did she know that his son found seeing his father's name online highly offensive and is there any indication that she specifically wanted him to suffer additional distress? The article itself mentions that "circumstance or intent" no longer matters.

  • Did you omit any consideration of culture?
  • Did you assume a general "Stack Overflow Community Culture" based on your understanding of what constitutes current consensus on community norms here?
  • Did you use your own culture to frame your understanding of what offensive content is and is not?

As another example, cultures have a great deal of variance in how appropriate it is to point out an error made by someone else. In some cultures (E.g. Japanese), one must make only vague hints lest one be accused of insensitivity (e.g. "when one programs, one might look out for memory leaks...."), while in others (e.g. Israeli or Scottish), one may generally point out the error explicitly and then launch into a tirade of "friendly" insults ("you forgot to unallocate that pointer, numbskull"). Which culture prevails in a dispute as to whether a comment is offensive?

Your mention of how passersby might interpret an old comment makes me think about culture in a different way. Previously, I understood the intersection of culture and offensive comments on Stack Overflow to relate to the participants themselves - that is, a comment was considered offensive if and only if it was offensive according to the culture of the commentator and/or the intended recipient of the comment. If we allow anyone from any culture to interpret for themselves whether to interpret any comment as offensive, I fear that could lead to a lack of standards. Maybe you and I communicate just fine, but then someone comes along a year later who comes from a culture in which failing to invoke a deity before every comment is considered super-highly offensive, they flag both of us, and boom, we are banned for failing to consider this third party.

  • Can we even be sensitive to all cultures at the same time?
  • Can we be the Culture Police for everyone in the world?
  • Are we to standardize on a single "site culture" with corresponding site-specific social mores and to require everyone to conform to such rules as a condition of participation?

The only real solution I see is to have a single site standard, but then that feeds right back in to the idea that Stack Overflow might just have to be culturally insensitive to some people. Can we live with that?

9
  • 11
    The customs and social norms of particular cultures were not incorporated into this research. Stack Overflow has had oblique "cultural" guidelines in the past ("Be nice.") and we've gotten more prescriptive as time has gone on (now we have the Code of Conduct). Commented Apr 9, 2020 at 19:00
  • 15
    The particular "culture" (if we want to use the term very loosely) that is embodied in the robot is that of comments that were flagged by users in the past and handled by our moderators. It is defined by action, and there's probably some level of caprice (which moderator handled the flag probably impacts how it was handled). Commented Apr 9, 2020 at 19:00
  • 15
    Good point. It's hard to deny that there is some cultural difference between Stack Exchange, Inc. and its users, given the events of 2019.
    – MSalters
    Commented Apr 9, 2020 at 23:21
  • 3
    In this case, the cultural influence on comment friendliness is solely determined by the moderators because the "ground truth" is just what they decided, whatever it was. I guess, moderators coming from all over the world, it's effectively a mixture of the depicted extremes. You can be somewhat direct, but only up to a certain threshold. To summarize: a friendly comment is what moderators deem to be a friendly comment. The culture applied to the decision is the effective moderation culture, whatever it is. Commented Apr 10, 2020 at 12:14
  • 1
    @Trilarion then is there any guidance provided to moderators on what to do in culturally diverse contexts? E.g. "If you are a moderator from Culture A and are handling a flag raised by a member of Culture B on a comment posted by a member of Culture C that was directed at a member of Culture D, you should adjudicate offensiveness according to Culture D standards and no other standard." Commented Apr 10, 2020 at 13:14
  • 1
    I don't know, only moderators will be able to answer that. I guess that in doubt a reasonable guidance could be to be as nice as possible, i.e. rather err on the side of indirectness than directness. On the other side there might be a programmer sub-culture which is a bit more direct than how the mean culture is. The current code of conduct and some Q&A with further explanations for example regarding pronoun usage, may help the moderators too. Commented Apr 10, 2020 at 13:28
  • 5
    There's also the fact that the careful circumlocution which some cultures require for politeness is often impenetrable to speakers of English as a second language, and some level of directness might be required for them to understand what you're saying. (And the fact that Americans perceive themselves as very direct, but actually aren't, so when they meet someone who actually is direct they're taken aback.)
    – TRiG
    Commented Apr 10, 2020 at 13:39
  • 6
    Absolutely agree with this. Stack Exchange Inc. is hopeless at understanding the international dimension; it has a very US-centric perspective. Even the difference between British English and American English is significant; but Indian English is very different from either. Above all, though, the problem is that experienced programmers don't like circumlocution, they like to be factual, accurate, and direct -- and inexperienced programmers can find that unfriendly. If you try and force experienced programmers to circumlocute, you're going to lose the site's main asset. Commented Apr 16, 2020 at 11:03
  • @MichaelKay exactly. It's not too difficult for everyone to agree on or at least agree to comply with a rule not to use "the seven words you can't say on television" with each other. It's another thing entirely to get everyone to understand all of the cultures of the world and how anything they say or do may be interpreted in every and all cultural contexts. Maybe there's a culture out there in which my username "Robert Columbia" is the most vile insult there is. There probably isn't, but what if there is? Should I be banned because I didn't actively seek it out? Commented Jun 19, 2020 at 13:51
21

Interesting read. Detecting unfriendliness is a form of sentiment analysis, and my employer has been doing something similar for the last 20 years so I recognize quite a few of the challenges.

We fully agree with the notion that you can't just take humans out of the loop. But AI is very good at prioritizing work for humans. You can train neural networks to also produce confidence scores. These shouldn't be interpreted as "percentages correct", but only to sort the flags for the human moderators to judge. Start with the ones that the network classifies as unfriendly with high confidence, and work down the list.

As you gain experience with the AI, you may find that the highest confidence scores are certain hits, so you can in fact deprioritize those. Instead, pending human verification you label them as unfriendly.

One area we're actively looking into is the ranking of sentiment. Your current classifier is a binary classifier, because you train it on two classes. Something is either friendly or unfriendly. But your human baseline can be more nuanced than that, and your AI can learn that.

As you might notice, unlike Jason Punyon and Dave Robinson, I have no hesitation using the term AI. To me that is a broad category. I agree that this is also Machine Learning, and also Pattern Recognition. Those are overlapping subfields of AI.

3
  • 14
    *psst* they're using regex
    – S.S. Anne
    Commented Apr 9, 2020 at 17:36
  • 3
    Our labels are binary, but the model outputs a probability which we then threshold. From the last graph here you can see the percentage of flags that were accepted conditioned on score (above the threshold) for the past two models. Even 100's are only accepted 90% of the time, I don't think we're at the point where the best usage of limited moderator resources is working on the bubble. Commented Apr 10, 2020 at 12:59
  • 2
    @JasonPunyon: In the setup of humans judging AI results, it can be useful to distinguish 3 outcomes: Right, Wrong, Agree to disagree. In a sense that's anthropomorphizing the AI, but the underlying reason is that reality isn't that black and white, and some comments would be a much greater loss if rejected by the AI than others. A comment that's not unfriendly but just unhelpful? Don't worry if the AI rejects it for the wrong reasons.
    – MSalters
    Commented Apr 10, 2020 at 15:11
15

It would be useful to have a FAQ or help page on how to write comments in a more respectful and constructive way.

I know this is briefly gone over in the CoC, but it would be nice to have a list of what's seen as "unfriendly" and what's not, and some examples. Perhaps there might be a way to word constructive comments in a way that's not seen as rude.

8
  • 8
    This is something I absolutely want to see done. Commented Apr 9, 2020 at 18:36
  • Maybe the unfriendly comments can be clustered in some way. I have no experience with NLP, but maybe there is some kind of formula of what makes most of the unfriendly comments unfriendly. Commented Apr 9, 2020 at 22:00
  • 11
    Argh, please, no! Every time that staff or community members have tried to give examples of polite or impolite comments, swathes of the community have disagreed with the characterisations, often to the point of having directly opposite opinions (such that they found the "polite" version insulting and the "unfriendly" version respectful). Trying to codify what is polite is a fool's game, and having the staff do it will in effect mean encoding the linguistic norms common among wealthy US progressives as rules and punishing people of different backgrounds for their different habits of speech.
    – Mark Amery
    Commented Apr 10, 2020 at 10:25
  • 1
    It's one thing to try to codify or automate detection of comments so bad that they should be deleted on sight, but it's quite another to try to codify the fuzzier and more contentious question of what's respectful, and it's another yet again to ask an authority with the power to censor and punish to issue such a codification by decree. We've gone over this ground again and again and it's caused nothing but conflict and bitterness; surely we should've learned by now?
    – Mark Amery
    Commented Apr 10, 2020 at 10:28
  • 8
    @MarkAmery I agree but surely it's better to keep those cultural clashes out in the open under active discussion, than have them as unspoken, unwritten rules that non-US users will constantly hit? Obviously the ideal would be an FAQ / blog that acknowledges different norms and explains that if, for example, you're American and feel something is rudely blunt, or you're German and feel it is patronisingly gentle, they might just have different norms to you... Commented Apr 10, 2020 at 13:02
  • 2
    @MarkAmery At least we'll have an idea of what they see as "polite". Maybe we could word our comments so they don't seem impolite, or at least don't trigger the bot.
    – S.S. Anne
    Commented Apr 10, 2020 at 13:28
  • 1
    I think people have to bite their tongue. I bite my tongue every time someone calls me "bro"; I have to force myself to realise that they probably thought they were being friendly. I certainly wouldn't want everyone who uses the term banished for being unfriendly or impolite. Commented Apr 16, 2020 at 11:09
  • 1
    @MichaelKay: Rather than biting your tongue, I'd just politely ask them to refrain from the behavior that bothers you - assuming it's directed at you, anyway. (If not directed at you, of course, it often makes more sense to simply say nothing unless it still offends you or makes you feel unwelcome.) Most people are willing to abide by what you ask. If they intentionally continue to engage in the behavior, well, that's when I'd consider whether it was worth flagging...
    – V2Blast
    Commented Apr 25, 2020 at 9:30
13

I feel that you are fighting symptoms without attempt to treat root cause. When moderators just removed the negative comments without any feedback, it does not improve much welcomeness of the sites.

The commenters are not informed, that they should use more polite language/tone and continue their behaviour.

The person to whom the potentially unfriendly comment was addressed ( usually the author of the post, but may be one of previous @commenters) is not aware that SO/SE is not tolerated such comments and still consider SO/SE unfriendly.

It will be good if commenter will receive feedback after each removed comment that it was considered as unfriendly to ask the commenter to reconsider the language.(see related discussion Comment deletion notification)

The person to whom the unfriendly comment was addressed also should be informed that the unfriendly comment has been deleted - it will highlight to the users, that SO processes are friendly despite than some commenters are not.

The feedback should appear as a notification, visible only to addressed users, not as another comment, visible to everyone. Notification should include the link to the parent post ( question or answer) and text of the comment ( to be clear what was deleted)

2
  • 5
    If the classifier is cheap to run, then it should run when someone posts a comment, and if it flags it, it should prompt the poster with a "Your post may be considered unfriendly; do you really want to post it?" confirmation dialog. This would weed out many of the posts that were not intended to be offensive. OTOH, this will provide the trolls with a mechanism to precisely modulate their offensiveness! Commented Apr 15, 2020 at 15:02
  • 2
    I think it is important to make some attempt to meaningfully identify to the user in what way their comment is a problem (identify aspect of the CoC?). And if possible bland, automated messages should be avoided in favor of polite suggestions on how to be less offensive? That is, if the objective is a better /community/ and not simply erasure of insufficiently friendly people by proxy, then there has to be some engagement not just a rap on the knuckles and a kind of censorship. Commented Apr 16, 2020 at 2:12
9

Thanks for the article! Very clear, informative, and transparent. Here are a few questions:

  • From 2009 until mid-2018, comments could be flagged as "offensive". Happened for ~0.1% of all comments. Now we can flag comments as "unfriendly or unkind". When was the new flag introduced? (I guess in mid-2018?) How many comments are flagged as "unfriendly or unkind"? In your study with moderators and other users, the median person classified ~3.5% as "unfriendly". What was the average "unfriendly" flag percentage in that study? Did you try to measure how much of the difference between ~0.1% and ~3.5% is due to "unfriendly" being a broader criterion than "offensive"?

  • Are you trying (or planning) to measure how the percentage of unfriendly comments affects user experience? Is there a correlation (which might indicate causation) between percentage of unfriendly comments and percentage of users who say they perceive Stack Overflow as unfriendly or unwelcoming?

  • In my personal experience a scientific study (n=1) I found that I care more about how my contributions to a platform (e.g. Stack Overflow, Wikipedia, a forum...) are received (get upvotes, are deleted, get useful responses, ...) than about the tone of comments. Are you measuring how such experiences affect the impression of friendliness/unfriendliness of SO? If it turns out that users perceive downvotes, close votes or having their questions deleted as unfriendly, are you going to discourage such votes / actions?

  • What are the numbers for comments that got flagged both by humans and the robot in the two periods?

3
7

Thank you for sharing your latest efforts and results on computer assisted comment curation. Well knowing the hot mess that some comments can turn into, I hope to see this system become progressively better at handling comments and hopefully their comment removal faster when needed.

The What's next? section shows that you're interested in continuing research with a special focus on sentiment analysis. The way I see it, such a system in practice could have also contributed to the quick detection and clean up of comments which are not constructive. Alas, the comment classification initiative under the Welcoming Wagon only considered a categorization based on "friendliness" (OK, unfriendly, abusive), rather than overall usefulness. In other words, there would have been value in introducing a category for comments which could be removed at any time without losing information, like in the Higgs application at SOBotics. Food for thought nevertheless.

I also have one topic that I would like to see clarified: Are moderators making substantial false positives in their unfriendly comment handling?

This statement caught my attention:

We know that not all of the comments the robot flags are unfriendly and moderators do accept some of those flags because the comment is worthy of removal for other reasons.

This seems to contradict how comment flags from humans are usually handled: they can be declined if the comment should still be removed but is not "hot" enough to merit an unfriendly flag. Some moderators are even known to have a user-script to facilitate this 2-step operation. Do you think that the number of accepted unfriendly flags on non-unfriendly comments in the data set is non-negligible still?

4
  • 6
    To answer the question in your last paragraph: yes, because not all moderators use the userscript you described. And not all moderators go through the song-and-dance of declining a flag, then going back to the page and deleting the original comment anyway. They especially don't do this dance for a bot.
    – Cody Gray Mod
    Commented Apr 9, 2020 at 17:20
  • The last paragraph seems like a moot concern as mods vary wildly in what they consider worthy of removal when it comes to comments.
    – TylerH
    Commented Apr 9, 2020 at 20:50
  • 5
    It's a concern for me. If my responses are being used to validate the bot's flags and I'm on a machine that doesn't have the user script (ie. work computer...I need a break once and a while), it's easier to just delete those comments that should be gone but aren't rude. On the other hand, if I'm on a personal machine, I have that user script so I will decline/delete at the same time. Both remove the comment, which is the ultimate goal, but one is sending a bad signal to the bot.
    – Andy Mod
    Commented Apr 10, 2020 at 3:09
  • 4
    @Cody (and Andy but I can only notify one of you): I'm gonna open up the discussion on decline and delete today. Commented Apr 10, 2020 at 13:20
3

Time will tell, but I wonder if those that won't be driven off by potentially unfriendly comments won't be balanced by the people that will be driven off because their bona fide comments are refused by some inscrutable AI.

How many people on SE aren't native English speakers and how do you distinguish a friendly Uzbek trying his/her best from an uncivil American?

This kind of site works because there is tolerance. Both ways.

2

Thanks for the interesting blog post. Some random thoughts (I don't have experience with NLP).

  • Is it planned to take into account context like the comments immediately before and after each comment? My feeling is that sometimes tension builds up gradually until someone kind of crosses the unfriendliness border. It might be important what was said before or after (mostly before I would guess).
  • If I understood that right, moderators do mark 3.5% of random comments as unfriendly and about 70-80% of an automatically preselected set of 1% of all comments as well as 70% of a manually preselected set of 0.14% of all comments. What about increasing the workload even more and maybe feeding the moderators 2% of all comments (if that is a feasible workload), some even maybe randomly selected? That should result in more training material of the kind that we may be missing and even more removed unfriendly comments. If it's not feasible permanently, it should at least be possible for short times like feeding the 10% of most suspicious comments of a single day and then identifying which new unfriendly comments appear that the robot would normally not detect and overweight them in training.
  • I wonder how long unfriendly comments live on average on the site before they are deleted? Is it minutes, hours, days? And how many people have seen them before they are deleted approximately?
  • What about feeding those comments where the robot is very, very, very sure that they are unfriendly and very likely to be deleted to a separate review queue that has a higher priority and actually hides the comments from view until they are cleared by a moderator (awaits moderator approval shown to comment poster)? That way the decision is still with a moderator but the damage is further minimized.
  • Just to be sure, it could be that moderators with an increased supply of potentially unfriendly comments have also lowered or increased their threshold for unfriendliness. Is there any indication of a changing ground truth, for example by feeding comments from the past that were already reviewed within some kind of audit?
  • If I remember correctly, the UC-R1 did show a lowered performance towards the end of its lifetime. Is there a similar effect visible for version 2 so far, i.e. is the performance constant or changing significantly with time?
  • Finally, from the blog post it seems that it's not always clear why a moderator deleted a comment. Are comments that are deleted by moderators but were not flagged (for example when just reading content and not working a review queue) included as training data? Maybe moderators should have different delete buttons (delete because unfriendly, delete because spam, delete because other... maybe 3-4 most common options). That could make the robot even better in the future.
1
  • 1
    I'm glad we are beyond the "we cut the number of unfriendly comments in half" statement. If the underflagging is true, we don't really know how much the unfriendly comments have been reduced. The important feature to tout about would indeed have been the detection rate which went up dramatically. That surely made SO significantly more friendly, even though it's not clear by how much. It's also not that important, one can just try the maximum possible and leave it at that. Commented Apr 10, 2020 at 9:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .