My answer to your last question was primarily focused on volume - specifically, on how the volume of flags has varied over the past few months, possibly in connection with various events and system changes. But you did ask for a "helpful" percentage, and I provided it... And as you've observed, that percentage has dropped a bit.
So today, let's try & dig into that a bit. My goals here are as follows:
- Focus specifically on Rude, Unwelcoming or Something Else comment flags raised on Stack Overflow during the month of August
- Put that percentage in context (what does the drop mean in terms of total helpful flags?)
- Identify specific contexts where problems are occurring (user class, flag type, comment type)
Let's start with...
Some raw numbers
First, let's re-run the query I used here for the month of August:
Overall comment stats
951,890 comments were created during August, of which 100,965 - just shy of 11% - have been deleted at the time I'm writing this. Those deletions break down as follows:
Reasons DeletedComments
--------------------------------------------------------------------- ---------------
Self-deleted 68338
Automated deletion 11807
CommentNoLongerNeeded 9968
Moderator discretion 8513
CommentUnwelcoming 954
Comment Other 335
Comment Rude Or Offensive 292
Comment Rude Or Offensive, CommentUnwelcoming 50
CommentNoLongerNeeded, CommentUnwelcoming 46
Comment Rude Or Offensive, CommentNoLongerNeeded 15
Comment Other, CommentUnwelcoming 7
Comment Other, CommentNoLongerNeeded 4
Comment Other, Comment Rude Or Offensive 3
Comment Rude Or Offensive, CommentNoLongerNeeded, CommentUnwelcoming 2
Comment Other, Comment Rude Or Offensive, CommentUnwelcoming 1
Now, this only looks at comments created in August; it's useful as context, but it doesn't tell us anything about flags on older comments or flags which were declined. So let's get some broad stats for those too:
Overall comment flag stats
34,676 comment flags were raised during August, of which 33,245 were marked Helpful. 33,053 of the flags were raised on comments that were later deleted - this is important to note, both because of cases where the flag would've been declined but then the comment later deleted for another reason... And for the cases where comments would later be undeleted, thus leaving a helpful flag on a visible comment. Those flags break down by type as follows:
Type FlagsCreated Helpful On comments which were deleted
------------------------- ------------ ------- ------------------------------
Comment Rude Or Offensive 918 666 718
CommentUnwelcoming 2572 1922 2004
CommentNoLongerNeeded 30221 29999 29702
Comment Other 965 660 629
Couple of things worth digging into here:
- the Rude / Unwelcoming flags are the only categories where the comments are more likely to be deleted than the flags are to be marked helpful.
- "other" (currently named "Something Else" on the sites) is far and away the least likely flag to be marked Helpful.
We'll get back to those later. Next, let's separate out those numbers by context.
Just in time comment flag stats
The first subsection above looked at comments created in August; the second looked at flags created in August. What about flags raised in August on comments in August?
Type FlagsCreated Helpful On comments which were deleted
------------------------- ------------ ------- ------------------------------
Comment Rude Or Offensive 766 580 603
CommentUnwelcoming 2003 1513 1502
CommentNoLongerNeeded 10723 10559 10282
Comment Other 646 428 397
The big difference here from the last table is a massive drop in the number of No Longer Needed flags - which stands to reason, as an awful lot of comments get less useful as time rolls on, so there's somewhat less need for them immediately. Of course... There are also folks who make a hobby of finding older comments to flag, so that feeds in here too.
A more subtle difference is that "Other" flags seem to be slightly more useful on contemporary comments. Not by a lot, but... Another reminder to dig into this later.
One more thing I'd like to get some broad numbers on before digging in though:
Comment flags by type of handler
There are five ways by which comment flags are normally "handled" - and only one of them can decline flags:
- Moderators can delete comments (helpful) or dismiss (decline) comment flags
- Moderators, privileged users, and post authors themselves can delete the post on which flagged comments reside, thus immediately marking the flags as helpful (without, strictly-speaking, deleting the comments themselves).
- If the author of a comment deletes it, any flags on that comment are marked helpful.
- The system will automatically delete a comment if it gets a certain number of flags, marking all those flags helpful in the process.
- The system will automatically delete a comment the first time it is flagged if the comment contains certain words or patterns - this instantly marks that flag as helpful.
(there are other ways for flags to be dismissed, but they're far less common)
So let's see how those break down for each type of flag for flags raised in August...
Type Handled by FlagsCreated Helpful
------------------------- ---------------------------------------- ------------ -------
Comment Other comment automatically deleted 1 1
Comment Other post deleted by vote 2 2
Comment Other comment deleted by author 26 26
Comment Other post deleted by author 26 26
Comment Other flag handled by moderator 872 567
Comment Other flagged comment deleted based on keyword 31 31
Comment Other post deleted by review 7 7
Comment Rude Or Offensive post deleted by review 7 7
Comment Rude Or Offensive post deleted by 3+ flaggers 81 81
Comment Rude Or Offensive comment deleted by author 37 37
Comment Rude Or Offensive post deleted by author 55 55
Comment Rude Or Offensive flagged comment deleted based on keyword 71 71
Comment Rude Or Offensive flag handled by moderator 649 397
Comment Rude Or Offensive post deleted by vote 18 18
CommentNoLongerNeeded post deleted by 3+ flaggers 48 48
CommentNoLongerNeeded comment automatically deleted 1 1
CommentNoLongerNeeded flag handled by moderator 17882 17661
CommentNoLongerNeeded comment deleted by author 342 342
CommentNoLongerNeeded post deleted by review 25 25
CommentNoLongerNeeded post deleted by vote 52 52
CommentNoLongerNeeded post deleted by author 214 214
CommentNoLongerNeeded flagged comment deleted based on keyword 11656 11656
CommentUnwelcoming post deleted by review 10 10
CommentUnwelcoming comment deleted by author 70 70
CommentUnwelcoming post deleted by author 165 165
CommentUnwelcoming flag handled by moderator 2071 1422
CommentUnwelcoming flagged comment deleted based on keyword 111 111
CommentUnwelcoming post deleted by vote 35 35
CommentUnwelcoming post deleted by 3+ flaggers 109 109
Observation here? For everything except No Longer Needed, the number of inaccurate flags being handled by moderators is worse than suggested by the previous tables. "Other" starts to actually look pretty reasonable...
Last thing we need here...
Comment flags from low-rep flaggers
One of the major changes (at least in terms of potential) over the past few months was in allowing low-rep users to flag comments on their own posts. Let's look at the stats for only those flags during August:
Type FlagsCreated Helpful On comments which were deleted
------------------------- ------------ ------- ------------------------------
Comment Rude Or Offensive 132 48 85
CommentUnwelcoming 262 147 150
Comment Other 121 50 42
Well, that's pretty terrible compared to... Even the previous table. OTOH... There aren't very many of these flags; that's about 12% of all rude/unwelcoming/other flags raised during August. It's more than I originally expected, but not insanely high. However, these flags constitute 22% of all declined rude/unwelcoming/other comment flags - that's a pretty sizable chunk!
Anyway. Enough confusing raw data.
Let's dig in...
First, let's have a look at how the number of Rude, Unwelcoming or Other comment flags as handled by moderators has varied over the past few months...
This has the same time period and annotations as the charts I posted the other day... But this is just flags on Stack Overflow, and just flags handled by moderators as helpful. Oh, and it's not a percentage - it's the actual count of flags per week.
...You'll notice that it still goes up. We're getting more useful flags. The cost is some additional number of not-so-useful flags, but we're at least getting something in exchange.
With that in mind, let's try & figure out if there's a pattern here for the sorts of comments that are Not Helpful. We've already touched on low-rep flaggers above, so I'm gonna look at two specific scenarios hinted at by your examples...
Flags on system-generated comments
There are three types of system-generated comments to consider here...
Comments generated by Low Quality review when a reviewer selects a canned comment from the Delete or Recommend Deletion dialog. Those collected 84 flags in August, broken down as follows:
Type FlagsCreated Helpful On comments which were deleted
------------------------- ------------ ------- ------------------------------
Comment Rude Or Offensive 5 2 2
CommentUnwelcoming 9 6 1
CommentNoLongerNeeded 56 56 48
Comment Other 14 10 6
Comments generated when a close vote or flag is raised, to provide a link to an original question while the question is not yet closed. There were 160 of these flags:
Type FlagsCreated Helpful On comments which were deleted
------------------------- ------------ ------- ------------------------------
Comment Rude Or Offensive 3 0 1
CommentUnwelcoming 22 4 5
CommentNoLongerNeeded 84 45 47
Comment Other 51 17 18
Comments generated when an "off-topic" vote is cast with a custom explanation provided by the voter. There were 57 of these:
Type FlagsCreated Helpful On comments which were deleted
------------------------- ------------ ------- ------------------------------
Comment Rude Or Offensive 8 4 4
CommentUnwelcoming 14 5 7
CommentNoLongerNeeded 27 24 22
Comment Other 8 4 6
All together, these account for about 8% of declined comment flags, and about 7% of declined rude/unwelcoming/other flags.
Other flags
The "other" / "something else" option has the distinction of having the second-highest number of declined comment flags by volume and the second-lowest percentage of flags marked helpful.
This is still a relatively small number of flags overall, with about 300 declined for August... But let's see how those declined flags break down by keyword frequency:
194 the
150 is
138 to
115 not
102 this
89 comment
82 a
79 question
75 i
70 and
66 answer
60 it
51 in
44 of
37 my
36 for
34 an
31 duplicate
31 comments
30 be
Two-word phrases:
31 this comment
29 the question
28 is not
19 comment is
18 this is
18 it is
16 an answer
15 to the
15 my question
14 does not
13 question is
13 a duplicate
12 i dont
11 not a
11 a comment
10 the answer
10 of the
10 in the
10 but it
9 to be
Three-word phrases:
11 this comment is
7 it is not
7 answer the question
6 to answer the
6 this is an
6 not a duplicate
6 an attempt to
5 this is not
5 not an answer
5 my question is
5 attempt to answer
4 to the question
4 to do with
4 this comment says
4 the question this
4 nothing to do
4 is not a
4 is an answer
4 i dont think
4 comment is not
I suppose we shouldn't be surprised, after seeing the results for system-generated comments above, that an awful lot of "other" flags are raised to dispute duplicates.
It's maybe a bit more interesting that a fair number of "other" flags are also raised on comments that try to answer the question... We kinda expected this might need to be addressed at some point, and it looks like that is the case.
Action items
First, I think it's important to remember that these are still relatively small changes. But, that doesn't make them any less irritating: these flags tend to be treated as high-priority, so even a small jump in noise means less time devoted to situations where a moderator's intervention is actually needed urgently - sorta the flag equivalent of crank calls to 911.
With that in mind, I'd recommend the following:
Blacklist duplicate patterns in Other flags
This should be easy: match on some trivial expression (say, \bnot\s+(a\s+)?duplicate\b
) and reject the flag with some nice message about editing:
If this question has been misidentified as a duplicate, please edit it: make the title specific to this problem, and then reference the other question explaining why the answers there do not solve this problem.
Update: I've added this now:
Better guidance for new / low-rep flaggers
I have a sneaking suspicion that some non-trivial number of new users think "flag" means "reply". When we ran into this before, I changed the description of "other" to "in need of moderator intervention" to kinda emphasize the fact that no one in the conversation sees these flags.
But now "other" is "something else", and nothing else in the flag dialog hints at what flagging actually does... So I guess I could kinda see some folks thinking this is sort of a "reaction" or "postscript" option instead of a means of reporting abuse... So, maybe add a note indicating that no one in the conversation sees these? That, uh, flags are reviewed by third parties who will delete comments as needed? Open to suggestions.
Answers as comments... what do we do with them?
It's probably time to discuss some sort of strategy for handling flags on comments-as-answers. Flagging doesn't accomplish much; then again, neither does declining those flags. We've been telling people for years to create an answer that elaborates on the comments when they see these things, but there's nothing in the UI itself that encourages this - and what sometimes happens instead is that new folks see these comments and think they have to earn the comment privilege in order to answer!
I recommend kicking off another discussion specifically about this problem, with an eye toward identifying UI changes (either for flagging or commenting) that might assist in some way.