Skip to main content
added 211 characters in body
Source Link
Shog9
  • 159.4k
  • 177
  • 1.2k
  • 1.2k

...unfortunately, the current blacklist is ignored by comment flags. Oops.Update: I've submitted a patch to correctadded this. now:

flag text: "this is not a duplicate". Response: "Don't flag mistaken duplicate suggestions; instead, edit the question: make the title specific and detailed, then explain why the answers to the other question do not solve this problem."

...unfortunately, the current blacklist is ignored by comment flags. Oops. I've submitted a patch to correct this.

Update: I've added this now:

flag text: "this is not a duplicate". Response: "Don't flag mistaken duplicate suggestions; instead, edit the question: make the title specific and detailed, then explain why the answers to the other question do not solve this problem."

phrase list fix
Source Link
Shog9
  • 159.4k
  • 177
  • 1.2k
  • 1.2k
3231                   this comment
29                   the question
2928                   is not
19                   comment is
18                   this is
18                   it is
16                   toan theanswer
1615                   anto answerthe
15                   my question
14                   does not
13                   aquestion duplicateis
1213                   questiona isduplicate
1112                   noti adont
11                   inot don'ta
11                   a comment
10                   the answer
10                   of the
10                   in the
10                   but it
9                    to be
11                  this comment is
7                   it is not
7                   answer the question
6                   to answer the
6                   this is an
6                   not a duplicate
6                   an attempt to
5                   this is not
5                   not an answer
5                   my question is
5                   attempt to answer
4                   to the question
4                   to do with
4                   this comment says
4                   nothingthe toquestion dothis
4                   isnothing notto ado
4                   is annot answera
4                   iis don'tan thinkanswer
4                   commenti isdont notthink
4                   comment is not
32                   this comment
29                   the question
29                   is not
19                   comment is
18                   this is
18                   it is
16                   to the
16                   an answer
15                   my question
14                   does not
13                   a duplicate
12                   question is
11                   not a
11                   i don't
11                   a comment
10                   the answer
10                   of the
10                   in the
10                   but it
9                    to be
11                  this comment is
7                   it is not
7                   answer the question
6                   to answer the
6                   this is an
6                   not a duplicate
6                   an attempt to
5                   this is not
5                   not an answer
5                   my question is
5                   attempt to answer
4                   to the question
4                   to do with
4                   this comment says
4                   nothing to do
4                   is not a
4                   is an answer
4                   i don't think
4                   comment is not
4                   comment
31                   this comment
29                   the question
28                   is not
19                   comment is
18                   this is
18                   it is
16                   an answer
15                   to the
15                   my question
14                   does not
13                   question is
13                   a duplicate
12                   i dont
11                   not a
11                   a comment
10                   the answer
10                   of the
10                   in the
10                   but it
9                    to be
11                  this comment is
7                   it is not
7                   answer the question
6                   to answer the
6                   this is an
6                   not a duplicate
6                   an attempt to
5                   this is not
5                   not an answer
5                   my question is
5                   attempt to answer
4                   to the question
4                   to do with
4                   this comment says
4                   the question this
4                   nothing to do
4                   is not a
4                   is an answer
4                   i dont think
4                   comment is not
Source Link
Shog9
  • 159.4k
  • 177
  • 1.2k
  • 1.2k

My answer to your last question was primarily focused on volume - specifically, on how the volume of flags has varied over the past few months, possibly in connection with various events and system changes. But you did ask for a "helpful" percentage, and I provided it... And as you've observed, that percentage has dropped a bit.

So today, let's try & dig into that a bit. My goals here are as follows:

  • Focus specifically on Rude, Unwelcoming or Something Else comment flags raised on Stack Overflow during the month of August
  • Put that percentage in context (what does the drop mean in terms of total helpful flags?)
  • Identify specific contexts where problems are occurring (user class, flag type, comment type)

Let's start with...

Some raw numbers

First, let's re-run the query I used here for the month of August:

Overall comment stats

951,890 comments were created during August, of which 100,965 - just shy of 11% - have been deleted at the time I'm writing this. Those deletions break down as follows:

Reasons                                                               DeletedComments 
--------------------------------------------------------------------- --------------- 
Self-deleted                                                          68338           
Automated deletion                                                    11807           
CommentNoLongerNeeded                                                 9968            
Moderator discretion                                                  8513            
CommentUnwelcoming                                                    954             
Comment Other                                                         335             
Comment Rude Or Offensive                                             292             
Comment Rude Or Offensive, CommentUnwelcoming                         50              
CommentNoLongerNeeded, CommentUnwelcoming                             46              
Comment Rude Or Offensive, CommentNoLongerNeeded                      15              
Comment Other, CommentUnwelcoming                                     7               
Comment Other, CommentNoLongerNeeded                                  4               
Comment Other, Comment Rude Or Offensive                              3               
Comment Rude Or Offensive, CommentNoLongerNeeded, CommentUnwelcoming  2               
Comment Other, Comment Rude Or Offensive, CommentUnwelcoming          1               

Now, this only looks at comments created in August; it's useful as context, but it doesn't tell us anything about flags on older comments or flags which were declined. So let's get some broad stats for those too:

Overall comment flag stats

34,676 comment flags were raised during August, of which 33,245 were marked Helpful. 33,053 of the flags were raised on comments that were later deleted - this is important to note, both because of cases where the flag would've been declined but then the comment later deleted for another reason... And for the cases where comments would later be undeleted, thus leaving a helpful flag on a visible comment. Those flags break down by type as follows:

Type                      FlagsCreated Helpful On comments which were deleted 
------------------------- ------------ ------- ------------------------------ 
Comment Rude Or Offensive 918          666     718                            
CommentUnwelcoming        2572         1922    2004                           
CommentNoLongerNeeded     30221        29999   29702                          
Comment Other             965          660     629                            

Couple of things worth digging into here:

  1. the Rude / Unwelcoming flags are the only categories where the comments are more likely to be deleted than the flags are to be marked helpful.
  2. "other" (currently named "Something Else" on the sites) is far and away the least likely flag to be marked Helpful.

We'll get back to those later. Next, let's separate out those numbers by context.

Just in time comment flag stats

The first subsection above looked at comments created in August; the second looked at flags created in August. What about flags raised in August on comments in August?

Type                      FlagsCreated Helpful On comments which were deleted 
------------------------- ------------ ------- ------------------------------ 
Comment Rude Or Offensive 766          580     603                            
CommentUnwelcoming        2003         1513    1502                           
CommentNoLongerNeeded     10723        10559   10282                          
Comment Other             646          428     397                            

The big difference here from the last table is a massive drop in the number of No Longer Needed flags - which stands to reason, as an awful lot of comments get less useful as time rolls on, so there's somewhat less need for them immediately. Of course... There are also folks who make a hobby of finding older comments to flag, so that feeds in here too.

A more subtle difference is that "Other" flags seem to be slightly more useful on contemporary comments. Not by a lot, but... Another reminder to dig into this later.

One more thing I'd like to get some broad numbers on before digging in though:

Comment flags by type of handler

There are five ways by which comment flags are normally "handled" - and only one of them can decline flags:

  1. Moderators can delete comments (helpful) or dismiss (decline) comment flags
  2. Moderators, privileged users, and post authors themselves can delete the post on which flagged comments reside, thus immediately marking the flags as helpful (without, strictly-speaking, deleting the comments themselves).
  3. If the author of a comment deletes it, any flags on that comment are marked helpful.
  4. The system will automatically delete a comment if it gets a certain number of flags, marking all those flags helpful in the process.
  5. The system will automatically delete a comment the first time it is flagged if the comment contains certain words or patterns - this instantly marks that flag as helpful.

(there are other ways for flags to be dismissed, but they're far less common)

So let's see how those break down for each type of flag for flags raised in August...

Type                      Handled by                               FlagsCreated Helpful 
------------------------- ---------------------------------------- ------------ ------- 
Comment Other             comment automatically deleted            1            1       
Comment Other             post deleted by vote                     2            2       
Comment Other             comment deleted by author                26           26      
Comment Other             post deleted by author                   26           26      
Comment Other             flag handled by moderator                872          567     
Comment Other             flagged comment deleted based on keyword 31           31      
Comment Other             post deleted by review                   7            7       
Comment Rude Or Offensive post deleted by review                   7            7       
Comment Rude Or Offensive post deleted by 3+ flaggers              81           81      
Comment Rude Or Offensive comment deleted by author                37           37      
Comment Rude Or Offensive post deleted by author                   55           55      
Comment Rude Or Offensive flagged comment deleted based on keyword 71           71      
Comment Rude Or Offensive flag handled by moderator                649          397     
Comment Rude Or Offensive post deleted by vote                     18           18      
CommentNoLongerNeeded     post deleted by 3+ flaggers              48           48      
CommentNoLongerNeeded     comment automatically deleted            1            1       
CommentNoLongerNeeded     flag handled by moderator                17882        17661   
CommentNoLongerNeeded     comment deleted by author                342          342     
CommentNoLongerNeeded     post deleted by review                   25           25      
CommentNoLongerNeeded     post deleted by vote                     52           52      
CommentNoLongerNeeded     post deleted by author                   214          214     
CommentNoLongerNeeded     flagged comment deleted based on keyword 11656        11656   
CommentUnwelcoming        post deleted by review                   10           10      
CommentUnwelcoming        comment deleted by author                70           70      
CommentUnwelcoming        post deleted by author                   165          165     
CommentUnwelcoming        flag handled by moderator                2071         1422    
CommentUnwelcoming        flagged comment deleted based on keyword 111          111     
CommentUnwelcoming        post deleted by vote                     35           35      
CommentUnwelcoming        post deleted by 3+ flaggers              109          109     

Observation here? For everything except No Longer Needed, the number of inaccurate flags being handled by moderators is worse than suggested by the previous tables. "Other" starts to actually look pretty reasonable...

Last thing we need here...

Comment flags from low-rep flaggers

One of the major changes (at least in terms of potential) over the past few months was in allowing low-rep users to flag comments on their own posts. Let's look at the stats for only those flags during August:

Type                      FlagsCreated Helpful On comments which were deleted 
------------------------- ------------ ------- ------------------------------ 
Comment Rude Or Offensive 132          48      85                             
CommentUnwelcoming        262          147     150                            
Comment Other             121          50      42                             

Well, that's pretty terrible compared to... Even the previous table. OTOH... There aren't very many of these flags; that's about 12% of all rude/unwelcoming/other flags raised during August. It's more than I originally expected, but not insanely high. However, these flags constitute 22% of all declined rude/unwelcoming/other comment flags - that's a pretty sizable chunk!

Anyway. Enough confusing raw data.

Let's dig in...

First, let's have a look at how the number of Rude, Unwelcoming or Other comment flags as handled by moderators has varied over the past few months...

charting the number of helpful, mod-handled comment flags since March on Stack Overflow

This has the same time period and annotations as the charts I posted the other day... But this is just flags on Stack Overflow, and just flags handled by moderators as helpful. Oh, and it's not a percentage - it's the actual count of flags per week.

...You'll notice that it still goes up. We're getting more useful flags. The cost is some additional number of not-so-useful flags, but we're at least getting something in exchange.

With that in mind, let's try & figure out if there's a pattern here for the sorts of comments that are Not Helpful. We've already touched on low-rep flaggers above, so I'm gonna look at two specific scenarios hinted at by your examples...

Flags on system-generated comments

There are three types of system-generated comments to consider here...

  1. Comments generated by Low Quality review when a reviewer selects a canned comment from the Delete or Recommend Deletion dialog. Those collected 84 flags in August, broken down as follows:

     Type                      FlagsCreated Helpful On comments which were deleted 
     ------------------------- ------------ ------- ------------------------------ 
     Comment Rude Or Offensive 5            2       2                              
     CommentUnwelcoming        9            6       1                              
     CommentNoLongerNeeded     56           56      48                             
     Comment Other             14           10      6                              
    
  2. Comments generated when a close vote or flag is raised, to provide a link to an original question while the question is not yet closed. There were 160 of these flags:

     Type                      FlagsCreated Helpful On comments which were deleted 
     ------------------------- ------------ ------- ------------------------------ 
     Comment Rude Or Offensive 3            0       1                              
     CommentUnwelcoming        22           4       5                              
     CommentNoLongerNeeded     84           45      47                             
     Comment Other             51           17      18                             
    
  3. Comments generated when an "off-topic" vote is cast with a custom explanation provided by the voter. There were 57 of these:

     Type                      FlagsCreated Helpful On comments which were deleted 
     ------------------------- ------------ ------- ------------------------------ 
     Comment Rude Or Offensive 8            4       4                              
     CommentUnwelcoming        14           5       7                              
     CommentNoLongerNeeded     27           24      22                             
     Comment Other             8            4       6                              
    

All together, these account for about 8% of declined comment flags, and about 7% of declined rude/unwelcoming/other flags.

Other flags

The "other" / "something else" option has the distinction of having the second-highest number of declined comment flags by volume and the second-lowest percentage of flags marked helpful.

This is still a relatively small number of flags overall, with about 300 declined for August... But let's see how those declined flags break down by keyword frequency:

194 the
150 is
138 to
115 not
102 this
89  comment
82  a
79  question
75  i
70  and
66  answer
60  it
51  in
44  of
37  my
36  for
34  an
31  duplicate
31  comments
30  be

Two-word phrases:

32                   this comment
29                   the question
29                   is not
19                   comment is
18                   this is
18                   it is
16                   to the
16                   an answer
15                   my question
14                   does not
13                   a duplicate
12                   question is
11                   not a
11                   i don't
11                   a comment
10                   the answer
10                   of the
10                   in the
10                   but it
9                    to be

Three-word phrases:

11                  this comment is
7                   it is not
7                   answer the question
6                   to answer the
6                   this is an
6                   not a duplicate
6                   an attempt to
5                   this is not
5                   not an answer
5                   my question is
5                   attempt to answer
4                   to the question
4                   to do with
4                   this comment says
4                   nothing to do
4                   is not a
4                   is an answer
4                   i don't think
4                   comment is not
4                   comment

I suppose we shouldn't be surprised, after seeing the results for system-generated comments above, that an awful lot of "other" flags are raised to dispute duplicates.

It's maybe a bit more interesting that a fair number of "other" flags are also raised on comments that try to answer the question... We kinda expected this might need to be addressed at some point, and it looks like that is the case.

Action items

First, I think it's important to remember that these are still relatively small changes. But, that doesn't make them any less irritating: these flags tend to be treated as high-priority, so even a small jump in noise means less time devoted to situations where a moderator's intervention is actually needed urgently - sorta the flag equivalent of crank calls to 911.

With that in mind, I'd recommend the following:

Blacklist duplicate patterns in Other flags

This should be easy: match on some trivial expression (say, \bnot\s+(a\s+)?duplicate\b) and reject the flag with some nice message about editing:

If this question has been misidentified as a duplicate, please edit it: make the title specific to this problem, and then reference the other question explaining why the answers there do not solve this problem.

...unfortunately, the current blacklist is ignored by comment flags. Oops. I've submitted a patch to correct this.

Better guidance for new / low-rep flaggers

I have a sneaking suspicion that some non-trivial number of new users think "flag" means "reply". When we ran into this before, I changed the description of "other" to "in need of moderator intervention" to kinda emphasize the fact that no one in the conversation sees these flags.

But now "other" is "something else", and nothing else in the flag dialog hints at what flagging actually does... So I guess I could kinda see some folks thinking this is sort of a "reaction" or "postscript" option instead of a means of reporting abuse... So, maybe add a note indicating that no one in the conversation sees these? That, uh, flags are reviewed by third parties who will delete comments as needed? Open to suggestions.

Answers as comments... what do we do with them?

It's probably time to discuss some sort of strategy for handling flags on comments-as-answers. Flagging doesn't accomplish much; then again, neither does declining those flags. We've been telling people for years to create an answer that elaborates on the comments when they see these things, but there's nothing in the UI itself that encourages this - and what sometimes happens instead is that new folks see these comments and think they have to earn the comment privilege in order to answer!

I recommend kicking off another discussion specifically about this problem, with an eye toward identifying UI changes (either for flagging or commenting) that might assist in some way.