53

In Summary

Which types of Stack Overflow content generate the largest counts of helpful flags? Do we have any (anonymized) statistics? Otherwise, what is the anecdotal evidence?


In Detail

The current election's main page mentions the volume of flags generated on SO:

Due to the size of Stack Overflow (averaging around 2,700 flags per day), moderation can be a significant responsibility...

It's only natural, therefore, that discussions about candidates mention flagging activities - for example:

89k helpful flags; 99.61% helpful

27000 flags

...and so on. To me, these are mind-boggling statistics.

Several discussions refer to users' bots, which may be performing the bulk of that flagging work - I don't know.

It seems that some users are able to raise tens/dozens/scores of useful flags per day - day after day, month after month. My own experience has been very different. I see some "not an answer" content and a small amount of rude & abusive content - but nothing in any great volumes.

Maybe that's a testament to the speed & effectiveness of the community, and the incredible dedication of our moderators. This is more impressive to me when I consider how flags can age away - which happened to most of my (admittedly very small volume of) old "flag to close" flags.

But even accounting for bots, what are all these flags? What is the type of content which is being usefully flagged so much? Are there any statistics on useful flags raised by flag type (but anonymized, of course)?

I looked at the schema on SEDE. I saw a FlagTypes table containing: Question Recommend Close, Question Close, and Question Reopen as its 3 types. I understand why flagging data may be restricted due to its sensitive nature. So maybe the data I am looking for is not publicly accessible.

Reasons for asking:

I feel I must be looking directly at a lot of flaggable content, and just not seeing it. What am I missing?

Having a better understanding of this will probably also help me to avoid generating flaggable content myself...


For reference, assuming a user has the cast close and reopen votes privilege, here is a summary of the main flag types:

Question flags:

  • spam
  • rude or abusive
  • needs improvement
  • a duplicate
  • very low quality
  • in need of moderator intervention (with specific reason)

Answer flags:

  • spam
  • rude or abusive
  • not an answer
  • in need of moderator intervention (with specific reason)

Comment flags:

  • contains harassment, bigotry, or abuse
  • unfriendly or unkind
  • no longer needed (outdated, conversational or not relevant)
  • something else (with specific reason)

Resources I found:

Overview of Flagging

What does the helpful flag mean?

What are the “spam” and “rude or abusive” (offensive) flags, and how do they work?

When should comments be deleted?

What are valid reasons for flagging comments?

What can I use 100 flags for each day?

I also found some posts discussing users with high flag counts, and strategies for increasing one's flag counts - but those are not really the purpose of this question:

Who raised the most helpful flags on Stack Overflow and Meta?

How did this user raise 5,000 helpful flags in only 52 days?

I don't think there is any way I could ever be able to generate flags (or reviews, or votes, for that matter) in these quantities and at these speeds, and still be accurate enough to be doing good work.

9
  • 2
    Hang out here and flag the spam that appears. Commented Oct 24, 2021 at 22:32
  • I think most of those flags are spam and "Not an Answer" flags for posts. Comments flags could be "No longer needed" flags on for example old "...From review" comments.
    – Scratte
    Commented Oct 24, 2021 at 22:41
  • 9
    Joining either the Charcoal HQ or SOBotics chat rooms give more than enough links to potential flaggable posts to hit your quota each day. Those with high flag counts hang out in those rooms. Some run bots, some just react to the bots. Commented Oct 24, 2021 at 23:47
  • 5
    @DanielWiddis usually Charcoal alone wouldn't be enough to hit your daily quota. There's a bunch of spam (and NAAs), but not max-out-your-quota levels of it, at least once you have 100 flags/day.
    – Ryan M Mod
    Commented Oct 25, 2021 at 2:10
  • 10
    See: 2020: a year in moderation for some perspective
    – Cody Gray Mod
    Commented Oct 25, 2021 at 2:21
  • 6
    When you see anything resembling one or more the following, then flag it as No Longer Needed: "thanks", "+1", "I never thought of that", "you are a god", "I owe you a beer", "I just updated my question/answer", "Does my answer work for you?", "Please upvote, accept, like, follow, copy, hug my answer", "This answer is the best / my favorite / better than the others ...but I'm not going to add any thing credible to substantiate my claim", "How to change this answer if my requirements are slightly different?", ... I can go on and on about comments that don't improve the researcher experience. Commented Oct 25, 2021 at 7:20
  • 7
    "To me, these are mind-boggling statistics." Yes, most of the users with several tens of thousands of flags use "auto flagging" which allows a script to flag content automatically (e.g. they are not personally involved in the flag). This is a big part of how the Charcoal team (and others) functions effectively across the Stack Exchange network. See Can a machine be taught to flag spam automatically? for more info on that project.
    – TylerH
    Commented Oct 25, 2021 at 14:12
  • 2
    @TylerH "...they are not personally involved in the flag..." That hints at a potential problem. If somebody isn't personally involved in the flagging, than using a personal account for that seems a bit the wrong choice. Maybe automatic/bot flagging should be its own category and these flags should be marked as automatically flagged. I would put more weight on manually selected flags. This isn't to say these flags aren't really helpful. Commented Oct 26, 2021 at 11:00
  • 2
    @Trilarion Well, the threshold for when a flag is cast without their intervention is something like 99%, so they do put a lot of effort into making sure incorrect flags are avoided... I am not that familiar with the specific criteria (I don't participate in auto flagging, myself), but you can read more about it on charcoal-se.org/flagging
    – TylerH
    Commented Oct 26, 2021 at 14:04

2 Answers 2

25

It's actually not that difficult to cast 200 flags each day. The problem is that most users usually don't spend that much time on Stack Overflow.

Your flag quota increases with the number of helpful flags up to the maximum of 100 flags on posts and 100 on comments.

Comment flags are the easiest to raise as there's plenty of comments that need to be removed. Of course, we do not want to flag every single comment. Whenever you see conversational comments (e.g. "thanks", "please wait, I will edit the post", "it's working now"), they need to be flagged as no longer needed. Either the system will automatically remove it or it will go to moderators who remove hundreds if not thousands of such comments each day.

Comments are easy to flag because you don't need to look hard for them. They are pretty much everywhere, especially on new questions with answers. You could also use a SEDE query to look for some easy to flag comments, but that is often considered a waste of time by many. If you can do it via a SQL query then you can write a piece of software to do it automatically for you.

Post flags are the second type of flag. These included spam, R/A, NAA, close and custom flags.

  • Spam and R/A are not easy to find because Charcoal and moderators take care of it very quickly. You can watch Smoke Detector in Charcoal and you might be able to cast a few of them.

  • Close flags can only be raised by users with less than 3k reputation

  • NAA (not an answer) and VLQ flags are for posts that aren't an answer in any kind of shape or form. For example "I am having the same problem", "thanks for all answers", "here are more details for my question", etc. You might be surprised to learn that despite all the warnings, people post hundreds of such not-an-answers each day. All of them need to be deleted, but these are a little more difficult. While there are keywords and common phrases, each post is different. You have to read an answer, even if just to look for keywords, to see if it is an answer or a comment/question that needs to be deleted.
    You can find some of them when browsing casually, but to find them in larger quantities you have to watch new answers as they are posted (Charcoal has a nice Blaze page that helps with this), search for keywords, use 10k tools NATO page, or write some automated software to sieve and filter them out.

  • Custom flags can be about any issue that moderators need to be aware of or need to handle (but the community can't). These are the most difficult to rack up as there needs to be an issue in the first place that you can report.

Stack Overflow doesn't offer a detailed flag statistic overview so it's not easy to calculate which flags you have raised of the most. I can offer some insight into what I have been flagging and my rough estimates.

I have raised ~42k post flags. Most of them are on NAAs. I have spent some time in SOBotics and other tools available on site. I also wrote my own bot to find them and auto flag the most obvious ones. This saves me a lot of time, but there are still plenty of those that need to be looked at by a human. Over 1.5k on spam or R/A content. I have also flagged hundreds of issues I saw such as voting fraud.

I raised 49k+ comment flags. Most of them using automated software. I remove "thanks", "edited", "worked" on older posts. I estimate that I raised close to 10k of them manually.

Some trivia:

3
  • 3
    A tangential discussion about comments and their deletion has been moved to chat. The irony is strong with this one.
    – Cody Gray Mod
    Commented Oct 27, 2021 at 22:09
  • Shouldn't "...less than 3k reputation" be "...more than 3k reputation"?
    – skomisa
    Commented Aug 9, 2022 at 17:37
  • @skomisa Users with more than 3k reputation cannot raise close flags anymore because they gain the privilege to vote to close.
    – Dharman Mod
    Commented Aug 9, 2022 at 20:54
-6

My flags are almost 50/50 "Not an answer" and "No longer needed".

I don't use any automated tools or participate in any SIGs so my experience should be typical for a regular SO user.

9
  • 21
    How is this helpful? It's like someone asking "what is the most popular language spoken in the world?" and someone replying with "I speak Portuguese". That doesn't tell us anything about the most popular language in the world.
    – DavidG
    Commented Oct 25, 2021 at 14:16
  • 5
    @DavidG the OP wrote in his post "Otherwise, what is the anecdotal evidence?"
    – Marijn
    Commented Oct 25, 2021 at 17:51
  • 6
    @Marijn You expect every person to post an answer with their anecdotal evidence? That would be chaos.
    – DavidG
    Commented Oct 25, 2021 at 18:12
  • 8
    @DavidG that can be expected, yes. Most likely the anecdotal evidence of multiple people will be similar, so they can just upvote a similar existing answer. Then a few answers will appear with different trends in flagging behavior that all lead (as the OP asked about) to high numbers of flags.
    – Marijn
    Commented Oct 25, 2021 at 18:41
  • 3
    @Marijn I've explained in an edit how my experience should be typical. Commented Oct 25, 2021 at 19:20
  • 8
    @Marijn Just because someone asks a silly question, doesn't mean it needs to be answered. Anecdotes are not data, not ever, and it was a poor thing to request in the first place. Also, this post could have just been a comment.
    – DavidG
    Commented Oct 26, 2021 at 9:20
  • 3
    @DavidG if you think the question is silly then you can downvote the question. If you think only a small part of the questyion is silly then you can comment on the question asking to OP to remove the silly part. It is not necessarily the best option to downvote the answer if it actually answers the question as asked (see meta.stackoverflow.com/questions/255459/…, although that is about main questions and not about meta questions), although you can of course vote as you please.
    – Marijn
    Commented Oct 26, 2021 at 20:14
  • 5
    More on point: anecdotes are not data but they can provide useful information - in this case the OP wants to know which types of flags can lead to a high number of useful flags, this answer states that 50/50 NAA and NLN works for at least one person, so that is a good starting point for the OP to test if it works for them as well.
    – Marijn
    Commented Oct 26, 2021 at 20:17
  • 1
    For what it's worth, @ivan_pozdeev, I found your answer and subsequent update helpful, thank you. But I think I should probably not have used the phrase "anecdotal evidence" - that was my mistake. I still have a lot to learn about SO and Meta SO. Commented Oct 28, 2021 at 23:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .