111

As we say goodbye to the old year and welcome the new one, we have a tradition of sharing moderation stats for the past 12 months.

As most of you here are aware, sites on the Stack Exchange network are moderated somewhat differently to other sites on the web:

We designed the Stack Exchange network engine to be mostly self-regulating, in that we amortize the overall moderation cost of the system across thousands of teeny-tiny slices of effort contributed by regular, everyday users.
-- A Theory of Moderation

That doesn't eliminate the need for having moderators altogether, but it does mean that the bulk of moderation work is carried out by regular folks. Every bit of time and effort y'all contribute to the site gives you access to more privileges you can use to help in this effort, all of which produce a cumulative effect that makes a big difference.

So as we welcome 2021, and in keeping with tradition, let us look back at what we accomplished as a community... by looking at some exciting stats. Below is a breakdown of moderation actions performed on Stack Overflow over the past 12 months:

Action Moderators Community¹
Users suspended² 1,196 1,512
Users destroyed³ 4,653 0
Users deleted 5,434 0
Users contacted 4,865 0
User suspensions lifted early 34 0
User review-bans lifted early 179 0
User banned from review 22,340 2,258
Tasks reviewed⁴: Triage queue 610 741,283
Tasks reviewed⁴: Suggested Edit queue 1,726 817,980
Tasks reviewed⁴: Reopen Vote queue 3,256 220,473
Tasks reviewed⁴: Low Quality Posts queue 53 285,613
Tasks reviewed⁴: Late Answer queue 5 284,066
Tasks reviewed⁴: Helper queue 8 11,072
Tasks reviewed⁴: First Post queue 1,084 849,618
Tasks reviewed⁴: Close Votes queue 56,427 225,745
Tags merged 285 0
Tag synonyms proposed 178 209
Tag synonyms created 273 86
Tag highlight language set 59 0
Revisions redacted 438 0
Questions unprotected 12 89
Questions reopened 2,544 12,197
Questions protected 57 4,288
Questions migrated 435 1,267
Questions merged 109 0
Questions flagged⁵ 2,146 415,492
Questions closed 96,631 424,627
Question flags handled⁵ 73,556 343,268
Posts unlocked 155 1,789
Posts undeleted 4,966 75,194
Posts locked 1,707 10,691
Posts deleted⁶ 155,196 1,449,456
Posts bumped 0 33,260
Escalations to the Community Manager team 748 0
Comments undeleted 1,083 0
Comments flagged 373 446,811
Comments deleted⁷ 756,159 1,009,253
Comment flags handled 269,372 177,751
Bounties canceled 157 2
Answers flagged 4,313 345,227
Answer flags handled 268,537 80,891
All comments on a post moved to chat 1,884 0

Footnotes

¹ "Community" here refers both to the membership of Stack Overflow without diamonds next to their names, and to the automated systems otherwise known as user #-1.

² The system will suspend users under three circumstances: when a user is recreated after being previously suspended, when a user is recreated after being destroyed for spam or abuse, and when a network-wide suspension is in effect on an account.

³ A "destroyed" user is deleted along with all that they had posted: questions, answers, comments. Generally used as an expedient way of getting rid of spam.

⁴ This counts every review that was submitted (not skipped) - so the 2 suggested edits reviews needed to approve an edit would count as 2, the goal being to indicate the frequency of moderation actions. This also applies to flags, etc.

⁵ Includes close flags (but not close or reopen votes).

⁶ This ignores numerous deletions that happen automatically in response to some other action.

⁷ This includes comments deleted by their own authors (which also account for some number of handled comment flags).

Further reading:

A big thank you to Shog9 for writing the queries and script to facilitate fetching and posting this data to all the sites in the network, and to Brian for the subsequent work making the whole thing more user friendly.

Wishing everyone a happy 2021!

38
  • 4
    Are things like the 2 bounties cancelled performed by moderators that stepped down? AFAIK there's no way a normal community member can cancel a bounty
    – Erik A
    Commented Jan 19, 2021 at 17:07
  • 77
    Over half a million questions closed compared to 340k in 2019, a 50% increase year-on-year. If that isn't the clearest indication that there's a quality problem, I don't know what is.
    – Ian Kemp
    Commented Jan 19, 2021 at 17:16
  • 8
    @IanKemp well obviously Stack Overflow is becoming more popular... plus one could consider that a good thing, no? More questions being closed by both mods and users. Plus, ever since a moderator started reviewing the CV queue, more questions have been closed that would have normally entered the void. More questions are just being identified and acknowledged.
    – 10 Rep
    Commented Jan 19, 2021 at 17:17
  • 137
    @IanKemp It's the clearest indication that the close vote threshold was lowered from 5 to 3, and that this change is having the desired effect.
    – Cody Gray Mod
    Commented Jan 19, 2021 at 17:19
  • 4
    Moderators don't have the ability to bump a post, @10Rep. As Jon says, that is only done by the Community user, which does have moderator privileges, but in this sense is acting more like a bot. The "Posts bumped" line item in the table does not refer to posts incidentally bumped due to edits. It's only talking about the automated bumping that Community does.
    – Cody Gray Mod
    Commented Jan 19, 2021 at 17:22
  • 4
    I would be interested in up/downvote statistics as well, either on question and on answers. I remember there was a problem that people were not motivated to downvote on answers, so I would like to see if that changed at least a little. Even the -1 rep means nothing (and usually returns on deletion), it seems it is a big deal for most users.
    – Ruli
    Commented Jan 19, 2021 at 17:38
  • 3
    @10Rep Mostly on review. Comment flags, when multiple users flag the same comment as NLN, for example, it gets deleted automatically. I think that several spam flags will auto-delete a post.
    – yivi
    Commented Jan 19, 2021 at 17:53
  • 2
    @yivi Spam/abusive flags cause the post to get locked by Community when the threshold is reached.
    – Cody Gray Mod
    Commented Jan 19, 2021 at 18:29
  • 10
    @CodyGray If a hospital put out a press release saying "the number of gunshot wounds treated has increased by 50% thanks to us decreasing wait times from 5 to 3 minutes", the public response would not be "hey that's great", it would be "holy hell, why are there so many gunshot victims?"
    – Ian Kemp
    Commented Jan 19, 2021 at 22:28
  • 6
    Nah, the response is mostly just that's business as usual. Any time you put a textbox out there on the Internet, people are going to type nonsense into it. The trick isn't in finding a way to avoid the nonsense getting posted, it's in finding a way to deal with it. I think we have found one. We should continue to seek improvements to how we deal with it. It isn't the job of the hospital to reform the society to reduce the number of gunshot victims.
    – Cody Gray Mod
    Commented Jan 19, 2021 at 22:30
  • 11
    @IanKemp The problem is that the statistics tell us how many questions were closed (handled by community). We don't know how many needed to be closed but never got closed. There is a quality problem but the number of closed questions doesn't show that. It only shows us that we handled more than before. If anything we improved the quality a little by closing more questions, but there is still plenty that need to be closed.
    – Dharman Mod
    Commented Jan 19, 2021 at 22:31
  • 15
    @CodyGray "...the close vote threshold was lowered from 5 to 3, and that this change is having the desired effect." Yes, it looks like we were limited by close votes, not by close worthy questions. Commented Jan 20, 2021 at 11:34
  • 3
    Is the What post get deleted and why? going to be also updated?
    – Braiam
    Commented Jan 22, 2021 at 15:14
  • 3
    Give me unilateral and unlimited delete privilege and I'll fix your quality problem. Commented Jan 23, 2021 at 3:10
  • 9
    @SotiriosDelimanolis We call this privilege "diamond moderator". Keep your eye out for the next election. :-) Note that, even when you have unlimited access to the firehose, it's difficult to drink all of the water.
    – Cody Gray Mod
    Commented Jan 23, 2021 at 5:58

2 Answers 2

38

Wanna see how these numbers have changed over time?

Yes. And now my first question is: why is there such a huge difference in Comments deleted by Moderators?

Action Mod2020 Mod2019 Change Comm2020 Comm2019 Change
Users suspended² 1,196 1,340 -11% 1,514 1,615 -6%
Users destroyed³ 4,653 4,305 8% 0 0
Users deleted 5,434 4,155 31% 0 0
Users contacted 4,866 4,475 9% 0 0
User suspensions lifted early 34 45 -24% 0 0
User review-bans lifted early 179 64 180% 0 0
User banned from review 22,340 3,023 639% 2,258 4,004 -44%
Tasks reviewed⁴: Triage queue 610 34 1,694% 741,283 1,239,611 -40%
Tasks reviewed⁴: Suggested Edit queue 1,726 2,940 -41% 817,980 1,141,695 -28%
Tasks reviewed⁴: Reopen Vote queue 3,256 223 1,360% 220,473 185,719 19%
Tasks reviewed⁴: Low Quality Posts queue 53 219 -76% 285,613 343,007 -17%
Tasks reviewed⁴: Late Answer queue 5 25 -80% 284,066 282,584 1%
Tasks reviewed⁴: Helper queue 8 9 -11% 11,072 31,119 -64%
Tasks reviewed⁴: First Post queue 1,084 89 1,118% 849,618 804,290 6%
Tasks reviewed⁴: Close Votes queue 56,427 3,126 1,705% 225,745 318,431 -29%
Tags merged 285 534 -47% 0 0
Tag synonyms proposed 178 423 -58% 209 107 95%
Tag synonyms created 273 534 -49% 86 77 12%
Tag highlight language set 59 28 111% 0 0
Revisions redacted 438 490 -11% 0 0
Questions unprotected 12 8 50% 89 75 19%
Questions reopened 2,544 1,235 106% 12,197 9,112 34%
Questions protected 57 141 -60% 4,288 6,039 -29%
Questions migrated 435 418 4% 1,267 758 67%
Questions merged 109 94 16% 0 0
Questions flagged⁵ 2,146 1,758 22% 415,492 472,085 -12%
Questions closed 96,631 33,862 185% 424,627 306,210 39%
Question flags handled⁵ 73,556 45,195 63% 343,268 430,080 -20%
Posts unlocked 155 147 5% 1,789 574 212%
Posts undeleted 4,966 3,325 49% 75,194 70,345 7%
Posts locked 1,707 432 295% 10,691 8,517 26%
Posts deleted⁶ 155,196 142,938 9% 1,449,456 1,251,256 16%
Posts bumped 0 0 33,260 34,352 -3%
Escalations to the Community Manager team 748 1387 -46% 0 0
Comments undeleted 1,083 1,770 -39% 0 0
Comments flagged 373 611 -39% 446,811 471,440 -5%
Comments deleted⁷ 756,159 6,075,670 -88% 1,009,253 981,980 3%
Comment flags handled 269,372 244,261 10% 177,751 227,661 -22%
Bounties canceled 157 227 -31% 2 0
Answers flagged 4,313 3,707 16% 345,227 366,618 -6%
Answer flags handled 268,537 265,031 1% 80,891 105,362 -23%
All comments on a post moved to chat 1,884 1,568 20% 0 0
6
  • 22
    Because a script ran last year which deleted some millions of comments. Commented Jan 19, 2021 at 19:46
  • 13
    The script still runs, just at a slower pace now ... Also dang, that tags stuff really took a hit ...
    – Bhargav Rao Mod
    Commented Jan 19, 2021 at 19:50
  • 7
    If we deleted over 6 million comments every year, we'd eventually run out of comments to delete. It was necessary to pace ourselves. :-)
    – Cody Gray Mod
    Commented Jan 19, 2021 at 20:19
  • 17
    This is a really useful presentation, thanks for putting it together - it's certainly telling that a number of rather unsung moderator activities (tag cleanup, redactions, bounty oversight) have taken a palpable hit.
    – Shog9
    Commented Jan 19, 2021 at 22:20
  • 5
    @Shog9 I feel the “bounties cancelled” and “posts redacted” metric changes are within the margin of error. “A few hundred a year” has plenty of room for fluctuations :-) Tag handling is a much more manual job and definitely took a hit when the most active moderator in that area suspended activity for a stretch.
    – Martijn Pieters Mod
    Commented Jan 20, 2021 at 9:32
  • 6
    Fair point, @Martijn - redactions jumped out at me simply because of the surrounding context: question volume increased this year esp with wfh, and I would have expected corresponding sensitive info mistakes to go up, but that's hardly guaranteed. Would need to dig into overall bounty activity to normalize that one, but pretty massive drop over the past couple of years.
    – Shog9
    Commented Jan 20, 2021 at 16:22
23

The (main) purpose of the LQP queue is to reduce our moderators' workload so that they have more time to review custom flags rather than deal with "me2" or "thanks you" answers all the time. When the community handles less than 1/4 of the answer flags, then you know something's wrong.

Graph of LQP reviews as a percentage since 2016

Therefore, I believe it's time to:

Ideally, if users understand which answers are NAA, then:

  • DO NOT clear any NAA/VLQ flags when a post is edited or teach the reviewers what this button actually does.
  • Decline flags raised on answers where the LQP consensus is "Looks OK" (similar to Leave Open in CVQ).

After that, the number of flags handled by the community should hopefully increase.

10
  • if most of mod handling goes to declining incorrect flags this may be good thing indeed. Mod declines carry warnings and flag suspensions - educational for incompetent / disruptive flaggers. Compared to that, "disputing" flags by regular users from review is toothless and useless
    – gnat
    Commented Jan 19, 2021 at 21:39
  • Maybe improve the queue as well, make it more obvious which answer should and shouldn't be deleted as many 2kers delete answers which aren't stricly NAA.
    – 10 Rep
    Commented Jan 19, 2021 at 21:40
  • 1
    @10Repsaysgetvaccinated incorrect deletions from the LQP should be indeed rare because per my recollection when they changed vote limit testing has shown that amount of 4 votes gets close enough to how posts are handled by diamond moderators - meaning same answers would be deleted by diamond mods
    – gnat
    Commented Jan 19, 2021 at 21:51
  • @gnat I still see plenty of "Recommend deletions" on posts that don't desevrve it
    – 10 Rep
    Commented Jan 19, 2021 at 22:03
  • 1
    FWIW, vlq flags are probably a drop in the bucket compared to NAA; that's why they both go into the same queue. Love your 2nd suggestion though
    – Shog9
    Commented Jan 19, 2021 at 22:15
  • 11
    The major purpose of VLQ flags is for moderators to get to lawyer over the definition of the word "very". Yes, please; let's finally get rid of it.
    – Cody Gray Mod
    Commented Jan 19, 2021 at 22:31
  • 2
    If we are going to remove VLQ then I would like to be able to flag the same post twice with NAA. It's the biggest advantage of having two flags.
    – Dharman Mod
    Commented Jan 19, 2021 at 22:32
  • 1
    I would say that's a scenario where you should just be raising a custom moderator flag, @Dharman. You're effectively asking a moderator to override the community's initial decision. That should just be expressed directly, not as an end-around with a second type of flag. (As a bonus, the moderator can review the actions taken by the reviewers in the first round and determine whether review suspensions are in order.)
    – Cody Gray Mod
    Commented Jan 19, 2021 at 22:36
  • 2
    @10Repsaysgetvaccinated data collected during testing of voting in LQP queue suggests that your observations do not indicate a problem worth worrying about
    – gnat
    Commented Jan 20, 2021 at 6:18
  • 12
    Hey look it's another answer with excellent suggestions that have been made for years but will never be implemented because SE Inc. hates us!
    – Ian Kemp
    Commented Jan 20, 2021 at 10:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .