212

Stack Overflow, due to its size, has some unique problems. One of those is flags. For example, in the last 30 days as of the time I am writing this, there have been 26,710 flags. That is 890 flags per day on average.

One of the primary duties of a community moderator is to look at and process flags, as noted in the theory of moderation blog post:

Even with active community self-regulation, moderators occasionally need to intervene. Moderators are human exception handlers, there to deal with those (hopefully rare) exceptional conditions that should not normally happen, but when they do, they can bring your entire community to a screaming halt — if you don’t have human exception handling in place.

The most common moderator task is to follow up on flagged posts. Every post contains a small flag link, which anyone with 15 reputation can use.

Flags are good. Flags help us maintain Stack Overflow and improve it. But there are so many of them. If you do the math, and assume each flag takes about one minute to look at, load the page, and decide (probably an optimistic number), that is 890 minutes of work per day -- almost fifteen hours in total.

In addition to using every clever trick in the book we can think of to expedite and automate flag handling, we've continually been expanding the number of community moderators on Stack Overflow to help share the work of handling this tide of flags:

Note: This list only reflects the results of the moderator election at that time, they are not indicative of who is currently a moderator of the site

As of now we have sixteen community moderators, visible on the /users page of Stack Overflow.

Of course, serving as an elected community moderator is, and always will be, a completely voluntary at-will activity. But I believe the sheer scale of Stack Overflow now demands some additional policies specific to Stack Overflow community moderators.

We are not asking anyone to make special accommodations, but it is increasingly untenable for us to have elected moderators who spend substantial amounts of time on Stack Overflow, but do not participate in the primary duty of a moderator -- that is, helping us follow up on and resolve flags from Stack Overflow users. It's unfair to the other elected moderators, and it is unfair to the community.

Just as in open source, "many hands make light work", but this requires each of the moderators to be truly pitching in and contributing by resolving flags by clicking that bright orange number periodically throughout the day.

We are of course open to adding more community moderators. But that is avoiding the issue.

I believe we regrettably must have a new, specific policy for community moderators on Stack Overflow, due to its size and scale: I propose that on Stack Overflow, all elected community moderators must close a 'reasonable' number of flags while they are on the site. If they do not, they cannot continue to hold the position of Stack Overflow community moderator.

"Reasonable" is of course To Be Determined, and the point of this is not to demand specific numbers -- all we're looking for is a fair and equitable balance of flag handling among community moderators over a period of 2-3 months.

Comments? Opinions? Thoughts?

30
  • 8
    Wouldn't it make sense, instead, to increase ever so slightly the cost of flagging on SO, given that scarcity of flags currently isn't an issue? Of course, it depends on how many posts on the site only get 1 flag before they're handled on, and thus how many of those risk never getting moderator attention as a result of the change.
    – badp
    Commented Aug 9, 2011 at 6:40
  • 64
    almost all the flags are good and valid flags that need handling, and each one makes Stack Overflow a little better. The problem is not invalid or superfluous flags. Commented Aug 9, 2011 at 6:42
  • 6
    This sounds reasonable. Do the moderators have anything to say? Are all flags similar in how much time they take to process? Are there any cases of moderators spending a very long period of time just to sort out one flag?
    – Pekka
    Commented Aug 9, 2011 at 7:04
  • 3
    @waffles ... or discharge the soldiers that don't fight and hire new one's that do?
    – DMA57361
    Commented Aug 9, 2011 at 7:35
  • 5
    @DMA which is the exact problem ... 1) do you discharge them? 2) what is a good soldier? Commented Aug 9, 2011 at 7:36
  • 1
    @waffles I was reading the If they do not, they cannot continue to hold the position of Stack Overflow community moderator. in the question above - which implies discharging is being considered. And I can only assume 2 will be determined by the "performance chart" on the history page, but I think that is the real point being asked here.
    – DMA57361
    Commented Aug 9, 2011 at 7:39
  • 6
    @Will 2) is something only you moderators can really comment on. Is the number of processed flags a reliable metric for whether you're a good moderator? Are there cases of moderators processing very few flags, but still doing a lot of mod work? Those would have to speak up
    – Pekka
    Commented Aug 9, 2011 at 7:39
  • 25
    @waffles: Us un-diamonded masses don't know the details of how many moderators are "never participating in fighting" (nor do I think we should), so it's pretty hard to comment on that aspect other than to shrug and give a general "That seems reasonable". Which makes it strange to me that this is a Meta post -- it seems like this would be better handled by emails among the devs and mods.
    – jscs
    Commented Aug 9, 2011 at 7:50
  • 3
    @waffles In your example there are infinite rations -- what finite resource are slacker mods consuming? Commented Aug 9, 2011 at 14:22
  • 7
    What is the cost of none active moderator? (e.g. eat the rations, does not seem to relate to moderators) Commented Aug 9, 2011 at 15:22
  • 6
    @Ian: During elections, there are a certain number of moderator posts available. These are then taken. If one of them is filled by somebody who then does not do any moderating, then something is wrong. Seems pretty obvious to me. Commented Aug 9, 2011 at 18:40
  • 9
    I feel this question deserves more time than I can give it, but my post would revolve around the thought "Moderation, rather than being an act of guiding the community, has largely become an act of cleaning the toilets." I'm suggesting we take a hard look at some of the flags the mods clear, and assign them to 40k users, which would immediately add another 100+ people to the flag handling queue: meta.stackexchange.com/questions/101772/need-new-30k-abilities
    – Adam Davis
    Commented Aug 10, 2011 at 22:26
  • 1
    @Jeff 1) True, assuming all 890 daily flags require that level of leadership. 2) The community might also be able to provide some of that leadership.
    – Adam Davis
    Commented Aug 10, 2011 at 22:47
  • 1
    @Adam we have spent the last three months exhausting every angle on that. Again, I am not at liberty to disclose the moderator activity data but it is not a subtle difference. We are talking 2 orders of magnitude PLUS. Commented Aug 11, 2011 at 1:12
  • 8
    lol, I think all you need to do is ask a question like this any time mod activity starts slacking. The queue hasn't been over a hundred since you posted this.
    – user1228
    Commented Aug 11, 2011 at 11:57

15 Answers 15

123

Automate and offload to the community.

High rep users can already vote to close, migrate or delete bad or off-topic questions. They can not do much for equally bad answers. Perhaps we could trust the system to delete answers identically flagged by a group of high rep and/or high flag weight users?

Similar to the close reasons for questions, we could have standard reasons like for example:

  1. the post just says "Thank you"
  2. the answer is really a follow up question
  3. the post just contains a link to the guys blog
  4. the post contains mostly four letter words
  5. the post is just unintelligible

and leave the more difficult cases to the moderators.

7
  • 3
    Items 1,2,5 in your list can be flagged not an answer; 3 could be edited, spam or not an answer (depends on how blatant the advertising is); and 4 could be edited, offensive or not an answer (depending on if it can be cleaned up in to a real answer). Maybe the point is not an answer flags need automating? But apparently some users do try to use flags to say "I don't agree with this"... so caution is required.
    – DMA57361
    Commented Aug 9, 2011 at 7:30
  • 3
    Yes, that was kind of my idea. If the flags were more precise than "not an answer" perhaps some of them could be automated and relieve the moderators to work on the harder cases. Also high flag weight might be part of this, trusting people who have a history of correct flags.
    – Bo Persson
    Commented Aug 9, 2011 at 7:42
  • 2
    I think, as per the answer that's been deleted and other comments, this answer misses the point that Jeff wishes to discuss - not about reducing the workload of mods, but making sure that all the mods are at least making a suitable effort to clean the site. Otherwise, would it be better to then replace the "less good" mod with another member of the community who will put more effort in?
    – DMA57361
    Commented Aug 9, 2011 at 7:45
  • 8
    Well, if there is less work to do wouldn't the work actually done then be sufficient?
    – Bo Persson
    Commented Aug 9, 2011 at 7:48
  • Probably, but even with a reduced workload is it acceptable to have the majority of moderation work done by a minority of moderators?
    – DMA57361
    Commented Aug 9, 2011 at 7:57
  • @DMA57361, yes, the less flags that have to be handled by the moderators, means more time that they can spend on Meta, or in SO answering questions.
    – jzd
    Commented Aug 9, 2011 at 11:25
  • And what, @jzd, if again only that same minority of moderators are then on meta helping? What about the "other" moderators who aren't helping out anywhere? Less work won't solve that problem - sure, it's not a bad thing to reduce the workload, but it's not the point of this question. Incidentally, please do note that I'm a SU mod, not a SO mod, so my comments here RE minorities doing the work are conjecture based off of the implications in the question.
    – DMA57361
    Commented Aug 9, 2011 at 11:29
60

We can't simply keep adding moderators in order to compensate. This doesn't just side step the issue, it creates more work for the existing moderators as well. New moderators require training and help initially, which means more must be asked from the people who are willing to spend time focusing on moderating.

If you are able to spend time asking or answering three to five questions per day, I don't see why it would be unreasonable to ask that you also spend twenty minutes helping to keep the pressure in the flag queue down to a reasonable level. While none of us committed to a certain level of participation when accepting our agreement, I think that we all understood that:

  • Moderators should want the responsibility
  • The main part of a moderator's job is to attend to flags

I can't conceive of someone wanting the responsibility unless they visit and use the site on a regular basis with plans to continue doing so.

Things do happen in our lives and I don't think anyone should feel bad about not being able to spend much time in their capacity as a moderator due to personal issues. Additionally, some days are simply more exhausting than others. I think we should set some expectations, as long as they remain 'on the average'.

With that being said, I don't think that it's unreasonable to ask that each moderator:

  • Services an average of 60 - 100 flags per week. Note that it's extremely common to clear ten flags within a minute, depending on the nature of the flag. As said above, of course there will be periods where a moderator simply can't give that much time, but on average I don't think it would be difficult.

  • Participate here on meta. Answer a few questions each month asking for support, help the community manage the frequent duplicates, help with the (few) flags that we get here from time to time.

  • Participate in chat on occasion. A lot of ideas regarding moderation are discussed in the moderator room, like "what the heck do we do with these old and famous questions that aren't on topic any longer?"

I realize that some may view any help that someone can give as welcome, but I think that outlook sours when it becomes obvious that a slot that could otherwise be much more productive is being blocked.

While moderating, we come in contact with everything that the community objects to, which is frequently very smelly stuff. It really is not fair for some to spend the bulk of their time earning esteem, reputation and badges while the rest of us labor in the proverbial salt mines that double as an adult day care center.

It comes down to 'responsibility' in the term 'want the responsibility'. While it is a completely volunteer effort, I don't think it's unreasonable to imply a bit of obligation when mentioning the responsibility.

10
  • I agree that participation on meta and in chat should be encouraged, but we're not particularly lacking in those areas. I do hope that in future elections participation on meta will influence voting, but I think to address the issue at hand only a baseline of flags cleared should be established as an expectation.
    – Bill the Lizard Mod
    Commented Aug 9, 2011 at 13:03
  • @Bill - I was speaking in the over all sense. Still, per 1k flags serviced participation on meta is almost compulsory, so I guess the two would go hand in hand. I agree that setting some kind of bar for flags is the most important thing.
    – user50049
    Commented Aug 9, 2011 at 13:22
  • 4
    @BilltheLizard: Meta participation is not always thought of positively: twitter.com/#!/cky944/status/29549656941797376. Maybe it's the unicorns and silly stuff, or maybe it's the feeling that there's a clique of high-rep users (I've heard both reasons).
    – Michael Myers Mod
    Commented Aug 9, 2011 at 13:35
  • @mmyers: I hadn't considered that. We've reduced the amount of unicorns and silly stuff considerably of late, but I don't think there's any way around the perception that there's a clique of high-rep users here. I think that's a natural consequence of having a meta site. (Edit: Of course, I went to the front page and saw an "Eeek!" question posted by a cartoon unicorn right after saying that...)
    – Bill the Lizard Mod
    Commented Aug 9, 2011 at 13:45
  • 2
    New moderators require training and help initially -- True, but Kev and Lasse stepped into the fray with a minimal amount of hand-holding.
    – Robert Harvey Mod
    Commented Aug 9, 2011 at 16:53
  • @Robert Harvey Yes, they did. I mentioned it because many (describing those who might be interested in moderating) are often intimidated by 300+ flags, especially if called into question on MSO during their first month or so. The trepidation surrounding making a single mistake can be and often is something that people need a bit of help with to overcome.
    – user50049
    Commented Aug 9, 2011 at 17:02
  • 1
    Completely agree about meta participation. I only vote for people that I have seen make level headed decisions on meta.
    – John
    Commented Aug 9, 2011 at 17:43
  • You will always have the issue with those who delay until the end of their measurement time span to make their quota. On the one hand, they have lent a hand with the workload, but the other 6/7 days, everyone else had to make up for them.
    – rlb.usa
    Commented Aug 9, 2011 at 19:13
  • 2
    While moderators are few and especially picked for dedication, it is a deadline none-the-less, and a weekly one. A lot can happen in a week: people get sick, have crises, have babies, loose their jobs, or have other deadlines. It seems like a larger time frame might be better to reflect if the moderator is doing good work. I know this directly contradicts my statement above.
    – rlb.usa
    Commented Aug 9, 2011 at 19:16
  • @rlb.usa What I posted was just a suggestion for the start of a guideline (please take the question into account), and I did note that things do happen.
    – user50049
    Commented Aug 9, 2011 at 19:25
53

I don't have access to moderator tools. So, it is entirely possible that these options are already available. May be this is a stupid idea but here are my two cents about this topic:

In my opinion, I feel that moderators are doing a very good job. Moderators shouldn't be punished for a volunteer job that they do out of passion. Having said that, I would concentrate on ways to help them to do their job better.

Here it goes...

  • Classify the general users into four categories based on the criteria defined below.

Buckets

  • Gray bucket indicates users
    • with less than 500 flag weight.
    • and also have less than 60% of their flags being deemed valid.
  • Red bucket indicates users
    • with less than 500 flag weight.
    • and also have a minimum of 60% of their flags being deemed valid.
  • Yellow bucket indicates users
    • with flag weight between 500 and 700.
    • and also have a minimum of 80% of their flags being deemed valid. Anything less than 80%, they fall under Red bucket.
  • Green bucket indicates users
    • with flag weight above 700.
    • and also have a minimum of 95% of their flags being deemed valid. Anything less than 95%, they fall under Yellow bucket.

If someone belongs to Green bucket, we can make a reasonable assumption that the flags will be more likely legitimate and similarly the flags from users in Gray bucket would need more attention. So, the level of attention required on flags from Gray, Red, Yellow and Green bucket would very high, high, medium and low respectively.

However, there could be some disambiguations that may arise when Green bucket users flag a post that might need moderator's intervention. I believe this is where Moderator's should spend their valuable time instead of concentrating on flags from Gray bucket users. Following point discusses how to handle that.

Have a pool of volunteers from Green bucket for flag moderation. If someone has reached the Green bucket, I believe that they know what they are doing when they flag posts. These volunteers are not elected moderators but are very active members who are doing an excellent job in helping SO to maintain the quality of content. I call them Flag Validators. They can validate the flags from Gray bucket users, which I believe could form quite a percentage of 890 or so flags every day.

Flag validators should be given tools to validate only flags from Gray bucket users. Their good work as flag Validators can be used as a criteria during future Moderators election.

Diamond moderators can occasionally monitor how these Flag validators are doing their job and intervene if there are any serious disputes.

Diamond moderators can concentrate on flags from the top three buckets that need higher interpretation. This way they can better utilize their time avoiding flags from users who don't understand the concept of flagging.

I hope that I am making some sense here.

Flag Moderation

5
  • 6
    In addition, users with 10K rep should be automatically made into "Flag Validators" where they can vote on flags. Two votes makes a Flag Valid. Commented Aug 9, 2011 at 17:17
  • @0A0D: I was trying to suggest that Diamond Moderators don't have to watch Flag Validators closely thereby relieving the mods from checking trivial flags. The dotted arrow indicates that mods will intervene only when there is a dispute from the action taken by Flag Validators. If mods also vote on Flag Validators, then that doesn't lessen the burden of moderators.
    – user162697
    Commented Aug 9, 2011 at 17:23
  • 3
    @0A0D: Sorry, never mind. I read 10K reps as diamond moderators. I didn't go by the rep points because rep points don't necessarily correlate with flag weights. Hence the reason, I classified users by flag weight because the topic is about flagging.
    – user162697
    Commented Aug 9, 2011 at 17:25
  • 2
    I think part of this are already done: the number and flag weight of the flags on one post/comment determine the order in which diamond moderators get to them. Commented Aug 10, 2011 at 16:32
  • 1
    One problem with this is a positive feedback cycle: the more "trusty" a user is presumed, the less scrunity their requests get, the more likely they are to be accepted and earn them ever more "trust". The "blind test" solution mentioned by Paulo does look like the best approach here, negating this bias. Commented Nov 15, 2016 at 5:09
42

It's long bugged me that there is a double standard for moderator duty. On every Stack Exchange site except this one, the standard is doing moderator-type things once in a while. As long as an inactive moderator isn't destructive, the only reason we de-diamond is to avoid a minor security risk. It's not a huge deal to appoint or elect a more active moderator while retaining less active moderators. We can always create more diamonds, if needed.

When I was a moderator on a tiny site, I really felt it was a luxury to have a less-active moderator waiting in the wings. It's handy to have someone in a respected position who hasn't handled many flags to step in when there's a conflict. When there are only a handful of flags pending at any one time, clearing flags is less important than being a mediator and leader in the community.

But on Stack Overflow, the primary (and often only) responsibility of a moderator is clearing flags. Since there have been 2.8k flags so far today and since there are 18 moderators, we need them to average 150+ flags handled to stay above water. In practice, that means some moderators handle hundreds of flags a day while others clear far fewer. Moderators have access to a leaderboard that shows how everyone is performing, so the discrepancies are obvious.

I've had the privilege of talking with several former Stack Overflow moderators (who happen to be co-workers) and heard firsthand the frustration with being unable to clear the flag queue. When there are thousands of unhandled flags aging away, the moderator leaderboard turns out to be discouraging. It's easy to blame the people at the bottom of the list for not pulling their weight. Having handled another type of support queue, I can certainly see how this is a problem.

Ultimately, there are only two things we can do to scale up with the flag queue:

  1. Increase the number of people handling flags. (We just had a second election this year.)

  2. Reduce the size and difficulty of the problem. (We've tweaked the criteria for certain flags to be put in front of moderators and smoothed out some of the rough edges of moderator UI. There's still plenty to do there, however.)

Unfortunately, a standard of duty tends to hurt us on #1 and not make any difference on #2. If anything, removing moderators for inactivity reduces the overall effectiveness of the team. I've done just enough flag clearing to know it's a skill that I haven't developed. Recently we've seen moderators who have handled 10s of thousands of flags removed from the role. That's a lot of experience to lose. Just as bad: it's not a great way to send off people who have done so much for this community.

Moderator Emeritus

One solution might be to create an emeritus position we could offer to particularly deserving moderators upon their retirement. I envision moderators being eligible if they have cleared a large number (50k?) of flags. Functionally the main difference would be to move emeritus moderators to a new section of the moderator dashboard. (There's already a separate section for employee moderators.) They also would not be shown the flag indicator in the top bar. (They would still see flags in situ or by visiting the moderator dashboard.) Otherwise, they retain full access to the moderator tools and chat rooms. These moderators would not be asked to clear flags (though they certainly can). Rather, they continue serving the community as mediators, advisors and by guiding new moderators in their duties.

The goal would be to continue applying a standard of duty for moderators on Stack Overflow, but not push out experienced moderators who have burned out. We can "free up a slot" for new blood without sacrificing the good will of long-serving volunteers.

20
  • 1
    "Recently we've seen moderators...removed from the role." Use of "removed" sounds to me like it was due to some problem, which probably isn't what you meant?
    – jscs
    Commented Dec 10, 2015 at 1:05
  • 9
    I envision moderators being eligible if they have cleared a large number (50k?) of flags If you're going to put these people in a position to lead the community, guide other moderators, and mediate tough situations, then I'd think that you'd want to put people in this role because they've demonstrated strong leadership skills, are respected by the community at large, are one of the people mods go to when having a problem, etc. Those don't necessarily correlate to people with a high volume of flags handled. I'd really look at different criteria for selecting such mods.
    – Servy
    Commented Dec 10, 2015 at 2:06
  • 1
    Another option is to have tiers of flags for mods. Have all flags cast go to the level 1 queue, that's primarily serviced by less experienced mods, and give them an option to escalate the difficult flags to a level 2 queue that more senior mods focus on, so that the more experienced mods are handling the tickets that actually require the expertise that they have. And when handling harder/more involved flags, there would of course be a lower expectation in numbers of flags cleared.
    – Servy
    Commented Dec 10, 2015 at 2:09
  • 17
    Communication with the mods who have been inactive about why they were inactive probably would go a long way.
    – casperOne
    Commented Dec 10, 2015 at 2:51
  • 8
    Oh! I guess you really did mean "removed".
    – jscs
    Commented Dec 10, 2015 at 3:54
  • So the only functional change would be moving them to a new section on the dashboard? They keep their diamond, and presumably their spot on the /users?tab=moderators? Seems like the communication problem would still play a role.
    – Undo Mod
    Commented Dec 10, 2015 at 5:35
  • 1
    @Undo: Correct. I think an emeritus option would help, however. Part of the problem (in my estimation) was that we have one imperative: hold moderators accountable to a standard of duty. That's fine if we are looking ahead one or two years. But now that we have moderators pushing 5 years of duty, it's just not reasonable to ask them to keep up a high rate of flag clearing. At some point, it should be possible to step away from the grind. We just haven't considered that scenario seriously enough. (Clearly we didn't handle other bits of the communication problem in this particular case.)
    – Jon Ericson Staff
    Commented Dec 10, 2015 at 6:43
  • 3
    I somewhat wonder if it's a good idea to bring far more people up than we have been (already mentioned this in TL, for context) - I get that communication is an issue, but at this point communication might end up scaling better than the current flag queue is. Definitely agree that it isn't reasonable to expect a mortal human to handle mass quantities of flags for years upon years.
    – Undo Mod
    Commented Dec 10, 2015 at 6:52
  • 16
    @casperOne: It looks like we screwed up somewhere along the way. Several somewheres, in all likelihood. You put in years of hard work and it's got to be terrible to feel like we summarily dismissed you and your contributions. I'm so sorry. This is a problem I'd like to fix for the future and, if possible, make things right with you.
    – Jon Ericson Staff
    Commented Dec 10, 2015 at 7:13
  • 2
    @Servy: While I agree with the premise that flag clearing is not a sufficient criteria for finding leaders, that's not what this proposal is about. Rather it's a recognition that getting elected and handling tens of thousands of problems on the site are, together, strong indicators that these users are already worthy of the honor. It also gives existing moderators something to aspire to.
    – Jon Ericson Staff
    Commented Dec 10, 2015 at 7:30
  • 2
    @JonEricson Yes, and I support all of that. I'm just thinking that maybe the mods should select their own emeritus mods, or the CMs should select them, or something like that, maybe with a minimum number of flags cleared as a nomination criteria.
    – Servy
    Commented Dec 10, 2015 at 13:51
  • Have you considered posting this as a separate question? I'm not sure it's really getting the attention it deserves as an answer to an old question (unless you're not really looking for feedback, which is fine)
    – Undo Mod
    Commented Dec 11, 2015 at 2:53
  • 4
    I support this only if Bill the Lizard is required to return as moderator emeritus.
    – Michael Myers Mod
    Commented Dec 14, 2015 at 17:21
  • 1
    Therein lies the problem with this idea, @MichaelMyers: it'd make it harder for folks to step down. Even if they really, really wanted to / needed to. Bill's not the only one who's wanted to step down purely to free up time for other distractions; this kinda reminds me of the "delete but not really and we'll still send you emails" feature on Facebook - you'd never really be able to get away from it.
    – Shog9
    Commented Jan 4, 2016 at 19:38
  • 4
    @Shog9 but with "They also would not be shown the flag indicator in the top bar." there (hopefully) wouldn't be the pressure to visit the flag queue unless you really, really, really wanted to.
    – ChrisF Mod
    Commented Feb 8, 2016 at 15:37
37

The logical first step would be to 'reach out to' the 'underperforming' moderator to find out if it is a temporary hiccup (personal life), if they have misunderstood their responsibilities, or if they've just run out of steam. Said discussion is best undertook by someone with a modicum of sensitivity...

If they have run out of steam, maybe asking the question will make them realise that they don't really want to be a moderator anymore anyway. Give them an "I resign" button and the problem is solved.

Now I'm guessing that you've already done that, so now the question is (my inference):

Can SE decide to 'fire' moderators, and on what basis?

My answers:

Yes, if 'due process' is respected (as outlined above), and on a 'reasonable' basis that can only be determined (in my opinion) by the moderators, in conjunction with SE.

2
  • 7
    I agree that this is the logical first step. Priorities in people's lives change over time, and not everyone is able to volunteer several hours every week. I'm only guessing, but I imagine a few of the current mods would be willing to step down if it meant someone with more time to volunteer would take their place.
    – Bill the Lizard Mod
    Commented Aug 9, 2011 at 12:30
  • 8
    @bill to be very specific, it is time spent on Stack Overflow while not resolving any flags that is the issue. If you are not on the site at all, this does not apply. Obviously you can't resolve flags if you have no internet access, etc. Commented Aug 10, 2011 at 21:56
29

I think it's entirely reasonable to expect a minimum amount of flags processed on a weekly or monthly basis - I'm opposed to a daily expectation, as my own personal day-to-day can change drastically permitting time on SO one day, and none the next.

I do wonder if there's potential for some misunderstanding regarding the involvement of a moderator as measured by flags. If I'm not mistaken, flags only measure a moderators reactive events, and not their proactive events. For instance, while working on a flag five minutes ago I noticed two other non-answers in the same question that needed to be dealt with, but weren't flagged. Thinking that I may not get credit for these items, I went ahead and flagged them first, then addressed them afterwards (perhaps that's what I should be doing, rather than dealing with them immediately?)

One other improvement that could be explored is sorting flags not only by the the flag-author, but sorting them also depending on which moderator is reviewing them. For instance, I'm a PHP/CSS/jQuery guy, not a C#/Java/Python guy. If I see flags claiming some esoteric-sounding Java answer is 'Not an answer,' I'm really not in any position to evaluate that answer (in some cases the answer will be obvious, but not in all). Perhaps some type of shifting of content around for moderators can be done according to the tags or categories we're most active in.

I may not have a solution, but I do see the problem (and see my part in the problem). Glad to see we can discuss this and work toward resolving the issue, and bettering the experience for others.

5
  • 1
    I would want the same (sort by tags) for the 10k user moderator tools. Of course, without letting flags without any moderator with specific field knowledge drop away. Commented Aug 9, 2011 at 15:30
  • 3
    I don't like a "minimum amount of flags processed on a weekly or monthly basis" as in most of the world people often go on holiday for 2 or 3 weeks. Commented Aug 9, 2011 at 15:33
  • "short by tabs" using the same system as the home page would be good for the "surgested edit" lists as well. Commented Aug 9, 2011 at 15:34
  • 5
    You shouldn't pre-flag your tertiary actions. Flags imply a degree of community consensus, which is absent if you are processing your own flags.
    – Robert Harvey Mod
    Commented Aug 9, 2011 at 16:59
  • 10
    @Robert That was what I suspected as well; however, that exposes a fault in the system that moderators who are proactive may appear to be inactive to some degree.
    – Sampson Mod
    Commented Aug 9, 2011 at 18:14
18

I'm going to take this opportunity to suggest something I have been mulling over, but have yet to drop in a question.

I feel I'm chumming the waters in the -1 tank and then jumping in, but here it goes...

What about giving moderators some rep for clearing flags?

Not a lot... Less than accepted edits for < 2000 rep users.... .1 per flag?

The reason I suggest this is that we (should be) spending time clearing flags rather than answering questions, so we lose out on the rep we could be getting during the time we can allot to SO.

There, I did it. Seems petty, but it might help motivate mods to clear out more flags.

9
  • 4
    mmmm... chum. Commented Aug 9, 2011 at 10:29
  • 14
    After reading the first two sentences I was afraid the next line would mention bitcoin. In all seriousness I suppose that a bigger carrot at the end of the stick might help, but I'd (personally) like my rep to come from answers that I provide .. or (what little I have) kind of loses its significance, at least for me.
    – user50049
    Commented Aug 9, 2011 at 10:50
  • 1
    @TimPost: Yeah, I completely understand. But, as an active moderator, we do have to split our time between handling flags and answering questions. It isn't exactly an equal tradeoff (assuming the answers we would normally be providing aren't worthless), but at least we do get some compensation for our work (other than the occasional nastygram or "why did my flag weight just go down?" questions here).
    – user1228
    Commented Aug 9, 2011 at 11:06
  • I always thought the ability to click that "invalid" button was compensation enough for you. Has this changed?
    – Cody Gray Mod
    Commented Aug 9, 2011 at 12:13
  • 10
    @CodyGray: Oh, that gets me through some hard times, but without actually seeing the users driven before me and without actually being able to hear the lamentations of their women and children it just isn't that satisfying.
    – user1228
    Commented Aug 9, 2011 at 12:17
  • @Tim Post: Let's not cast this bitcoin idea aside without giving it the consideration it deserves. ;)
    – Bill the Lizard Mod
    Commented Aug 9, 2011 at 12:24
  • 2
    @Bill My worry is that they'll consider it during the next podcast, and we'll have to make all new bingo cards.
    – user50049
    Commented Aug 9, 2011 at 13:13
  • 8
    Rep has very little value once someone has enough to be a mod on SO. Commented Aug 9, 2011 at 15:28
  • 10
    @IanRingrose: But what about catching up to Jon Skeet???
    – user1228
    Commented Aug 9, 2011 at 18:52
18

I don't know what a reasonable weekly (daily, monthly, etc.) workload is for a Stack Overflow moderator, but I don't think it's at all unreasonable to expect moderators to perform the duties they were chosen to perform.

It's a volunteer position, sure, but I could go out and volunteer somewhere in my town and I can bet they'd politely tell me to not bother if I didn't actually do the work. There's no harm or shame in stepping down if one can't really fulfill the commitment anymore because their circumstances or interests changed.

There is also potential for absentee moderators to miss important announcements or changes. So when they come back, they might act in ways that are no longer fitting in with the site.

With that in mind, I think it's important to look at consistency of action instead of raw flag processing power. Someone who deals with 20 flags a week but is doing so consistently every week is a more active and involved moderator than someone who shows up every 3 months and blows through 240 flags in one go.

16

I'm all for giving more guidance to moderators about what is expected of them: I think many people run for election on various platforms like "hands-off" or what-have-you and have no idea, if they become a moderator, the amount of work that's involved in being one.

But reading your post and talking to waffles in chat, it sounds like replacing—instead of supplementing—moderators who are below-average in their activity is at least in part about making sure the moderators who are pulling their weight aren't demoralized or feel their moderatorship is cheapened because they have the same status. That it's cause for the hard working moderators to say, "hey, if Joe Smith isn't doing his fair share, why should I?"

It's all a little too Ayn Rand for my taste. Who is John Galt?

I can understand that it can be demoralizing to take care of a ton of flags, then check the moderator dashboard to see Joe Smith didn't handle very many at all this month but got 2,000 rep answering questions. But I think the suggested cause for the demoralization wrong: it's not because my diamond is cheapened because a "non-moderator" moderator also has one, it's because Joe Smith got something for his time on the site while all I got was another 100 flags. The incentive structure is messed up.

One way to solve that is to increase the incentives for doing moderator activities: facetiously, it might be to give rep for handling a flag or something silly like that. But besides rewarding rep for an activity that doesn't promote the core function of the site (Q&A), it attracts the wrong kind of moderator.

And it's really the other side to the same coin: whether you remove moderators who aren't as good as the diligent ones—thus rewarding the diligent ones with a "more meaningful" diamond next to their name—or you give rewards for doing moderator activities, you make moderating more about getting a carrot than about doing one's duty.

That is, people volunteer to become a moderator because they love a particular site and they want to do their part to keep it running smoothly: they consider it to be their fulfillment of the social contract. You want people who are actively engaged in the main site, even when the flag list is overwhelming, because they are the type of people to take it seriously: they have skin in the game, and they want a better site because they use it themselves. The reward for them isn't a more meaningful diamond, it's a better site.

At the same time, I do agree it is untenable to have an ever-growing list of moderators to supplement the changing interests of what are ultimately volunteers. But instead of making it about meeting a quota to ensure the diamond next to one's name is a substantial reward, make it a referendum.

Every six months (or if you think it's necessary to evaluate every 2-3 months, do it every quarter), hold elections: if someone wants to keep being a moderator, they can run. If they can't be bothered, they can leave and no harm no foul. Make the moderator statistics on the dashboard visibile during the elections and let the community decide whether that person is pulling their weight.

3
  • 10
    I don't think the correct answer is to massively increase the number of moderator elections, and force all existing moderators to re-run every time. That'd create a lot of stress for everyone involved. This is also utterly and completely irrelevant on every single other site except Stack Overflow. It's sort of like the difference between being the mayor of a small town and a member of the US Senate. Commented Aug 9, 2011 at 9:35
  • 5
    @Jeff Senator or mayor, you run for the gig knowing what it entails and you work with what you're given. The president doesn't get to fire senators who don't pull their fair share: if a senator doesn't do his job, he doesn't get reelected. If a senator thinks the job sucks, he can resign. While I think 6 months is an eternity in the life of an internet community, I don't think being a moderator constantly in fear of potentially getting the axe for being at the bottom of the list is any less stressful.
    – user149432
    Commented Aug 9, 2011 at 9:54
  • 8
    no need to fear anything if you're performing the basic functions of the position you were elected to. We had a moderator on math who got elected and never even showed up, for example. These are extreme cases; I am not at liberty to share the data but subtle, it ain't. Commented Aug 9, 2011 at 9:57
12

I suppose this is a logical next step after the history page breaking down participation per mod and per action. I would certainly expect mods to be clearing flags frequently, but as long as they're doing something (however minimal), is there a reason not to let them keep at it? In other words, what's the point of losing whatever amount of effort they are putting in? There isn't a maximum number of moderators -- more can always be elected -- so what's the benefit in dropping people that aren't dismissing enough flags? If someone clears 5 flags/day and you un-mod them, that's 5 more flags every day that the other mods will have to take care of, and I'm not sure what was gained

1
  • 17
    the point is to make it clear that community moderators, if they wish to be elected, need to participate in the primary duty of a moderator when they are on the site. Otherwise they are "moderators" in name only. Commented Aug 9, 2011 at 6:48
9

I'm not a mod, so I don't know the behind-the-scenes details, but it seems reasonable to quantify somehow the expected work of a moderator.

There are two points that I think are important to make sure this stays reasonable, though, given the volunteer nature of the job.

First, you're talking about moderators performing their duties, primarily the processing of flags, while on the site. Such duties should not interfere with the moderator's ability to be a "civilian", i.e., use SE for their own research. In other words, I don't think a mod should be expected to be processing flags any time the system has them logged in. Your proposed "standard of duty" will probably help with this, in fact. Given a standard, moderators can know when they have put in their time for the day and can feel free to use the site in a personal capacity.

Second, I think throwing bodies at this problem is a good thing to try -- I don't think it's "avoiding the issue" at all -- and your proposal would also make this easier. Once you come up with a rough figure: "A mod should process N flags a day on average", you can do some simple division and figure out how many moderators a site should have in order to keep the per-moderator workload stable.

8
  • 4
    if those "bodies" close ~50 flags in a 30 day period, while being active on the site the entire time.. and the average number of flags per day is ~900.. I'm sorry, how does that help again? Commented Aug 9, 2011 at 7:26
  • 2
    I say "Yes" to the standard -- any new moderators will be held to the same standard. The "more bodies" part is, once you establish the standard, deficit = num_existing_mods - (900 / N) tells you that the site needs deficit new moderators.
    – jscs
    Commented Aug 9, 2011 at 7:43
  • 5
    @Jeff Josh is saying he supports your proposal -- but it's up to you to pick a good number for N (number of flags to close per day on average). He's simply saying that if "reasonable" still ends up with a deficit, having a standard will help you figure out how many more moderators you need.
    – Nicole
    Commented Aug 9, 2011 at 8:00
  • @renesis will we get soldiers who fight, or soldiers who only eat rations? It does nothing to resolve that problem... and adding a bunch more soldiers and/or rations does not either... Commented Aug 9, 2011 at 8:03
  • 2
    Sure, but I'm a little confused, since it seems there is a misunderstanding. It's your proposal, I think this is just one argument in favor of it.
    – Nicole
    Commented Aug 9, 2011 at 8:11
  • 2
    @Jeff: Which rations are inactive moderators eating? Commented Aug 10, 2011 at 16:35
  • @paulo unless you want to have hundreds of moderators, they are "eating" a moderator slot without performing the most crucial function of a moderator. Commented Aug 10, 2011 at 21:23
  • 3
    @Jeff: So, the problem is that you don't want to have to trust hundreds of people with the moderator powers? (This is okay, it just should be pointed out.) Commented Aug 10, 2011 at 21:31
8

I don't think that forcing a moderator to meet some standard is the the right approach. Some of us have more time than others, or just happen to be able to monitor the sites more frequently. At least on the SE 2.0 sites, if you only are on in the evenings, say, you might miss the only flag of the day because another mod was able to watch the site at work. (Obviously this doesn't apply to SO; my point is that there are higher traffic times that if a moderator doesn't catch, he won't appear to be doing as much.) There are some mods who seem superhuman in flag-clearing abilities; they can rip through a couple hundred flags without missing anything, while others need a bit more time to think about each one.

If moderators are the community's exception handlers, shouldn't SE be the moderator exception handler? Meaning, if a community coordinator finds evidence that a moderator isn't pulling his fair share, he can look into the situation furthur. Base this on the moderator statistics if you want, but don't publish the criteria, and for sure don't make it something we have to worry about. Use it as a guideline to alert the community coordinators that there may be a problem, and then handle it on a case by case basis.

3
  • 7
    I don't think Stack Overflow suffers from the "only flag of the day" problem.
    – Adam Lear StaffMod
    Commented Aug 9, 2011 at 13:45
  • 1
    @AnnaLear Obviously not :) It still has it's high times and low times though, which was what I wanted to emphasise. Another thing - if several mods all clear flags from 10-3 say, and another mod is active at other times, those several mods will probably not be clearing as many flags apiece simply because there's more active at that time.
    – Michael K
    Commented Aug 9, 2011 at 14:01
  • I get what you're saying here, but I think you're still missing the point that there are plenty of flags to go around on Stack Overflow at almost any time of the day that you happen to be around. A lot of people who haven't been SO moderators simply can't appreciate the sheer volume of flags that are raised on that site. And having not been one myself, I can only go by stories and screenshots that I've seen the moderators post. I don't really think this is the problem when it comes to SO in particular.
    – Cody Gray Mod
    Commented Aug 10, 2011 at 4:05
5

Processing flags isn't the only useful thing a moderator can do. Judging performance only by flag processing seems unfair and not necessarily useful.

1
  • 3
    it is of course not the only thing, but it is the primary and most important duty. Commented Aug 10, 2011 at 21:25
4

Perhaps diamond-hood should have a term limit?

Or, maybe, two types of moderators: full diamonds can box people and otherwise take radical steps. Mere paste moderators can just act on flags unilaterally. Maybe, oh, 30K makes a paste moderator?

3

I don't know what the statistics are on this, but bear with me...

Yes you cannot keep adding moderators to remedy the problem because the problem will keep growing as the user-base grows, thus more and more moderators... One thing that comes to mind, how often do moderators see people abusing flags? Are there steps in place to warn or maybe even take rep away from users who are caught abusing flags? Do the users know that they will lose rep when they abuse flags?

Also, the idea of using high rep users to sort out flags is an interesting thought...

Why not allow users with a considerable amount of reputation (at the moderator's discretion) "Apply" to be a "Flag-Moderator", and maybe give these Flag-Moderators a point system of their own, separate from the regular user point system based on number of flags covered in an hour/day/week/month/all time, the number of topics of flags covered, etc. This will give the same sense of "I MUST HAVE POINTS NOM NOM NOM" as the rest of the stack exchange community, while helping to remedy the problem.

On another note, when someone VOLUNTEERS for a position (such as Moderator) you DO NOT want to give punishment as stated in the first post

I propose that on Stack Overflow, all elected community moderators must close a 'reasonable' number of flags while they are on the site.

Yes moderators should be pulling some weight because they took on the role, but imposing a consequence of a type of "Quota" will only shun the idea of becoming a moderator. Instead install a reward system over going for a penalty system when it comes to volunteers. If moderators are inactive they should be let go from the position.

1
  • 4
    Users don't lose rep for bad flags, but flags are sorted by flag weight -- if a user posts bad flags they'll get marked invalid, their flag weight will go down, and their flags will go down in the list. As for your last paragraph, you said not to have a penalty system and then ended with "If moderators are inactive they should be let go" -- I'm pretty sure that's all Jeff is talking about, the "penalty" is not being a mod anymore Commented Aug 9, 2011 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .