17

I use "Other" flags on posts quite a bit, mostly to ask the mods to purge all of the post's comments when they're obsolete or non-constructive.

Many of these get handled within a couple of hours. I've had 6 handled within the last 2 days.

But others - including very similar flags, just asking for comment purges! - remain in review for ages. My oldest unresolved flag has been in the queue for 20 days now.

I'm curious - what's going on behind the scenes? How come I've had flags from the last 24 hours handled within hours of creating them despite having nearly month-old flags still in the queue?

14
  • 2
    We've been averaging 2.5k-3k flags in the queue lately. Some of your flags might get handled based on how a mod is working through the queue. Other flags will always linger for longer especially when there are 1k of them to process.
    – Taryn StaffMod
    Commented Nov 7, 2015 at 13:24
  • 9
    @bluefeet sure, but the question isn't "why is there a backlog?" - if it took a flat couple of weeks for my flags to be processed, I wouldn't be confused. The question is why my flags from the last couple of days have been handled while flags of the same type from weeks ago are still waiting to be processed - are they being presented to you in some non-chronological order, perhaps deliberately showing you the most recent flags first, or do you have some mechanism for putting a flag on hold to deal with later that's been used on some of my flags? Or is something else entirely going on?
    – Mark Amery
    Commented Nov 7, 2015 at 13:35
  • 1
    There could be any number of reasons - flags could be filtered in any number of ways. It's possible that they were filtered in a different way for a few days this week (by a mod) and where handle.
    – Taryn StaffMod
    Commented Nov 7, 2015 at 13:39
  • @bluefeet okay, that makes sense - I didn't know that your review queues had a bunch of filter options that us proles don't get to use!
    – Mark Amery
    Commented Nov 7, 2015 at 13:50
  • 7
    Maybe the mods have a life, (yes, perhaps even Brad and Shoggy). Commented Nov 7, 2015 at 14:05
  • 17
    @MartinJames as with bluefeet's first comment, you're missing the point; I'm not asking "why do my flags take a long time to be dealt with", I'm asking "why do only some of my flags take a long time to get dealt with". (The answer, per bluefeet, seems to be that the mods have lots of options about how to filter flags, and a small subset of my oldest unresolved flags have by some coincidence ended up being consistently filtered out by the majority of mods most of the time that they've spent in flag review - it's got little to do with how rapidly the mods process flags.)
    – Mark Amery
    Commented Nov 7, 2015 at 14:09
  • 7
    Some of us work bottom up, others top down. The problem is the top down is harder than bottom up, but we can clear more flags if we work bottom up.
    – George Stocker Mod
    Commented Nov 7, 2015 at 14:19
  • 3
    @pnuts: this request is sort of tangentially relevant to your comment: meta.stackoverflow.com/questions/278927/… Commented Nov 8, 2015 at 19:29
  • 1
    @pnuts I disagree utterly about cleaning up comment noise being low priority, at least on highly-viewed questions. I don't feel I can trust an answer until I've read all the comments on it to look for warnings and caveats and claims of brokenness; if the comments are just crap and noise, that means I waste time. Even if it's only an extra minute to read a comment thread, multiply that up by tens of thousands of people doing the same thing and a single unpurged comment thread can waste man-days of productivity. And that's not even touching on the risk that more important comments go unnoticed.
    – Mark Amery
    Commented Nov 8, 2015 at 20:00
  • Sure, editing badly-presented posts is very much more important than noise cleanup, @pnuts - if you're counting editing as a "mod activity by users" then I'll grant that it's the most important of them all. But most other moderation activities - closure, post deletion, comment deletion - is just about noise removal, and comment removal is no less important than the others; honestly I think it's more important than question closure.
    – Mark Amery
    Commented Nov 8, 2015 at 20:07
  • With that many flags there should be more moderators. I nominate myself.
    – JonH
    Commented Nov 9, 2015 at 19:15
  • @JonH You summoned it!
    – Kendra
    Commented Nov 9, 2015 at 20:06
  • 2
    @Kendra - Seriously? That was weird!!! I didn't even know that was happening.
    – JonH
    Commented Nov 9, 2015 at 20:07
  • @JonH I literally read your comment then checked for new Meta questions and saw that. Guess you better go fill out your nomination. :)
    – Kendra
    Commented Nov 9, 2015 at 20:07

4 Answers 4

39

I can provide my own (detailed) perspective on the variable handling time for "other" flags. As bluefeet states, this largely has to do with the incredibly diverse nature of these flags and the fact that all of these flags are mixed together in an unordered nature.

When we visit the flag queue, we see a dump of all flags of all types, sorted first by posts with multiple flags on them and then by chronological order. Posts that rack up multiple flags on them usually indicate something we need to look into sooner rather than later, so those tend to get acted on first.

For sorting, we can refine this to specific flag types: "not an answer", "very low quality", and spam / offensive. The spam / offensive flag sorting appears in bright red at the top of that list, and we tend to handle those before anything else.

"very low quality" and "not an answer" flags are now provided for community review in the Low Quality Posts review queue. They are delayed for an hour before they appear in the moderator queues, but it doesn't appear that extending that time would do much to improve the rate at which those are being handled.

The timing for how those flags are handled largely depends on the ebb and flow of moderator participation. We've had a few active moderators lately called away simultaneously for personal or work reasons, leading to an extension in the number of unhandled flags here. There also have been a higher number of bad flags here lately (people flagging any answer they see below -2 as "not an answer", flagging competing answers to theirs, flagging paragraph-long answers because they happen to have a link in them, etc.). I imagine the number of these will come back down as people return.

The "other" flags are where things often get jammed up. When processing flags, I'm most effective when I can get into a specific mindset and work through all flags of one type, then all flags of another. "Other" flags are all over the map in terms of quality, actions, urgency, and time to process. Aside from chronological order, they are completely unsorted.

I try to approach the flag queue by triaging flags based on urgency and how confident I am in what to do with them. Most of the "other" flags that take longer to handle are ones where either moderators weren't sure of what to do with them or they came in when moderators weren't around and were surrounded by these difficult-to-handle flags.

The "other" flag queue also unfortunately has a large number of absolutely terrible flags. For example, these are six actual flags I handled in a row on Thursday morning:

Sir, i got deadline tonight. Please help. Thank you..

Please I need answers to my question. I'm yet to resolve this challenge.

please Help for me..........

Need urgent help!!

Expecting a solution

need urgently

We also see a large number of people still using "other" flags instead of close votes, to indicate answers they think are wrong, to demand that we accept a certain answer, etc. All of these flags waste our time and bury legitimate issues.

Moderators have been working with Stack Exchange employees to improve this process, and it's clearly something they're putting effort into. One area that has shown promise is keyword or regex filtering for "other" flags. As an experiment, several of us are using an SE-provided userscript to filter flags and I've found it to be very helpful in grouping types of "other" flags.

For example, I can pull out many plagiarism flags by filtering for "plagia" or "copied" and apply a 1-2-3 workflow for handling those rapidly. Some might be from a week ago, some a few minutes old, but I can process these all at the same time. Likewise, I can filter for all the college students asking for their posts to be deleted (and decline almost all) by sorting on posts flagged by the original poster who have "delete" in them.

Somehow incorporating these capabilities into the system will be a huge help, and I know people at SE have been working on this for a while. Beyond that, I've been thinking about ways of implementing time-of-flag warnings for people attempting to leave terrible flags like the above and catching other cases where flags are being used improperly (migration requests on old questions, any flags about accept votes, etc.).

Sorry for the length of this, but I've seen some mistaken impressions about how this process works and I wanted to provide my thoughts from what I've seen over the years.

4
  • Is there really no way to mark a custom flag as "decline anything matching this"? Because I can feel the pain for those 6 flags, but I really wonder whether at least 4 of them aren't encountered often enough to fully automate their handling... Commented Nov 8, 2015 at 20:22
  • 5
    @Deduplicator - Every flag has to be processed individually. I'm trying to identify heuristics for picking out "likely bad" and "likely good" flags. I've identified some words and word pairings that are common for these flags. Context can also matter, such as a flag with "migrat" in it that was placed on an old question. I'm thinking that time-of-flagging warnings might be a way to stop these while avoiding problems we've had with outright banning keywords. Let people know that flags aren't for asking us for answers, that we can't migrate things, that close votes are better, etc.
    – Brad Larson Mod
    Commented Nov 8, 2015 at 20:35
  • 1
    More examples: meta.stackoverflow.com/questions/284338/…
    – BoltClock Mod
    Commented Nov 9, 2015 at 5:12
  • One thing that would help is a streamlined queue processing interface. Shog's user script should be tweaked and be standard for it. Jeff's idea of making declining 'harder' didn't scale with the site at all, and effectively slows down moderators by 50% without really helping anything.
    – George Stocker Mod
    Commented Nov 9, 2015 at 15:01
20

There are many, many reasons why it takes a variable amount of time for "other" flags to be handled. First, there is a gigantic backlog of flags - the SO mods have had anywhere from 2,500 - 3,000 flags in the queue on a daily basis. Of those, about 1,000 of them are custom "other" flags. These tend to take a bit longer to process because there is more investigation for them.

Mods do have ways to filter flags, including comment, VLQ, NAA, and other, plus some mods use additional user scripts to filter even further. You might have just been lucky this week and a mod came across a few of your flags that were sitting waiting to be process. All the flags will be handled...eventually.

5
  • 1
    You should get a rotation for "handling other flags", that way mods get accustomed to the tools and figure out how to handle the most diverse situations.
    – Braiam
    Commented Nov 7, 2015 at 21:54
  • Note that the filtering of flags only extends to the canned options that you see on the flag dialog, as well as system flags. There is no built-in functionality to filter custom flags by search term (which is where the userscripts come in, but I've not found them to be particularly effective).
    – BoltClock Mod
    Commented Nov 8, 2015 at 4:58
  • 1
    By eventually, I assume you mean 6-8 weeks, right? Commented Nov 8, 2015 at 6:58
  • 2
    @pnuts Aged away means there is nothing for us to handle, the only flags that age away are close flags and mods don't see those.
    – Taryn StaffMod
    Commented Nov 8, 2015 at 14:40
  • @pnuts Mods can handle any/all flags except close flags - those go to the queue. They handle - other, VLQ, NAA, comment, spam, offensive - pretty much everything.
    – Taryn StaffMod
    Commented Nov 8, 2015 at 15:01
15

So, my workflow (I know the other mods have different workflows):

  1. Completely arbitrary time of day, i.e. whenever I feel like it, jump on the queue, go to "other...", and pick either end of the queue (oldest, or newest) to handle flags
  2. Do my best to ignore "urgent plz help" flags, "delete this question" flags, and "close this question" flags
  3. Fail miserably, spend all my time declining the dozens of these flags because it's so much harder to find useful flags in all of this noise
  4. Get disheartened and leave to do something else, having only handled a few actually helpful flags as collateral while cleaning up the bad flags

So, in short, the "other" queue is so unpredictable because it's too long to process all at one go (when it shouldn't even be this long in the first place!), and when your flags get handled immediately it's because you caught one of us on a good day. The rest end up in limbo.

It's been said time and again that they're working on revamping the mod queue, but you know what I personally wish for? That people knew when not to use custom flags in the first place. But given that SE deliberately blurs the line between non-diamond mods and diamond mods, I can't really blame the users for not knowing which option to choose. There's also the users who think flagging for diamond mods is a way of expediting their requests. But it's not (and, funnily enough, this is how they learn it the hard way).

6
  • 1
    @pnuts: If you flag an accepted answer as NAA it automatically goes to the mods. If an NAA flag is unhandled by review for 15 minutes it goes to the mods as well.
    – BoltClock Mod
    Commented Nov 8, 2015 at 7:43
  • 4
    "given that SE deliberately blurs the line between non-diamond mods and diamond mods" - this is important, I think. I went on a little crusade a year or two back against Meta users telling people off for using mod flags when close flags would be more appropriate, or other kinds of misflagging, when the flagging dialog gives pretty much no information a new user would understand about what the effects of the flags are or who will handle them.
    – Mark Amery
    Commented Nov 8, 2015 at 11:22
  • If an NAA flag is unhandled by review for 15 minutes -- this seems like an awfully short time considering the typical length of the low-quality posts queue.
    – TZHX
    Commented Nov 8, 2015 at 13:02
  • 2
    This may already be in place, but is it possible to try to stem the number of "please help" and "please close" flags? Even it it shows a message to the user to ask them to confirm that they are sure. e.g. when the word "close" is spotted in the text.
    – DavidG
    Commented Nov 9, 2015 at 11:53
  • @DavidG good idea. Probably not an outright "keyword ban" like one the +1 in comments, but a message stating "Are you sure that you want to flag this post for moderator attention? This is not the place to ask for help solving your problem/closing the post etc"
    – Adriaan
    Commented Nov 9, 2015 at 18:30
  • Previously from BoltClock: lots of bad flag reasons. Commented Nov 10, 2015 at 2:46
8

There are multiple reasons, and other moderators have stated some of them.

  1. Not all moderators are active all the time. If every moderator was active every day for 30 minutes a day, we wouldn't have a moderator queue issue. But we aren't, and the flags don't stop whether or not we are all active all the time.

  2. We do have some moderators who aren't very active and haven't been for a while. We also have some (like me) who have spurts of lots of activity (handling hundreds of flags) followed by periods where I handle a few dozen flags a day at most, and organically moderate (don't go into the queue).

  3. The moderator queue interface is a "Cobbler's children" issue: Since only 16 or so users ever see the moderator queue, it doesn't have to be great. That works if you have a site with only a few hundred flags; but that doesn't work when you have a site that routinely gets over a thousand flags a day.

    • The queue is not optimized for actually clearing large numbers of flags. To decline a flag, it takes 3 clicks, whereas to accept a flag and do nothing, it takes 2 clicks, and to delete the post, it takes 1 click.

    • If we were going to optimize the queue for clearing flags; then it should show us the most common accept/decline reasons one click away. That alone would make us a lot faster (because most non 'other' flags can be cleared with a high degree of accuracy in a very short amount of time).

    • The "move comments to chat" does not have optimized actions. It takes the following actions:

      1. Visit the post
      2. click "mod" -> "move comments to chat"
      3. Click the special "Show all deleted comments"
      4. Undelete the comment that told everyone the moderator moved the comments to chat.
    • The "duplicate post" flag doesn't have an optimized workflow. It takes the following actions:

    • Click on all duplicates in new tabs, opening them.

    • Check each duplicate answer, and each question, and make sure they're actually duplicates, and that keeping them around is not viable.
    • if they shouldn't be kept around, delete the one with the flag so that another moderator will be able to see it in the future.

    How workflows could be optimized:

Not an answer

Make "Decline" a first level action, so instead of this:

enter image description here

Moderators see this: enter image description here

Other

Another moderator brought this up, and I don't want to steal their thunder, but it's a really good idea. The 'other' queue should segment flags based on keywords. Often the 'other' queue includes flags asking us to 'migrate' posts. While not foolproof, including customized buttons that handle posts with the word 'migrat|e|ion|ed' in them, including an one-step-decline button if it's older than 60 days.

"Too Many Comments"

I addressed this above, but simply posts with multiple flags for comments or an auto-flag for too many comments should have streamlined handling. the top level buttons (one click) should be:

  • No Further Action (it's an auto flag, so it shouldn't need me to pick helpful/decline)
  • Move and Delete
  • Purge

Sock puppet "other" flags

Sock puppetry is one of the most time consuming parts of moderating the other flag. This is a stretch goal (as I don't think it'll ever get fixed), but we should be able to see side-by-side data for users accused of sockpuppetry right in the mod queue (terms like 'sockpuppet' 'targetted voting', or change the dialog to look at puppets). We should be able to see the trend of votes between this user and their accused puppet, as well as if they share any identifying information, right from the mod queue (please log it, as the system currently does).

Over all, the mod queue needs many little tweaks to be optimized for handling the load of flags we get quickly. If you trust us to be your moderators, you should also be able to trust us with the tools for moderation.

Jeff Atwood wrote in 2011 about this; but from a point of view that "declining should be rare", so of course the moderator queue's workflow reflects that. But times have changed. We now have multiple review and triage queues, and we use flags to ensure those queues are right. We also use those flags to warn/suspend/ban users through the system (no human intervention) so it's critical that flags are accurate.

If you want to help moderators handle flags faster, there are multiple things you can do:

  • Help push for moderator Queue UI fixes
  • Use one of the predefined flag reasons wherever possible
  • Don't flag when you should be voting to close, even if you are out of close votes
  • help educate others on Meta about proper and improper flagging (I've seen a lot more of this lately, and that's great).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .