-10

The descriptive text for the "Not an Answer" flag contains the text:

This was posted as an answer, but it does not attempt to answer the question. It should possibly be an edit, a comment, another question, or deleted altogether.

The descriptive text aligns with the reasons for the flag's existence, which are to bring to moderator attention answers that are actually:

  • New questions
  • An edit to the question
  • An attempt to communicate with another user, or
  • Something that doesn't belong (spam, gibberish, etc.)

These are the only reasons for the flag's existence. The NAA flag is unsuitable for anything else, including bad answers, wrong answers, rhetorical questions posing as answers, answers that answer the wrong question, you get the point.

Central to the flag's stated purpose is that you don't have to know anything about the question to identify a non-answer. The moderator flag system is deliberately designed to work this way; it does not show mods question context unless the mod asks for it (an operation that greatly reduces a moderator's throughput on such flags).

Unfortunately, we've never been successful at communicating workable guidance on this flag. Long experience has shown me that, if how something works isn't clearly obvious from the start, no amount of documentation, Help Center articles, Meta articles or pestering users is going to fix it.

There is ample evidence here on Meta that the NAA flag consumes a disproportionate amount of time and energy from the community. There are posts floating the idea of changing the flag wording, which never get any traction. There are numerous posts about someone complaining that their NAA flag got declined, and the subsequent arguments about what constitutes an answer get rehashed all over again.

As a mod, I don't want to be focused on the finer points of answer definitions. That conversation does not interest me at all. What does interest me is removing posts from the site that clearly do not belong, regardless of the reason they were brought to my attention, and not having to worry about getting into another hopeless argument on Meta over what the definition of "is" is.

23
  • 15
    What do we do when someone posts one of the four? Just downvote, vote to delete?
    – user1228
    Commented Mar 1, 2016 at 22:16
  • 5
    Use a custom flag, of course, and explain why you are flagging.
    – Robert Harvey Mod
    Commented Mar 1, 2016 at 22:18
  • 8
    To put a slightly finer point on what @Will may be saying: how do we remove answers from questions which would normally be actioned by this flag? Flagging a moderator for these on a custom basis seems a bit of a mess to me.
    – Makoto
    Commented Mar 1, 2016 at 22:18
  • 2
    @Makoto: Why? If you don't feel like explaining, put the words "Not an Answer" into the custom flag, and roll the dice.
    – Robert Harvey Mod
    Commented Mar 1, 2016 at 22:19
  • 4
    "Rolling the dice" is what puts me off a bit. But, fair point.
    – Makoto
    Commented Mar 1, 2016 at 22:21
  • @Makoto to be fair, since NAA are already going to moderators, we aren't losing anything by using a custom flag instead. Well, I guess normal users have to type slightly more but that's it.
    – ryanyuyu
    Commented Mar 1, 2016 at 22:27
  • 13
    @ryanyuyu NAA doesn't go to the mods, it goes to a review queue. Eventually, these are placed in the mod queue but initially they are only in review. The only time NAA flags bypass review is when it is placed on an accepted answer - those skip review and head straight to the mods.
    – Taryn StaffMod
    Commented Mar 1, 2016 at 22:30
  • I stand corrected. Especially after Brad's statistics.
    – ryanyuyu
    Commented Mar 1, 2016 at 22:32
  • 8
    So you (as a diamond mod) want to be personally involved in deleting "did you solve this problem kindly post answer thx" posts via a custom flag, instead of letting them go to the LQP queue? I agree there's a mild disconnect between what the community considers to be NAA and the official position of the diamond mods, but despite the meta noise I think the system is mostly working. (Sort of like how, for each complaint about audits, many robo-reviewers are banned.) Commented Mar 1, 2016 at 22:37
  • @JeffreyBosboom: Those sort of flags are very easy to handle.
    – Robert Harvey Mod
    Commented Mar 1, 2016 at 22:39
  • Perhaps the phrase "Unintentional Answer" would work better than "Not An Answer"? It covers all the use-cases listed in the current description, while also allowing bad/wrong answers. I'll admit, it doesn't necessarily cover rhetorical questions, or answers to the wrong question, but it does seem like it would be an improvement. Commented Mar 1, 2016 at 22:51
  • 7
    @TylerMacMillan - Moderators can't be judges of technical correctness. There's no possible way that the handful of us could cover the breadth of technologies on this site. You also don't want a small group of users to impose our decisions as to what's correct or not. There are good reasons for restricting this flag to just the obvious situations listed above.
    – Brad Larson Mod
    Commented Mar 1, 2016 at 22:56
  • 6
    People are just looking for a way to say "this post does not belong here". You can remove NAA, then they'll pick something else. The most obvious solution is to, you know, allow them so say what they need to say. Suppressing speech never works. Commented Mar 1, 2016 at 23:11
  • 4
    Well, it gets used, not exactly enthusiastically. Expecting somebody to expend an effort on a shite post does not work either. Commented Mar 1, 2016 at 23:19
  • 1
    The current widespread misuse of the flag is just a symptom. A symptom that people feel like they have no efficient and quick way to get tosh deleted. Delete-votes are locked behind more rep then most users will ever get. Flags aren't I think the community ought to adress that instead of taking away flags that are currently used. The mindset that most people use nowadays in the VLQ queue is "If its in any way tosh, I'll delete it"
    – Magisch
    Commented Mar 2, 2016 at 12:20

5 Answers 5

54

While I understand the intention, I don't think this is practical. In the past 30 days, Stack Overflow has received 24,000 "not an answer" flags. That's 800 flags a day.

If this flag category is to go away, with people now having to type in custom reasons for each of these flags, that's a lot of custom flags. Not only does this add a lot of typing for flagging users, I think that would make the custom flag pile an even bigger mess.

We currently have no means (outside of experimental userscripts) to sort custom flags, so these would get randomly dumped into the middle of the rest. We handle flags fastest when we can go to specific categories, get into the mindset required for them, and burn through them.

The whole point of this flag is to triage answers and pick out ones that can be quickly deleted. That's why we restrict it to the above categories, so we can make a quick decision on these and move on. It's a fast-track to deletion, and without it I'm afraid that flags will bog down.

Also, "not an answer" flags currently feed into community review within the Low Quality Posts review queue. Of the 24,000 flags we've gotten in the last month, 8300 of those were handled by the community. This would be removing all of those from community review, which I think is going in the opposite direction of how things should be headed.

I understand that there are disputes and disagreements over the scope of the "not an answer" flag, but I'd estimate that 90% or higher of them are good flags (Jon's stats show us accepting ~97% of all flags on answers). Tossing out the entire category because of a relatively low rate of confusion would seem to be counterproductive.

7
  • 1
    I would be fine with not having to think about the finer points of whether or not an answer is not a non-answer answer, and just focusing on accepting those flags that merit post deletion and declining those flags that don't. Just don't expect me to explain my decisions on meta on a daily basis.
    – Robert Harvey Mod
    Commented Mar 1, 2016 at 22:37
  • There is a very good post by Shog9 on MSE about this topic. If only posts on MSE were somehow relevant on MSO, or there was some possible design where MSO was the meta for the exchange to limit the type of demand you are seeing for explanation. Without a way to make high quality content like this : meta.stackexchange.com/questions/225370/… relevant on MSO then you have to continue to explain these types of requests on a daily basis because that is the outlook. Duplication is merely another facet of the same topic.
    – Travis J
    Commented Mar 1, 2016 at 23:39
  • 4
    The community could only handle ~35% of the flags? Considering we have a queue dedicated to that (LQP), and most of these flags should be easy decisions, that's rather disappointing. Commented Mar 2, 2016 at 8:17
  • @JeffreyBosboom - The problem is that the Low Quality Posts review queue doesn't just handle "not an answer" flags. It also deals with questions and answers flagged as "very low quality" and questions and answers that fail the quality heuristics. The review of questions in that queue is jamming things up. It takes a very different mindset to judge question quality and decide actions than it does to judge whether or not something is an answer. I've been thinking about proposing a split in review queues, with one being dedicated to just NAA flags or just low quality answers.
    – Brad Larson Mod
    Commented Mar 2, 2016 at 14:15
  • 1
    @BradLarson LQ queue doean't handle questions anymore, these go to Triage instead. Only questions in that queue are audits which weren't (yet) removed because SE team didn't do this yet
    – gnat
    Commented Mar 2, 2016 at 14:20
  • @gnat - Good point, I'd forgotten about that change. Looks like roughly half of the recent reviews are for "not an answer" flags, and the other half for "very low quality" flags on answers or answers that the quality heuristic has identified. The quality heuristic ones almost always seem to be wrong, and there's some incorrect use of "very low quality" for wrong answers, so that can slow things down. Still, I'm surprised that the processing rate in the queue hasn't improved significantly over when questions were in this. Wonder if removing the heuristic-identified posts would help.
    – Brad Larson Mod
    Commented Mar 2, 2016 at 16:09
  • 2
    I haven't been using LQ queue for quite a while but yeah, per my recollection heuristic items in it were much tougher to crack, typically very far from clear cut. It could possibly help if they would dynamically rebalance queue giving higher priority to user flagged answers - similar approach seems to work well in close queue
    – gnat
    Commented Mar 2, 2016 at 16:18
16

The Not An Answer flag is fine. What is not fine is that the flag is abused constantly by users that are splitting hairs about the answer being more of a comment.

If I were a moderator, the default action on a NAA flag would be to decline if I have to think more than 30 seconds on it. Everyone here knows what "Not An Answer" means; yet everyone likes to discuss endlessly if the apple is more of an orange or if the apple is too ripe, or not ripe enough. Let's end this non-sense once and for all:

Each day, one user could spend 100 flags, at least 3 times a day on obvious non-answers

This is a matter of priority, not of hair-splitting. We have to focus first on those non-answers:

  • Thanks!
  • Me too!
  • I'm trying your answer but it doesn't work.
  • How can I do this instead?

I'm not pulling out those those numbers out of my hat: I have been flagging 100 answers like those each day since at least 5 months. You can look at my number of helpful flags: it is sitting at 11,302 helpful flags right now, with a percentage of 99.61 helpful. And I'm telling you: those non-answers are coming out of everywhere and, sadly, I can't flag them all. Also sadly, this means that probably a lot are going through the net, unnoticed for weeks, months, maybe more.

This is the priority: the NAA flag is here to make Stack Overflow cleaner by removing posts that do not fit the Q/A format. We all want to participate in that. We all want Stack Overflow to be cleaner. So let's focus our effort on the big chunk of non-answers. And if we manage to do that, we might consider moving to the tricky ones.

There are a number of a dedicated users that I know of who also spend great time cleaning. We need more people that are actually willing to help clean instead of posting a Meta for every NAA declined.

So the answer to the question:

  • For mods: decline flags by default if a moderator needs to think more than 30 seconds. The point is to educate and, sadly, declined flags educate more than FAQs;
  • For users: focus on real clear obvious NAA that fits into the 4 categories above. There are more than you can imagine if you look around the corner. The question is: do you really want to take it that step further and help clean?
13
  • 6
    I'd like to add that there are a bunch of users below 10k (such as myself) who would gladly help moderate, but are lacking information. Personally, I would have no idea that there are so many clearly NAA posts out there, if I wouldn't speak to high-rep people regularly. I also happen to know that a lot of these posts are late answers to old questions, however the corresponding review queue (stackoverflow.com/review/late-answers) is almost always empty, giving the false impression that half-rotten orangapples are the worst thing happening. Commented Mar 2, 2016 at 0:00
  • 3
    My experience in Late Answers was that about half were NAA. I don't know how to easily find more; searching for "thank you" or "me too" shows me lots of noisy answers that comment on other posts but include enough code to be answers in themselves. Presumably you have a more precise SEDE query? Commented Mar 2, 2016 at 0:32
  • 2
    @JeffreyBosboom that's exactly what I'm talking about. The 10k mod tools privilege includes nifty things, including proper access to "new answers to old questions". Your not seeing any more is the <10k problem I noted, and if we could bridge the gap (i.e. let 3k<user<10k see more late answers), we could handle a lot more of these posts (and learn what NAA actually means). Commented Mar 2, 2016 at 0:47
  • 2
    I think the main problem is that the tools to efficiently find these posts (New answers to old questions for instance) are locked behind 10k for what seems to be motivation to get 10k reasons. I can't even really spend 20 flags a day, and most of these are on obvious spam reported by the Smoke Detector. I would love to help more, but alas, can't. So I think its worth discussing if these very helpful sorting tools really ought to be 10k perks, when it seems like they shouldn't be.
    – Magisch
    Commented Mar 2, 2016 at 7:14
  • 1
    I'm taking the downvotes as: "No, we just like to split hairs here! Who cares about the overall quality?"
    – Tunaki
    Commented Mar 2, 2016 at 10:42
  • The problem with the crux of your post is that "everyone here knows what it is" is not really defensible because it's not actually describing what it is (see obscenity under the Roth test in Jacobellis v. Ohio; "I know it when I see it"). What happens when someone doesn't know it when they see it? What happens when that person is a moderator? IMO we should revise the NAA verbiage to be more clear (either make it broader wording or clearly restrictive wording).
    – TylerH
    Commented Mar 2, 2016 at 17:23
  • @TylerH I honestly think everyone knows how to correctly classify a post into the 4 categories mentioned in my post. I'm not talking about rhetorical posts or "Did you ..." answer.
    – Tunaki
    Commented Mar 2, 2016 at 17:59
  • 1
    @Tunaki I would hope they'd recognize the 4 types mentioned in your answer but this feature-request would affect more than just those 4 types of non-answers.
    – TylerH
    Commented Mar 2, 2016 at 18:38
  • For users: focus on real clear obvious NAA - What's real clear and obvious to some is anything but real clear and obvious to others, as this question (especially the answers and comments) perfectly illustrates -> meta.stackoverflow.com/questions/268369/… Commented Mar 2, 2016 at 22:13
  • @JohnSlegers Please read my answer again and the specific type of answers I'm addressing. There are 4 categories, into none of which the link you gave me fit into. No nitpick.
    – Tunaki
    Commented Mar 2, 2016 at 22:14
  • @Tunaki : So when you're saying focus on real clear obvious NAA, you mean to say focus on answers that can be categorized as one of the 4 types of non-answers I mentioned earlier in this answer. That's actually a pretty unambigous and easy to understand rule to apply! Commented Mar 2, 2016 at 22:20
  • @JohnSlegers Not sure I understand what you mean but I did edit that yes.
    – Tunaki
    Commented Mar 2, 2016 at 22:21
  • @Tunaki : I'll make sure to check it out if I have some more time to spare. I actually should be working on a project right now, but I'm kinda procrastinating * blush * Commented Mar 2, 2016 at 22:50
2

An option worth considering is to drop that garbage that moderators need to be strict "for audits" (I'll explain what's wrong with it below).

Simply start handling NAA flags using your common sense, just like it was before SE team tricked moderators to believe in their nonsense about strictness. Dismiss flags as helpful and decline them as you feel right, do not feel restricted by formal rules.

You will still have some friction because people will always try to push the border. But there will be less of it and, which is probably most important, it will be much easier to handle meta disagreements.

Just think of it, if you honestly handled a flag using your own judgement (as opposed to narrow formal criteria), it should be easy to explain your reasoning - or accept that it was wrong (and learn how to do right) if meta discussion presents compelling challenge.


Back to this old rotten reasoning about strictness for audits. It's not needed really: audits based on flags handled by moderator are already as good as it gets - because these have passed human verification by a trusted reviewer.

If reviewer clicks "good" on an answer that was considered bad by a moderator, chances are pretty damn high that reviewer was wrong. Mistakes can happen, as usual, but these should be rare enough to be handled through usual meta dispute.


I really can't believe that we were tricked for so long to believe in that nonsense. For "known bad" audits, system pretends that it can't rely on a judgement of a moderator - experienced, elected, trusted user.

But for "known good" ones, it blindly follows a handful random upvotes from 15-rep passers by. Give me a break. Give me a f#cking break!

19
  • Are you saying moderators should mark as helpful and delete the answer, by default, unless they explicitely want to decline?
    – Tunaki
    Commented Mar 2, 2016 at 11:04
  • 3
    @Tunaki I am saying that they need to return back to use their judgement, as it was before these audits-shmaudits. If I happen to disagree with their delete or decline, I would much prefer this to be discussed based on their judgement, and not on artificial (and wrong) idea of how they should twist their brain for audits. Whoever would "lose" in such a dispute, they will learn something useful, as opposed to obscure useless details about audits like it happens now
    – gnat
    Commented Mar 2, 2016 at 11:11
  • So, let's stop twisting our brains as flaggers. Let's flag only what clearly needs to be flagged (and there are enough of that to feed a lot of people).
    – Tunaki
    Commented Mar 2, 2016 at 11:15
  • 3
    Not to mention if we can't even teach trusted users to flag correctly, we have no hope in ever teaching reviewers the subtleties of the rules. Common sense is always a better strategy.
    – Cody Gray Mod
    Commented Mar 2, 2016 at 11:17
  • 2
    @Tunaki there is a difference. Flaggers do weird things because they are mistaken. Moderators do weird things "because audits". I could still accept that "because audits" mind you if it was right... but it is wrong. Audits should follow moderator judgement and not the other way 'round.
    – gnat
    Commented Mar 2, 2016 at 12:45
  • 1
    Im still completly baffled as to why we started splitting hairs about what is and what isnt NAA to begin with. Almost exactly all of those posts are utter garbage, deserving of deletion in one way or another, so why do we spend time discerning what animal the turd came from instead of removing it?
    – Magisch
    Commented Mar 2, 2016 at 13:35
  • @Magisch prior to audits, it was indeed so. Flaggers and mods could disagree and discuss whether particular post qualifies as garbage but they shared common understanding that if it's garbage, it should go away. This was broken by useless strictness which forced mods to act like brainless robots whose only purpose is to feed yet another brainless robot (audits)
    – gnat
    Commented Mar 2, 2016 at 13:54
  • 2
    You're making the assumption that moderators didn't review "not an answer" flags for the criteria Robert stated before Tim's post. That's a bad assumption. We're not unthinking robots who do whatever SE tells us. I know I applied strict criteria for when something was or was not an answer. Deleting content that someone wrote in an attempt to answer a question should not be done lightly. I might have been a little more forgiving in what I marked as helpful, but I haven't changed what I do or do not delete. In fact, I even disagree with stated policy towards highly voted link-only answers.
    – Brad Larson Mod
    Commented Mar 2, 2016 at 14:25
  • @BradLarson I don't mind how strict your criteria are, as long as these are your. I can discuss and challenge these or use these to adjust my own understanding, that's complelely okay and that is how things should be. I only want that senseless rule about strictness for audits to get off our way of communicating, Fair enough?
    – gnat
    Commented Mar 2, 2016 at 14:29
  • 3
    Amusingly, the whole "be strict for audits" policy was only necessary because some moderators had taken an older "mark helpful if you do ANYTHING" guidance a bit too literally. So we had situations where, for example, mods were marking VLQ flags as "helpful" on old, upvoted resource-request questions and then lamenting that some folks were routinely VLQ-flagging everything they voted to close, ballooning the flag queue. The purpose of the "be strict" guidance was simply to free moderators to use their own judgement, so that we could rely on that judgement for audits.
    – Shog9
    Commented Mar 2, 2016 at 16:33
  • @Shog9 if you wanted to rely on their judgement, you'd feed ("known bad") audits with posts deleted by moderator because that's reliable. As for indiscriminate marking flags as helpful, this issue would better be resolved gradually, via meta discussion and guidance, and not with senseless linking of these to audits feed. Reliability of audits and the way how mods mark flags (as opposed to acting on them) are separate concerns and would better be handled separately
    – gnat
    Commented Mar 2, 2016 at 16:47
  • 1
    @Shog9 One can't VLQ-flag an upvoted question.
    – user3717023
    Commented Mar 2, 2016 at 17:01
  • Not anymore, @Sally. Old oversights created a lot of long-lasting problems. There's still one long-time user who emails us regularly complaining that he can't VLQ-flag any random post he doesn't happen to care for.
    – Shog9
    Commented Mar 2, 2016 at 17:06
  • 4
    Except it wasn't reliable, @gnat. That's what we started with. And all of this stuff was introduced over the course of about three years; unless "gradual" means "not within my lifetime" we pretty much did what you asked for. The problem has never been that moderators shrink from meta discussion, or refuse to adjust their behavior... The problem is usually that flaggers can't lead moderators around by their noses, and this irks them. IMHO, more moderators need to grow a spine and tell folks to suck it up when their flags are straight-up wrong instead of trying to let them save face.
    – Shog9
    Commented Mar 2, 2016 at 17:07
  • 2
    Moderators can delete answers for all sorts of reasons, @gnat. That doesn't necessarily make them good audits. Plagiarism is a particularly insidious problem; obviously you do want to remove these in some cases, but it can be really non-obvious that's what's happening, especially if the answer was copied from the same page. This still crops up from time to time with spam, btw. There are other edge-cases too - moderators are exception-handlers, after all. We added a lot of heuristics over time to try & combat this, mostly based on meta discussion.
    – Shog9
    Commented Mar 2, 2016 at 20:06
2

I'm not sure if retiring the “Not an Answer” flag is the best approach, but questions like Why was this 'not an answer' flag declined? illustrate perfectly that whether to flag a question as "Not an Answer" is anything but obvious in some cases. That question perfecly illustrates that in some cases it's 100% obvious to some that a user did not even come close at attempting to answer the question, while it's anything but obvious to others.

If not a retiring the “Not an Answer” flag, IMO an attempt should be made to at least make it less ambiguous when to apply the flag, and more specificitly, how to treat what might be considered "edge cases". For example, consider tunaki's suggestion to restrict the "Not an Answer" flag to one of the following four categories :

  1. Thanks!
  2. Me too!
  3. I'm trying your answer but it doesn't work.
  4. How can I do this instead?

These should be pretty unambigous to anyone!

2
  • Obviously, I agree :D
    – Tunaki
    Commented Mar 2, 2016 at 22:44
  • @Tunaki : Obviously! :-) Commented Mar 2, 2016 at 22:48
0

What if we made it a submenu? So when you click it, it opens up into

x Not an Answer
       o It's a Comment    |    o It's a new question   |    o It's an edit 

Now it required 2 clicks (which decreases click and forget, and increases though). The extra information can be just added as a comment to the flag.

Down the track, perhaps it could go to separate queues, perhaps not. But for now, just adding a step of difficulty should decrease use.

All three options are needed.

Its common for new users to post comments as answers -- they actually have no choice. Its also common for new questions to be posted as answers, (or as comments at higher rep), because new users are often cautious about "Creating new threads, while an existing one is still active", which is an (minor) offense in some forums.

I suspect its an edit is rarest. But it happened to one of my answers, and it was one of the more frustrating experience til I flagged and had it deleted. I had missed a bracket when copying a code snip from my computer into the answer. The user helpfully pointed that out -- copy pasting my answer and fixing it. It was particularly frustrating since they got more upvotes. Minor really (like a handful of rep), but frustrating all the same, like a pebble in the shoe.


The next level of this is for it to incorporate drop down lists.

  • [dropdownlist Posts] to select which post (is it the question or an answer)
  • [dropdownlist reasons] to specify a valid comment reason: eg Ask for claffication, pointing out error (if linked to answer) etc.
x Not an Answer
       o It's a Comment on:   [dropdownlist Posts] which is [Dropdownlist reasons]
       o It's an edit to:     [dropdownlist Posts] which is [Dropdownlist reasons]
       o It's a new question    

This is about providing a larger number of steps to ensure consideration is given. I think going to the level of dropdown lists is a bit far.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .