108

Some of the Stack Overflow moderators were having (yet another) discussion about what makes "not an answer" fundamentally different from "very low quality", since the former implies the latter in a significant number of cases. Since Stack Overflow can easily hover at close to a thousand pending flags during peak use, moderators are always looking for new ways to let people in the community do more.

The "very low quality" flag has always been ambiguously defined because (as Jeff Atwood once put it) "toxic waste" comes in a variety of shapes and sizes. "You know it when you see it" I used to say. "It's incomprehensible gibberish" others have said. As a moderator, I'd often wonder if some folks had fallen asleep and just rolled their head around their keyboard while servicing these flags.

The thing is, it's really difficult to get more than two people to agree on something being "very low quality" in all but the most obvious of cases, and obvious cases aren't really the typical cases. In fact, this used to be my process for handling most of them:

  1. See the flag, get the full context (parent question if applicable, other answers, comments, tags, etc.).
  2. Make an edit (or maybe not).
  3. Dismiss the flag. Often, if I could sort of see why someone would have flagged it, I'd consider it a good-faith attempt and validate the flag.

By way of comparison, it was rare that I'd take the prescribed action, which is delete with extreme prejudice if there was any substance to the flag. In most cases, I just didn't know for sure, because while I was a pretty good objective judge of quality, I had little to no domain knowledge.

That got me thinking...

Why did we make this a moderator function, anyway?

Could gold and silver badge holders vested in these tags be handling these better than diamond moderators? Probably, because:

  • They know the system, and how it works. They know that Incorrect or sub optimal != Low Quality. They're in a better position to decline these flags as an attempt to use a moderator as a proxy to delete stuff that just mildly annoys someone for some reason.

  • They know the topic and can identify when something is really coming out of some strange place in the cosmos, and has no relevance to the question at all.

  • They know when a question is just an unanswerable mess that can't possibly be salvaged and might be able to help people avoid pile-on downvotes by quickly guiding users with something more helpful than "maybe try again".

What are the drawbacks of, say, opening at the silver tag badge level, allowing users to validate (somehow) or dismiss those flags? Should we open up a review queue just for this? How might that work? What should happen if two people with a silver badge agree on something being low quality and how do we surface it to them?

As we look through places where all of us would agree that users didn't get treated very well, there's this certain subset of questions where—well—there's just nothing nice or encouraging anyone could say about them. These questions turn out to bring out the worst in people and just make lots of ugly. Moderators hate stepping in when you really need domain knowledge to make a call, yet they hate seeing folks have this experience, and we've been debating how or if this flag should even exist for years.

How can we totally offload VLQ flags safely? What did we learn from the dupe-hammer initiative that applies here? Should we set up a review queue, revamp the helper queue to service this, or...?

Maybe just handle it from... something else? We need some input.

This is currently just blue-sky thinking.

But Shog and I always meant to do something with silver tag badges (in fact, I think he's typing his ideas on an implementation right now), but we wanted to get your ideas right now, as we do this, to see if there's any merit here in going further and writing an official specification.

Again, the goals (distilled):

  1. Let people with proven knowledge handle VLQ flags without moderator help. Part of this discussion is to define 'handle'. As they do that, let's decide if it's of any use, and how it could be better.
  2. Make sure any implementation improves the user's experience, even if it means deleting something quickly with some guidance (think outside of the box all you want here).
  3. Make sure there's a feedback loop so flaggers can hone their skills.

Anyone want to chew on it more? We're open to trying ideas and even getting a little nuts, but we need way more input than just Tim and Shog thinking about it in a back room. So, instead of getting an internal specification together to discuss externally, we're going to get an external spec together to discuss internally.

How could this work? What do you think? Is a cheese pizza with Canadian bacon really just very low pineapple? Let us know. We hope we can get enough ideas and concerns together to get a formal specification that we might be able to put into testing, but let's just start with the ideas part.

24
  • 45
    One problem I immediately see is that I frequent a relatively niche tag, and it doesn't have that many active silver badge holders. If we exclude those that aren't active reviewers, that leaves... well... me, afaik (maybe + 1 or 2 that sometimes review). That means if something requires more than a single vote, it will likely not get out of review.
    – Erik A
    Commented May 31, 2018 at 17:01
  • 3
    Will there be a difference between a gold badge holder and a silver badge holder? E.g. something along the lines of a gold badger having a double-counting vote. As for the cheese with Canadian bacon pizza: of course it's not low pineapple, as fruit doesn't belong on your pizza in the first place!
    – Adriaan
    Commented May 31, 2018 at 17:08
  • 31
    While given the current definition of the flag, there is no need for tag-specific knowledge to handle it. So the very idea that you're suggesting people familiar with the subject matter validate the flags implies that you're changing the entire meaning of the flag to be an evaluation of the technical merits of the post. Up until now that's been a matter for votes, not flags/deletion. If that's not your intention, I expect lots of people will interpret it as such, if it is, what's the reason behind the change?
    – Servy
    Commented May 31, 2018 at 17:13
  • 7
    @Adriaan I only eat pizza if it has pineapple on it, :D
    – user3956566
    Commented May 31, 2018 at 17:58
  • 5
    That's one of those notions that gets tossed out a lot, @Servy... Strictly-speaking, it's true - but it ignores the cost of evaluating answers in areas where you're unfamiliar even when the flag is invalid. An extreme example of this are chat flags raised in non-English chatrooms: sure, you can usually manage to figure out if the message is blatantly offensive, but it takes a LOT of extra time, and thus misguided flags tend to chew up a lot more energy than they normally would.
    – Shog9
    Commented May 31, 2018 at 18:01
  • 2
    @Shog9 Sure, I could imagine that SO is proposing this change because it thinks that people familiar with the subject matter will be able to more quickly realize if a given post is a sincere attempt at answering (even if it's wrong or otherwise having quality problems) or if it's just rubbish. My question is if that really is what this is about, or if the actual proposed change is to have VLQ flags and people handling them be judging the post on their technical merits (possibly in addition to what's currently flaggable). Reading the question it was unclear to me.
    – Servy
    Commented May 31, 2018 at 18:04
  • 6
    What do you think should happen, @Servy? The problem we have to solve is that the site has a lot of crap on it, and despite our best efforts mods and flags and queues all impose a lot of overhead right now while the earnable "moderation privileges" are kinda toothless in the face of it all. So folks get frustrated. This isn't new, and in a lot of ways we've made it better... But it's not good enough. That's the big picture; as always, the devil is in the details. What actually has a chance at substantive improvement here? Write an answer if you have thoughts...
    – Shog9
    Commented May 31, 2018 at 18:10
  • 4
    This is a discussion, not a proposal, @Servy. It's a "what-if?"
    – Shog9
    Commented May 31, 2018 at 18:17
  • 4
    @Shog9 The post has outlined a possible change and is asking for feedback on that change. The actual change described wasn't entirely clear for me, the "what-if" being asked about seems to have a few different interpretations. I was hoping for some clarification. I'm really confused as to why you're opposed to the question being clarified as to whether it's asking about changing what should be flagged or not.
    – Servy
    Commented May 31, 2018 at 18:23
  • 7
    Because first off, you have a narrow view of how VLQ flags are used now - which is understandable, but unhelpful: a tremendous volume of these flags already go through one or more review queues and are dispatched on criteria much different from those the moderators generally apply. But more than that... The flag exists to serve a need, not the other way around. What it means and how it's handled is malleable if the need requires them to be... In practice, moderators are not robots; they apply their experience and judgement. But experience and judgement isn't limited to moderators.
    – Shog9
    Commented May 31, 2018 at 18:30
  • 4
    We really need a lot more UI around tag-centric discussion and moderation. Kind of like how that stuff worked on Docs.SO
    – user4639281
    Commented May 31, 2018 at 19:42
  • 5
    more power to silver badge owners? who's against that? great idea! soon I'll be able to power-handle questions about [list] (or whatever that means) Commented May 31, 2018 at 19:44
  • 13
    Whatever you end up deciding, please make the definition of the flags obvious, or at least have a set of guidelines somewhere other than a useful infographic hidden deep in meta. Any kind of flag system is useless if the lack of instructions drowns the system in noise. Help us help. Commented May 31, 2018 at 20:08
  • 3
    @AndrasDeak and my point is that even users that doesn't know it get it mostly right, while the ones that know it choose to disregard it when isn't aligned with their own views.
    – Braiam
    Commented Jun 1, 2018 at 0:23
  • 3
    "They know that Incorrect or sub optimal != Low Quality." I'm skeptical of this - I see so many of them being the ones behind VLQ flags on incorrect or suboptimal answers in the first place.
    – BoltClock Mod
    Commented Jun 1, 2018 at 3:00

12 Answers 12

155

Four months ago, I promised a follow-up answer with a concrete proposal.

I've been mulling it over since then, trying to decide on how to phrase this answer. Not because I didn't know what I wanted to propose... But because I didn't want it to be a modest proposal. And to that end, I'm gonna have to spend some more time laying out my rationale first...

The future of Stack Overflow

For a very long time now, Stack Overflow has been headed in two different directions...

  1. The first direction involves building an increasingly-detailed set of Q&A on broad, popular topics. This is the easiest one to observe: if you're learning, say, C# then you'll find a very broad and deep set of information here, with more and more corner-cases being filled in hourly. The single biggest obstacle here is noise: with almost every possible topic covered in multiple questions expressed in multiple ways from multiple perspectives, finding a question that seems to match the problem you're having is easy - but finding the question that has an applicable answer can be a slog.

  2. The second direction involves covering an ever-more-vast set of topics. Everything from languages and platforms with only a few (initial) users, to specialized libraries and tools nominally under the umbrella of a more well-known language or platform. The big problem here is (and has always been) that these topics are nearly impossible to moderate effectively; they may have only a tiny handful of active users, they tend to not attract much voting, and the bulk of the moderation tooling (and elected moderator team) is skewed toward serving #1.

Solving the problems of #1 is... An almost insurmountable challenge. We tried hard to attack it head on, and eventually came to the conclusion that we could probably throw a billion dollars in dev time at it and still only maybe succeed; the root causes are simply a lot bigger and broader than the little Q&A site they affect. That doesn't mean we shouldn't try to mitigate those effects, but it very much means we should do so while compulsively reciting the serenity prayer.

Because of this... Or perhaps in spite of it... I've come to believe that direction #2 is the future of Stack Overflow, the area that both askers and answerers will find most rewarding in years to come. Specialization may not appeal to Lazarus Long, but as our field matures it's an increasingly-necessary option - and one we have a real opportunity to serve better here on Stack Overflow. But in order to do that, we have to solve the problem of moderation in those tags. And that's where this all ties back to your question here...

What have we tried?

We've tried... a few things...

  • For the first five months on Stack Overflow, anyone who could scrape together 3000 points could close any question - and reopen any closed question. That was a pretty high bar for who could moderate at the time, but made it very easy to moderate for those who earned the privilege - and put no bounds on what could be moderated.
  • Starting in 2009, closing became a vote-based system... And almost immediately, it became apparent that stuff in less-popular tags just didn't get closed. The immediate solution was to stop aging votes on questions that weren't getting many views... This didn't actually solve the problem, but it made it feel solvable in some vague way.
  • Soon afterwards, the VLQ flag was introduced... Mostly because folks kept typing "this is very low quality" into the "other" field on the flag dialog. Jeff started just deleting questions flagged as such. This actually did help for a little while, possibly at the cost of Jeff's sanity.
  • A couple of years later, we introduced /review for pending close votes, with the ability to filter by tags. Again, this didn't really help much, but made a solution look even closer albeit still just out of reach. We optimized the hell out of this tool, achieving at best limited success... And at the same time increasing the instances of questions being moderated by people who had no knowledge of the topic being moderated whatsoever.
  • A couple years after that, we introduced Mjölnir, which helped a LOT... for tags that had at least 200 questions, all answered by the same tireless person.

The goal of moderation tooling on Stack Overflow has always been to create a force multiplier, allowing a relatively small group to accomplish a much bigger task than they would otherwise be able to. But when it comes to these specialized tags, that "group" might consist of only one or two people; in some tags, only the author of the eponymous library has any interest in monitoring the tag at all...

And with the exception of Mjölnir and the original long-forgotten closing system, nothing we've ever done has really worked for tags with only a single knowledgeable user monitoring them.

That welcoming thing, again

There's another way in which much of our tooling has failed us when it comes to niche tags: it has too often created an inherently combative or judgmental relationship between contributors and maintainers. No matter how nicely-phrased our close reasons or canned comments may be, the subtext is always the same: you have been weighed and found wanting.

And each time, our solution to this problem - going all the way back to 2008 and the introduction of close voting - has always been the same: make the tools harder to use. Require more people, more votes, more effort for them to become effective. Frankly, this hurts everyone - but none so much as the already-beleaguered maintainer of the niche tag, already over-extended and under-appreciated.

It's time to put an end to that strategy. If quality and kindness are truly "a false dichotomy", then our tools should reflect this: they should be effective at encouraging quality while at the same time expressing kindness.

Summary of requirements

For the reasons I've outlined above, I believe that our goals for a new tool to handle low-quality should be as follows:

  • Possible to be used effectively by a single trusted member of the site
  • Responsible use should benefit both those who are invested in monitoring a given topic and those who seek to learn about the topic by asking questions.
  • Immediately effective in getting problematic questions out of view
  • Difficult to abuse

The proposal: ending a false dichotomy

A'ight, let's cut to the chase... Here's what I'm proposing to satisfy all of those requirements:

  1. If a question has no non-deleted answers,
  2. And the question has a tag matching one of the viewer's tag badges (bronze, silver or gold) and that tag was not added by the viewer themselves,
  3. And the viewer has not previously deleted or voted to delete the question...
  4. Then show the viewer this option:

    delete and request edits

    • If chosen, the viewer will be prompted to write a request for improvement from the author:

    a prompt for input: "What can the author do to make this question suitable?" and a button: "Comment and delete"

    • Upon submission, the question will be immediately deleted, the viewer's input will be rendered as a comment, and the author of the question will be notified via their inbox.
  5. When the author visits their question following such a deletion, it will be shown as follows:

    Instead of the status being "deleted", the question is shown as "awaiting edits from the author as of Jan 28 '15 at 20:11 please see the comment from Shog9♦ below for information on how you can improve this question"

    • If the author edits it, the question is automatically and immediately undeleted.
  6. Badge-holders will only be allowed to delete any given question once.

  7. If three separate badge-holders each delete the same question (it having been undeleted twice following the first two deletions), the question will not appear to the author as awaiting edits; it will show up as deleted and edits will not immediately undelete. We may wish to raise a flag or put the post into a queue in this scenario.
  8. Badge-holders will be allowed a limited number of deletions per day, based on the rules that currently apply to 10K users.
  9. Neither the badge-holder nor the asker will be allowed to comment on the question until it has been undeleted. The asker may raise flags on both the question and comment(s) if need-be.
  10. Moderators would also be provided this "delete and request edits" option on all questions, in addition to the normal "delete" option.
  11. A report should be provided for moderators and 10K users to allow oversight of the use of this feature (site-wide and per-tag).

Answers to anticipated questions

  • What does this have to do with offloading VLQ flags?

    Based on some scans I did of the access logs, badge-holders already visit quite a few VLQ-flagged questions without needing any extra encouragement to do so. And... Also are the ones raising VLQ flags in some non-trivial number of cases. This would hopefully replace the latter with a considerably more useful action, and deletion would automatically handle any pending flags as well.

  • Wait, so your solution to handling VLQ flags is to replace them with something better?

    For questions, yeah. They've never really worked very well; every alternative just worked even less well.

  • Why deletion? Why not closing? Didn't you mention something about closing in your last answer?

    Closing has a lot of baggage, both as a concept and in terms of implementation. There are a lot of unfortunate assumptions regarding what it does and how it should be used that don't really satisfy the requirements I outlined above. In particular, closing doesn't immediately get rid of the question - if your goal is to gently, kindly, get rid of a really ugly question so that the author has a chance to fix it... Then closing it doesn't do that: it keeps it around, visible, for a few days while folks mostly jeer at it and throw rotten produce.

  • Won't folks just make pointless edits and go back to junking up the site? What gives you the idea this will even work?

    We've been doing something very similar for low-quality answers for quite a few years now; we don't even require edits - the author can just undelete at will. It rarely happens, but when it does a decent number of the answers actually were improved.

    But more than that... We've been practically begging folks with closed questions to edit them for ages, and still most do not edit. The problem there is, when they do edit it takes a lot of work to get the post reopened; by the time all is said and done, it's taken 10 votes to go through that close-edit-reopen cycle. This proposal requires one person besides the asker to be involved in the common case; worst-case, three people.

  • What are your favorite colors?

    Green and yellow purple.

Conclusion

I realize that this... probably sounds a bit drastic. That's what has kept me from posting it for months now. But I keep running into situations where folks are trying to do the right thing: ask questions, encourage new askers, maintain their topics... And having to fight the system every step of the way. We need to break out of this drama triangle; after an awful lot of soul-searching, I believe this is the simplest thing that could possibly work.

63
  • 13
    It is drastic. But it is time to consider. Lessening the blow a bit, maybe rewind a bit to the days y'all thought about how to make the Triage queue work? But it never did? "Withdraw from views" is what actually counts to SO users. "Awaiting edits" was H&I. Germs of a good idea, just never actually implemented. Commented Sep 24, 2018 at 22:29
  • 14
    Yeah... Triage was predicated on us rewriting all of the views. Which... Very nearly happened. And then didn't. That's why I like building on deletion: it works out of the gate, no "this will work when we finish building it if we ever do".
    – Shog9
    Commented Sep 24, 2018 at 22:46
  • 11
    The most obvious failure mode for this is a plague of stupid deletion reasons. We already have a taste of the problem today - one of my hobby horses on Meta is commenters arguing that "How to do X" questions are inherently off-topic and demanding that every such question be converted into a debugging question by exhibiting a failed attempt. Allow questions to be unilaterally deleted with a comment and all of us are going to see a lot of questions deleted for reasons that we strongly oppose. Is that a good tradeoff for clearing away the garbage faster? I'm inclined to think not.
    – Mark Amery
    Commented Sep 24, 2018 at 22:52
  • 7
    Closing has always been most useful for posts that are already doing well... but probably shouldn't for whatever reason. I think we'd all be better off if it went back to being used for that.
    – Shog9
    Commented Sep 25, 2018 at 4:26
  • 16
    I'd say the bronze tag-badge is a bit much for being allowed to single handed delete a question, since bronze badges are more often than not 'collateral' so to speak. If a person focuses their efforts on, say, c++, they might very well get a string and array badge along, but that doesn't mean they know stuff about arrays in Python. Having a silver badge in a secondary tag already happens a lot less, and those that do usually know better how to work the site, also outside of their own main tag focus.
    – Adriaan
    Commented Sep 25, 2018 at 7:52
  • 10
    Or, to put it differently: if I would use my string bronze on a Python question, then that's because I actually am confident I'm doing the right thing.
    – Baum mit Augen Mod
    Commented Sep 25, 2018 at 9:33
  • 28
    I'm very very happy to see what may be the first hard evidence of the company acknowledging that kindness and quality control are different things. Let's hope the suggestion works out, but it can only be more productive than yet another CSS change ;) Commented Sep 25, 2018 at 9:47
  • 6
    Looking through those requirements it sounds pretty much like a delete-hammer with a custom comment. Stuff like this (more power to badge holders) has been asked repeatedly. I'd love it to happen but I can't see why this particular request would gain traction when for ~4 years it hasn't been possible to get tag badge holders anything more than dupehammer (and that for gold alone).
    – jpp
    Commented Sep 25, 2018 at 11:51
  • 9
    @jpp strong point of this proposal is that it abstains of innovating. As far as I can tell, Shog simply tries to somehow... "migrate" approach that has been proven to work extremely well on VLQ answers. As for angst, I don't expect it to become quite widespread if you take into account that undelete is just one edit and click away. It is also important that the moderating action is performed by a subject matter expert (tag badge holder) who is apparently more incentivized to answer than delete unless the question is really poor
    – gnat
    Commented Sep 25, 2018 at 12:24
  • 14
    It's Shog's idea, not "team's idea". I would not hold my breath waiting for it to be implemented.
    – user6655984
    Commented Sep 25, 2018 at 13:28
  • 21
    Just to be clear, @Yvette - Welcome is quite right: this is an idea I came up with after several conversations (internally and externally), but it is not anywhere even close to a done deal; there are a bunch of people within the company who'll need a lot of convincing before anything like this stands a chance of getting off the ground. Like every other meta discussion, this is intended to be fodder for the dev team, not a mandate.
    – Shog9
    Commented Sep 25, 2018 at 13:54
  • 11
    @Shog9 you've ruined my day. Why the heck does it have to be sooo hard? I'm tired. Time to have a break for a pout. exit stage left
    – user3956566
    Commented Sep 25, 2018 at 13:57
  • 11
    Awful lot of posts should really stay deleted, @Raedwald.
    – Shog9
    Commented Sep 25, 2018 at 20:49
  • 5
    (on a further thinking) I would recommend making positive score questions ineligible for proposed feature. First, because it would save us meta drama ("how dare they delete my question at +5") and second, because this will introduce balancing power to lower rep users, so that anyone with 15 rep will be able to save the question by upvoting it (being passionate deletionist I generally hate stuff like that but experience of other moderating actions suggests that such balance is nevertheless useful)
    – gnat
    Commented Sep 26, 2018 at 6:40
  • 10
    I like this, and beyond just the concept. This essentially narrates my workflow for VLQ flags in tags where I have limited-to-no breadth of expertise (could anyone answer it? Did they? Did that get upvoted? It's not VLQ!). My concern is what people are going to write in that box and the fact that eyeballs won't see it, but we've got some AI cooking that could alert us to unnecessary rants there. Very very interesting.
    – user50049
    Commented Sep 26, 2018 at 12:36
54

I have a lot of thoughts on this... I may add more later, but to start with I want to lay out why I think this has merit. I'll apologize in advance; this is a bit stream-of-consciousness.

VLQ is for questions

For now, I'm only going to talk about VLQ as applied to questions. It's also a useful flag for answers, but... Honestly, we could probably get rid of it without too much pain; not only is Not an Answer a much more common flag, 44% of all answers flagged VLQ also get at least one NAA flag. We could, as Yvette suggests here and as many people have suggested in the past, just combine the two flags.

OTOH... There's not a lot to gain there either, we'd almost certainly see an increase in "in need of moderator attention" flags (which MUST be handled by mods) and there's a LOT of plumbing that'd have to be adjusted.

Questions are another story... Your other options are Spam, Rude/Abusive, In Need of Moderator Intervention... and a big ol' pile of close flags. And there is no Very Low Quality close flag.

There are no bad questions

Every question posted represents a sincere desire for help by a struggling programmer who deserves our patient attention and guidance.

...what? You disagree with that? You heartless monster!

OK, fine, we all know that's not... strictly... true. There are a LOT of questions that aren't very good, and a noticeable amount that are, no matter how sincere, beyond our help. And... The latter are particularly demotivating, in the same way that spam is demotivating: it makes the place look trashy.

This was the original purpose of the VLQ flag: treat exceptionally poor questions like spam. Get rid of them ASAP.

But... we kinda lost track of that.

The problem: me

I think we need to be able to get rid of very poor questions as quickly as we do spam. The volume is comparable: about a thousand questions a month, give or take. That's not the volume of VLQ flags; that's the count of VLQ-flagged questions that are deleted relatively promptly.

I say "relatively" because the average time to deletion is a bit over 3 days, compared to just over 2 hours for question-spam, 45 minutes for answer-spam, and about 5 hours for VLQ answers. The median time to deletion is about 11 hours, compared to 1 minute for spam.

That's an awful long time for stuff that's so bad as to be offputting. And... That's my fault:

A too-localized, vague or overly-broad programming question? Meh. Just close it and move on.

Yeah... 2013 me was kinda naive. Well, even more naive. Don't get me wrong; I'm still pretty happy with how automated deletion has panned out - but the idea that there is no cost to closing, or leaving egregiously-bad (but sincere!) questions laying around... was, uh, stupid.

To be fair, I had it in my head back then to build some sort of 10K-only deletion review for this stuff. But, that never panned out. Nor did any real equivalent. So here we are, five years later, and the full extent of the tooling for quickly removing stuff like this...

How to make an Login system

...is the downvote button and the waiting game and maybe, if you're very, very lucky, an appearance on the 10K-only "delete votes" page.

Yeah. I dropped the ball. Sorry.

A lesser but still important problem

Now... We've had our disagreements about what constitutes spam. But that's nothing compared to how much we've argued over how terrible something must be to qualify as "very" low quality. Bad grammar? Poor formatting? Unclear to the point where even tagging is a guessing-game? Straight-up gibberish?

But here again, spam gets a better deal: about 73% of spam on Stack Overflow is deleted by ordinary users flagging it (in under 5 minutes). For most spam, mods don't even have to be aware of it, much less agonizing over whether it truly pushes the limits of what is allowable; if they even see a spam flag, it's probably borderline enough to warrant a bit of extra time and analysis.

Six flags from anyone with 15 rep or above gets rid of spam instantly. It takes 5 votes from 3K+ rep users to even close an extremely poor question, and then either 9 days for automatic deletion or another 3 votes from 20K users to delete it faster. To say nothing of the time spent from folks flagging, downvoting, skipping past it in review, commenting...

...oh yeah, commenting.

That third problem: the welcoming thing

Naive 2013 me apparently had this weird idea that even really terrible questions might get a helping hand from someone and, grateful for the advice, redemption for the asker. That's... always a possibility I guess...

advice makes the asker sad

...so is that. While certainly well-intentioned, this isn't good for anyone: the asker didn't get what they wanted, the would-be good Samaritan got a slap in the face for their troubles, and nobody learned anything except maybe "don't try to help". More fodder for the "Stack Overflow is elitist and unwelcoming" crew.

What I'd like to try as a solution

I really like Tim's idea of putting more tools in the hands of folks with demonstrable experience. Not necessarily because that's necessary to evaluate these questions, but because that demonstrates something of a vested interest in the topic. I don't care as strongly about accidentally deleting a useful PHP question as I do about, say, C++ or JavaScript; I don't use PHP. That's not to say I'd be intentionally careless... But there's just not as much intrinsic motivation to be careful.

As for what those tools should be... I think there are three valid outcomes for a legit VLQ flag:

  1. Editing
  2. Closing
  3. Deletion

...and I think each of those should be just slightly harder to impose than the previous. If I can unilaterally edit, then it should take two people to close and three to delete. Not close and then delete; just straight-up delete. If three people with demonstrated experience in a tag agree that a question is an eyesore with no chance of redemption... Just get rid of it ASAP.

That devil in the details

That still leaves a lot of open questions: what should "demonstrated experience" mean? How should flags get exposed? Should we finally pick a better name for this flag than "Very Low Quality"?

But, it's supper time. Sorry for this rambling response; I wanted to get some context hammered out while it was fresh in my mind after pondering this idea... More specific ideas tomorrow.

14
  • 20
    "The problem: me", I hope you don't mind if I start quoting this bit in all sorts of contexts... ;) Commented May 31, 2018 at 23:54
  • @AndrasDeak We were just given the perfect weapon for when Shog slap [status-declined] in our favorite FR.
    – Braiam
    Commented May 31, 2018 at 23:55
  • Great master plan to solve multiple issues with one blow. Removing crap, the close vote queue, rudeness in comments. I'd be curious to see under what post types the new user is most likely to be rude.
    – user3956566
    Commented Jun 1, 2018 at 2:47
  • 5
    You mean deleting all the PHP questions isn't taking care of the site? Goes back to dictionary on 'quality'
    – Adriaan
    Commented Jun 1, 2018 at 6:40
  • 2
    On editing VLQs: I thought the VLQ flag was meant for posts which are not salvageable for other people than the OP; like adding their own code etc. So if we, us badge holders that is, can edit a post to not be VLQ, was it VLQ in the first place?
    – Adriaan
    Commented Jun 1, 2018 at 6:43
  • 1
    I think we need to be able to get rid of very poor questions as quickly as we do spam That sounds like a nice goal, but one that's hard to achieve, especially when you're only allowing silver-badges to vote, on low-traffic tags. And exceptionally poor questions are usually tagged poorly too. The one you've cited might be tagged something like login forms access single-user webpage, allowing no-one to review it (except the 9 forms badge holders), unless we edit it first for the purpose of being able to review it.
    – Erik A
    Commented Jun 1, 2018 at 8:15
  • 4
    Your part about "another 3 votes from 20K users to delete it faster" is inaccurate: before 20k user can even vote the question needs to be at score -3 (which you may argue that the three of them can do it, but they would need coordination, and some people aren't comfortable with that) but 10k can delete vote in 2 days despite the score or answer count. So, the one sure way to delete a question faster than roomba is to wait 2 days unless score is at -3 and 20kers see the post.
    – Braiam
    Commented Jun 1, 2018 at 11:51
  • 1
    This sounds awesome and I can't wait to test it out. Two suggestions: 1) add a score qualifier as a safe guard, e.g. it won't work on questions that are at a score of 2+ or something. 2) Add a date qualifier - if the question managed to survive undeleted or unclosed for some time, like 6 months or more, then it probably deserves to go through the 'long-form' path to deletion as a sanity check/safeguard.
    – TylerH
    Commented Jun 1, 2018 at 14:27
  • 3
    As for demonstrated experience, here are two suggestions from two different perspectives. 1) gold tag badge and 10k reputation to start with; that demonstrates subject matter expertise and makes sure you have access to deletion tools already so you are aware of how that privilege works. 2) silver tag badge, 3k reputation, a marshal badge, and a steward badge; this demonstrates you have some vested expertise in a tag and demonstrates that you have a vested interest in moderating the content via the steward and marshal badges (and are hopefully aware of how question closure works)
    – TylerH
    Commented Jun 1, 2018 at 14:31
  • 5
    Yeah... 2013 me was kinda naive But that was Shog3. We've come so far since then, right?
    – Machavity Mod
    Commented Jun 1, 2018 at 14:40
  • "the welcoming thing" is probably one of the most important problems actually Commented Jun 2, 2018 at 5:47
  • 4
    If you want to treat VLQ the same as spam, then just make the flags work the same. Something fancier would be good too, but even simply not limiting its raise-ability so badly would go a long way.
    – jscs
    Commented Jun 2, 2018 at 15:32
  • 1
    @JoshCaswell I agree they should work in a similar way, without penalising the user's account though. It would make the community less frustrated at the difficulties in removing poor content and the fear the site is littered with garbage.
    – user3956566
    Commented Jun 3, 2018 at 18:19
  • > "and the fear the site is littered with garbage" — as if this isn't already happening at a major scale. Commented Jun 7, 2018 at 13:46
33

Start with an edge case.

One problem with the low quality flag on answers is it throws those posts into both the review queue and the moderator queue, and if the person has flagged it, as it's giving "very bad" or "wrong" advice that then leaves it to people who may not be experts in that domain to decide if this is true.

From this viewpoint, it would be a great idea for silver and gold badges to review these types of posts.

It does however raise a difficult point. Aren't moderators supposed to decline flags on answers that basically disagree with the content as:

Enter image description here

Should these types of answers be downvoted?

In all honesty, the criteria for raising this type of flag are unclear and your question has demonstrated how it is not clear, as you're now saying it takes experts to review those flags. The implication being, the answers have a "decent" level of content - as least superficially.

So this brings back to the point of:

Can we combine the two flags (NAA and low quality) into "Should be deleted"? If a user is especially concerned they could add comment, for reviewers to see, why it should be deleted.

If you then choose to add a search option on the low quality review queue to filter by tags, maybe that would be a simpler solution.

Or combine those and then create one for content disputes, which is probably what a lot of people would prefer.

It's a little confusing, a foot in each camp. Either you want poor quality answers deleted from the site or you want them downvoted. Judging the line between what makes an answer poor quality and an answer can become arbitrary and posts like You're doing it wrong: A plea for sanity in the Low Quality Posts queue make it all the more confusing, in that we are supposed to leave this content on the site.

The community has been asking for this for a long time now, and it would be good if we could finally address it.

For example:

Enter image description here

Enter image description here

7
  • 10
    The idea of a "should be deleted" flag is perpetually popular... with people who can already just vote to delete things. We may very well need to change the flag name / description, but... Put yourself in the shoes of someone who is new here, looking for questions to answer, and is dismayed to find that the question they're looking at is just terrible. They've never read meta, don't know anything about our moderation system, about what flagging does... What description captures the problem that they're itching to report?
    – Shog9
    Commented May 31, 2018 at 18:06
  • 6
    @Shog9 ”This question should be removed from the site“. “Low quality” doesn't exactly say that, but the fine print under it includes “might need to be removed” (assuming that the VLQ flag is available at all on the question) (and for an answer there's ”deleted altogether“). Having “removed” or “deleted” as the most prominent part of the description, and jargon such as “low quality” relegated to the small-print explanation, would be an improvement. Commented May 31, 2018 at 19:39
  • 5
    @Shog9 And for an answer, it's even worse. Moderators can't agree on when to use VLQ, when to use NAA and when to use custom flags (even if you restrict to cases where they all agree that the answer should be deleted). So how would you expect this new site user to figure those out? Commented May 31, 2018 at 19:40
  • 5
    @Shog9 FWIW I have never been able to delete answers, but I would love to have a flag that is just a blanket "should be deleted" flag for any reason, rather than having to split my flagging for "should be deleted" content into one of two different flags, and woe unto me if I pick the wrong one. The reason people flag as either VLQ or NAA (I assume) is always so that the content will get deleted... so what's the benefit of having two different reasons?
    – TylerH
    Commented May 31, 2018 at 20:03
  • @Gilles "So how would you expect this new site user to figure those out?" Trial by fire seems to be the rave these days, but more to the point, how about asking the would be reviewers what is their take in the particular subject?
    – Braiam
    Commented Jun 1, 2018 at 0:02
  • I think we should test that, @Gilles. For questions at least. I'd be really interested to see how folks react.
    – Shog9
    Commented Jun 1, 2018 at 2:50
  • @Shog9 I just posted about a complementary idea regarding dealing with those inaccurate answers, so you may be interested: meta.stackoverflow.com/questions/370708/…
    – Cœur
    Commented Jul 9, 2018 at 9:33
14

I think this idea has a lot of good potential. Here are some thoughts that come to mind when considering implementing this:

  1. This should not be taken completely away from moderators.

    Like Erik commented above, this could easily be a problem with VLQ flags on questions with low participation languish for days. Let the community start handling these flags, but if they expire, send them up to the moderator queue.

  2. I'm not sure this should be opened to silver badge holders. At least not to the same degree it is opened up to gold badge holders.

    If we are talking about the community being able to delete content on their own, that means a couple people with 2k reputation could start causing answers to be deleted through their own action... something that normally isn't possible until 20,000 reputation, which is an order of magnitude more reputation and 'trust' afforded by the system. There are a lot of silver badge holders who know enough about a language or tag to be dangerous, but can still be wrong a lot of the time. I am only a few points away from a gold badge in and I am definitely still wrong about CSS questions or answers sometimes, either by thinking the wrong thing is true or by misreading what someone wrote in a question or answer.

  3. On that note, why not a hybrid set of requirements? A gold badge and, say, 10,000 reputation (just an example) are both needed to use this queue. Or a silver badge and 15,000 reputation. This would ensure both some significant level of trust from the community that they know what good content is (because they produce it) and they have some subject-matter expertise in a tag.

  4. Servy makes a good point with his comment; we are told a lot that VLQ is for indecipherable garbage or something that would happen if a cat decided to participate in a programming contest. If we are going to start prescribing more power to people who are SMEs in a given tag, what does that mean for the definition of VLQ? Are we now supposed to flag wrong answers as VLQ? Or were we always? I thought we were only supposed to downvote wrong answers (and delete-vote them if we have 20k reputation).

7
  • maybe single deletion with silver gives too much power, yes. There are enough gold badges around already. Commented May 31, 2018 at 19:49
  • 5
    They don't even have to "expire"... right now the mods already have a 1-hour delay... extend that to 1-2 days instead (or whatever the average flag handling time might be) and after that time, kick it to the mods. As for point 4, this is about review, not flagging, so there is no "VLQ" in review... not in the same way. The review is "Low quality posts" and comes with several auto reply comments that give hints on when the flags should be validated.
    – Catija
    Commented May 31, 2018 at 19:55
  • @Catija Regarding your comment about #4, I disagree; the question above does focus on how to handle flags, that's true. But it suggests the solution might be to have subject matter experts (SMEs) be the ones to handle them. Implicit in that is the suggestion that SMEs are somehow more qualified to handle VLQ flags... which suggests a total departure from how VLQs are described currently. Currently there is no "subject matter" consideration in VLQ.
    – TylerH
    Commented May 31, 2018 at 19:59
  • 2
    That may be part of the confusion, then... You're right, (by my definition of them) VLQ shouldn't require any sort of subject knowledge... though I suppose that something could look like gibberish to a layman that's actually code to someone who is fluent in that language. Since they're united in a queue with NAA, that's where the mixture comes in. If you consider NAA to include "this answers something but not the question asked" (which I do), that's where the SME really comes in, not with VLQ posts.
    – Catija
    Commented May 31, 2018 at 20:18
  • 3
    TylerH and @Catija : I believe Tim's point about SMEs is not so much that they would be especially helpful for detecting that utter gibberish posts are indeed gibberish, but rather that they might be better suited for spotting meaningful content in posts that someone else saw as gibberish (i.e. efficiently declining misapplied flags).
    – duplode
    Commented Jun 1, 2018 at 2:45
  • 1
    @duplode that's the take away from Jon numbers meta.stackexchange.com/a/288121/213575
    – Braiam
    Commented Jun 1, 2018 at 12:41
  • @Braiam Interesting data, and an interesting broader discussion.
    – duplode
    Commented Jun 1, 2018 at 13:20
12

When dealing with Very Low Quality flags you really have two very different types of posts that are involved. There is the official guidelines on what the flag is actually supposed to be used for: complete an utter trash, things not even resembling an answer to the question, stuff not even in a position where it could possibly be judged on any technical merits to determine if the answer was even a wrong answer. Then there's the common misconception of what it means given the (rather misleading) name: that it's for low quality answers, answers that are wrong, fail to answer the question even though they attempt to, have major technical inaccuracies, or are simply written poorly enough to just not be a useful answer.

When it comes to the first category of posts, the flag is simply not useful. Any answer that is really that bad, to the point that it's either incoherent, or very demonstrably not a sincere attempt at answering the question, is already covered by the Not An Answer flag reason. I have never seen an answer that actually merits a VLQ flag that didn't also meet the NAA flag. If the only goal here is to make sure that those types of posts are flagged and dealt with accordingly, all you really need to do is remove the flag and let people use NAA instead. Some people seem to think it would need to be re-worded to accomplish that. I personally don't think that would be necessary, but it's possible that something clearer could be found.

As for the second category of posts, if this is really a suggestion that the flag's purpose and usage be altered to be a means of flagging said posts for deletion by the community, as the question implies by suggesting the flags be handled by people familiar with the topic, I think it's quite a poor means of doing so at best. Here, again, other tools generally do quite a good job of handling the situation. Downvotes and comments are quite useful for answers that are simply wrong, unclear, or just otherwise not very good answers. Rare cases where an answer is suggesting something that's actively harmful can be deleted by 20k users. Deleting said posts is rather hard, but said posts are also uncommon, and honestly I think it's appropriate that deleting such posts isn't easy, as it is effectively a way for a small handful of users to override the community's ability to even provide feedback on a post, which is a rather extreme measure.

If we think that it needs to be a bit easier to delete posts that are very clearly answers, but that the experts in the field feel strongly are very harmful, then we don't really need to radically change the system, nor do we really need to use "very low quality" at all. If you want to make it a bit easier for said answers to be deleted, then perhaps the privilege to delete an answer should be lowered a bit, or perhaps users with a gold badge in a tag should have the 20k privilege to delete answers in question that tag, even if they haven't reached 20k yet. Personally I think that a silver badge is probably not a high enough bar for such a privilege, but if you think that they should be qualified to vote to delete posts based on their technical merits then just give them the delete privilege rather than trying to shoehorn in the VLQ flag somehow.

As for a review queue of some sort, I'm pretty strongly opposed to that one. The whole design of a review queue is that it's a way of helping people find posts that they normally wouldn't have interacted with (either because it's not in the tags that they frequent, or because they looked at the title and specifically decided not to look into that question because it wasn't something that interested them). For the existing queues that's fine because they're not expecting much, if anything, in the way of subject matter expertise. The whole goal of those queues is to help people find posts outside of their areas of expertise that they are able to moderate. But when it comes to the very specific case of an answer being so harmful on its technical merits that it deserves to be deleted without the ability for others in the community to even be able to see it or give feedback on it, we just don't want that. The only people making those decisions for the post should be people who would feel comfortable enough to go to that post and interact with it without being directed their by a review queue for that specific purpose.

If, on the other hand, the goal here is not to remove particularly egregious answers (on their technical merits), but to actually delete your average, run of the mill, low quality answer, once again, the VLQ flag isn't really a good way to accomplish that. If the goal here is that for answers that are just generally poor, and not useful, but no so egregiously harmful as to require 20k (or other previously suggested) users to delete them, then just use downvotes for that. If we don't feel that deleting an answer should require such a high qualification, then simply "delete" answers that have a low enough score (perhaps something like -6). Maybe this would even be a different type of deletion than what we currently use the term to mean; maybe such an answer would just be hidden to users without X rep, or without a silver/gold badge in the tag (and that would allow upvotes by any user that could see it, potentially bringing it back above the threshold). I don't really think that answers of this nature need to be deleted or otherwise restricted so strongly, but if these are the types of answers that the community in general wants people to not be able to see then, again, there's no need for a review queue for it, let the people voting on the posts, the people who have come across the post naturally because they're actually interested in that question an invested in it getting good answers, make that call.

7
  • 1
    I like this idea, as I browse the single tag I observe (matlab), and don't do much queue work. I flag answers often enough when the pop up in SOCVR through various bots, but those are the clear cut gibberish usually handled by "abusive" flags. Extended privs on tags where you do own a badge (gold'd be fine for me, I'll just work harder) is a good way to weed out those technical inaccuracies current flags are not supposed to be for.
    – Adriaan
    Commented May 31, 2018 at 21:39
  • 1
    "When it comes to the first category of posts..." And is precisely that group of post that gives the hardest time, specifically the "stuff not even in a position where it could possibly be judged on any technical merits to determine if the answer was even a wrong answer" part, where some members just say nope.
    – Braiam
    Commented May 31, 2018 at 23:57
  • "The only people making those decisions for the post should be people who would feel comfortable enough to go to that post and interact with it without being directed their by a review queue for that specific purpose.". Could you elaborate a bit on this? I'm not sure I understand what you mean here. Why would the source of the content change whether or not the reviewer was suitable?
    – Rob Mod
    Commented Jun 1, 2018 at 0:53
  • 4
    @Rob Because, roughly speaking, if I naturally click through to a question, that means I've seen its title and thought "Aha, this sounds like the sort of thing I know something about - let me have a look" and if a question gets shown to me in review it simply means that it was in one of my tags. The former is a much more precise filter for subject-matter expertise than the latter is. Being an expert in FooScript++ doesn't mean you have even minimal knowledge about some random FooScript++ library asked about in the fooscript++ tag, and you can tell that from the title, but the site can't.
    – Mark Amery
    Commented Jun 1, 2018 at 12:09
  • 2
    @Rob On top of that, there's another difference that Servy also points out here, which is that I likely won't click through to a question naturally unless I have some minimal interest in the topic it's asking about. That seems likely to make me err more on the side of seeing the value in it and trying to refine and preserve it. When encountering a post in review, on the other hand, I probably don't care about it at all, so my willingness to carefully read and reflect on it is lower, and my instinct is more likely to be to view it as a nuisance that I should clear out of the way.
    – Mark Amery
    Commented Jun 1, 2018 at 12:16
  • I particularly like the last paragraph. It has the potential to automate the issues of getting rid of crap - which is a huge issue for the community - as currently downvotes will grey out the post, but if it's hidden from lower rep users, even better.
    – user3956566
    Commented Jun 1, 2018 at 15:05
  • One thing about the deletion-by-downvote: you can downvote from 125 rep, but the threshold for seeing 'hidden' answers will probably be need to put a lot higher. If I downvote an answer, I have a 5 minute grace period to remove my downvote (eg because I misclicked), and after there is an edit, I can revise my vote again. We all know that people don't go back to downvoted posts often because of not-caring or the system not letting us search them easily, but what if I'm at 200 rep and the answer is revised? I still won't be able to see it and thus revert my downvote.
    – Adriaan
    Commented Jun 11, 2018 at 8:40
9

It has always baffled me why there are two separate flags. Just merge them and let them all of them go through the low quality review queue.

That's all that needs to be done really, there's no need to discuss the inner details of why a post is bad. Some people on meta are obsessed with this. But apart from spam, rudeness and similar that call for moderator attention, we don't need to categorize crap. Either something should be removed from the site, or it should not.

The only thing that's important is that crap is removed from the site in a somewhat timely fashion. Preferably handled by the community as far as possible, so that moderators can focus on more important things.

I don't quite see why tag experts would be needed to do these reviews.

13
  • 3
    This looks like a tl:dr; version of Yvette's answer (don't know if that's a new trend, but Yvette already addresses merging VLQ and NAA as piling things that should be deleted unto one review queue, and also questions why people should be tag experts).
    – Erik A
    Commented Jun 1, 2018 at 11:04
  • 1
    we don't need to categorize crap. totally agree with all these points
    – user3956566
    Commented Jun 1, 2018 at 12:31
  • @ErikvonAsmuth Undo's answer is so popular. But I still stand by having a categorical list so people can refer to it.
    – user3956566
    Commented Jun 1, 2018 at 12:32
  • 2
    @Yvette This certainly isn't intended as criticism. I especially appreciate and agree with your answer on this question. I just noted that this answer has the same real points as yours, and that this has happened previously. The answer on that other thread is valuable as a resource (so people can refer to specific paragraphs of it), and certainly should stay imo.
    – Erik A
    Commented Jun 1, 2018 at 12:39
  • @ErikvonAsmuthoh I thought you were being supportive, no problems :)
    – user3956566
    Commented Jun 1, 2018 at 12:45
  • "I don't quite see why tag experts would be needed to do these reviews", well, there's the saying that the junk of someone is the treasure of another. What qualifies as junk vs treasure is in the eye of the beholder, and who more capable of evaluating objectively a post that someone that knows what's being talked about?
    – Braiam
    Commented Jun 1, 2018 at 12:47
  • @Braiam Sounds like a case for down votes to me.
    – Lundin
    Commented Jun 1, 2018 at 12:57
  • Actually, downvotes aren't that powerful to separating crap from gold. Lets remember that one of the most decried reasons for "easy" Q&A having disproportionate amount of votes is because it caters to the lowest common denominator. You can't form an opinion of something you do not understand (well, you can, but it wouldn't be objective). Also bikesheds.
    – Braiam
    Commented Jun 1, 2018 at 13:02
  • @Braiam In my experience they work ok. A fairly common scenario is: someone posts an answer which seems to make a lot of sense to beginners. They get 5 or so up-votes. Some expert comes along, spots a problem in the answer that nobody thought of, point out this in a comment and downvote. Perhaps even posting a correct answer by themselves. Then during the next day or so, other experts find the same thread, recognize that the remark by the first expert is correct, then cast their votes accordingly. This can completely turn the tables on a post.
    – Lundin
    Commented Jun 1, 2018 at 13:09
  • (I find myself in such a scenario right now as we speak. In a post, two 100k+ users have posted answers that I think are just plain wrong and misleading, I've downvoted, left comments and posted an answer of my own. We'll see how it pans out. But flagging for deletion would not be right thing to do at all.)
    – Lundin
    Commented Jun 1, 2018 at 13:10
  • 1
    I would instead point out a counterexample, which anyone would agree they are terrible answers: security holes. Check the timelines of the top answers which has the warnings, they still get upvotes despite the serious warnings that they are terrible solutions to the problem presented. So, voting actually doesn't work for something that should be universally bad (and even explained in non ambiguous terms), how do you expect to work for technically irrelevant posts?
    – Braiam
    Commented Jun 1, 2018 at 13:50
  • 1
    @Braiam I think that's a separate issue. It would best be solved by appointing a group of "technical moderators", such as PHP gold badge users, then give them free reigns to clean up such posts. This can be solved today even, if users are a bit creative. Suppose one PHP gold badger posts a community wiki reply on meta, stating the exact course of action for each post. Leave it there for some time until there's a fair community consensus, then poke your favourite diamond mod and ask if they can implement it.
    – Lundin
    Commented Jun 1, 2018 at 13:59
  • 1
    The point is: votes do not reflect technical correctness of the answer. Is more a proxy of popularity rather than technical accuracy. Votes supposedly should separate the useful vs not useful, but in something that is clear that the post aren't useful, they are still being evaluated as useful. You simply can't trust the crowd wisdom.
    – Braiam
    Commented Jun 1, 2018 at 14:12
8

I don't think we need specific VLQ reviewers because we let anyone raise VLQ flags. That would be a marked departure for how flags and review queues work, and it would balloon these queues. Plus mods aren't exactly experts in everything. But that doesn't mean it's a total waste of a suggestion. I would suggest something a bit more... simple

  1. Merge NAA and VLQ into "Should be deleted". Just today I got into two separate discussions about keeping some poor answers. Both were technically answers, but not very good ones. It's hard to convince people that junk/noise shouldn't be deleted just because it contains some shred of an answer. So let's drop the pretense and just make them one flag. Solves the confusion and it means we can more readily nuke noisy answers.
  2. Give silver and gold badges a one-shot delete in the LQP review on things in their own tag (provided they have at least a bronze badge in LQP).
1
  • Silver sounds low for a one-shot delete, as you can silver badges as a 'side dish' so to speak. I answer only MATLAB questions, and already got a bronze [matrix] badge, and [array] won't be far off. Continuing like this, I would be able to delete all kinds of matrix/array questions in tags I have no knowledge about (C++/Java/Python etc), thus the 'SME' part is lost that way. Perhaps make silver a two/three shot or something.
    – Adriaan
    Commented Jun 11, 2018 at 8:42
5

As we all (might) know there is this awesomely designed flowchart from Shog's question here: Help us test question triage!

Enter image description here Freehand circle mine

And we all know the fatal flaw with the VLQ flag that falls within the freehand circle. As a newly 2k+ user I gave a shot at the Helper queue and found my fair share of VLQ questions, which I flagged with the appropriate link, only to see them again 10 review items later.

So we can all agree Triage isn't the best place for handling these VLQ flags. Maybe most notoriously because it is one of the first queue's to open up to a user, so these users are:

  • Not the most knowledgeable on how things (are supposed to) work on this site
  • Not the best judges of quality (yet)
  • Might err. on the safe side of things because of above two reasons

Especially the latter would make users more prone to click "Should be improved" as there is no real consequence to it. They are not sure that it "Looks OK" and might be reluctant to choose "Unsalvageable" as their own posts may show striking similarities to the post they are reviewing.


It would be nice to have some statistics on the following part, to either prove or disprove my assumptions:


I think the Triage queue can do a good job at shifting out the real crap from the VLQ, say the worst 10%. (It would be nice to see what percentage of VLQ flags that go into triage actually result into "Unsalvageable") Now we could argue that this is the worst kind of garbage that comes with a VLQ flag attached to it, so why should we bother putting that into the close vote queue, when already 3 users decided this is beyond any help. Hence straight up delete a post when it amassed three "Unsalvageable" votes form Triage. I think the diligence of putting it in the close vote queue is just abundant, as most likely triage users err on the safe side and rarely use the "Unsalvageable" option as I reasoned before. (Statistics to (dis-)prove me are welcome.) This would deal with Shog's notion that garbage tends to linger around the site for too long.




Now as to whether or not we need tag experts or just all-round experts to judge the now remaining "Should be improved" flags is the last step.


To edit posts into general shape, mostly no domain knowledge is needed. Adding images inline, apply code formatting, reduce the occasional "Thank you for you help" and whilst at it improve some grammar and spelling, is something most of us can do in the Helper queue.

Then of course there are questions that are flagged "Should be improved", but are in no way near an actual question, or something that barely resembles an answer. One with domain knowledge is by default better able to judge these edge-cases and is certainly better able to find a duplicate to close against.

But isn't that where the Hammer and down/close vote are for? For questions that look well formatted, appear to be an actual question, and not some rambling about a vague problem without a desired result or any attempt at a solution. There is already power for those with domain knowledge to get these "pseudo-questions" out of the way.

So how to deal with those questions that "Should be improved", but are just beyond repair? Casting a VLQ flag on them certainly isn't helpful, as it will pop-up in the helper queue again after reviewing 10 items or so. Personally I have been doing some testing with a couple of review items in the helper queue. I opened the post in a separate screen and cast the appropriate close flag. I then went back to the helper queue and cast the VLQ flag as well. This way the VLQ flag is also shown in my flag history. In most cases my close flag was deemed helpful, whilst my VLQ flag was disputed, indicating that:

  • The post was close worthy
  • Triage reviewers didn't share that opinion

So I say we should remove the VLQ link from the helper queue as there really is no use for it and substitute it with the close flag menu. We can then keep Triage as a real Triage, separating the absolute garbage from the rest, and the Helper queue can either whip the remainder into shape or get rid of it.

So to get back to the original question:

No, we should not. We should let the garbage be shifted out by anyone willing to help (a.k.a. the Triage review). They can get rid of the real crap. What then remains as not up to standard ("Needs Improving") can be handled by anyone through the helper queue in which they either edit questions into shape or flag to close them. Domain experts can then use their normal tools to get rid of questions that were edited from the helper queue, but still make no sense by using close flags and down/close votes.

6
  • 3
    I'd like to hear Shog's feedback on this. Love the freehand circle
    – user3956566
    Commented Jun 1, 2018 at 12:26
  • 1
    This post might deserve it's own Q&A, since it's only slightly related to the question (the original question is talking about changing the Unsalvageable path). I've got my own views on how we could break the triage-H&I infinite loop, but it's certainly an important point that should be addressed, thus this should get the attention it deserves.
    – Erik A
    Commented Jun 1, 2018 at 13:10
  • 1
    @ErikvonAsmuth, it indeed has a broader scope then just this question alone. But I'll wait a few days to see if it gets any exposure and how well it is percieved. If it really turns out to be something I might post it as a Q.
    – Luuklag
    Commented Jun 1, 2018 at 13:42
  • @Peter Mortensen, thanks for editing. Wrote this from work where a checker isn't allowed to be installed. Company policy -_-
    – Luuklag
    Commented Jun 4, 2018 at 6:28
  • This is a reason to improve Triage.
    – Raedwald
    Commented Sep 25, 2018 at 18:25
  • @Raedwald I propose to just get rid of VLQ flags as IMHO they don't have any added value. A different solution to the same problem.
    – Luuklag
    Commented Sep 25, 2018 at 18:50
3

I like the idea about content-specific VLQ-ness.

Take the following hypothetical "answer":

You need to reticulate the spline because C is a very high level specification language. Upgrade your Mac from Windows 95 to 3.1 and install a double-bypass circuit wrench. Run the following command: set install all walls=off reroute-power-coupling=area-b run routine alpha charlie

Anyone with any meaningful experience in the field will recognize that the above "answer" is pure technobabble garbage, but someone who is computer illiterate might not. I don't think many of the mods here are computer illiterate, but I still get the impression that they would probably decline a VLQ or NAA flag on such an answer.

Is this what you are talking about when you speak about tag experts reviewing VLQ flags?

Right now, VLQ and NAA flags are not supposed to be used for answers that make any attempt to answer the question, no matter how incompetent. The following decline reason exists for such:

flags should not be used to indicate technical inaccuracies, or an altogether wrong answer

Perhaps we could change the rule so that VLQ and NAA flags are not supposed to be used for answers that make any educated attempt to answer the question. The flag decline text could be amended to:

flags should not be used to indicate technical inaccuracies, or an altogether wrong answer, as long as the answer demonstrates minimal domain knowledge as would be expected of an educated person with entry-level skills in technology.

Then, utterly incompetent answers can be marked as VLQ or NAA, and answers that are incorrect, but require more than basic computer skills to recognize as incorrect (e.g. claiming that strings can be made mutable in Java with a special compiler directive) would still be downvote-only.

One problem with this, however, is the inevitable debates on how much content knowledge should be expected of a minimally competent (that is, non-NAA/non-VLQ) answer. Should we require a basic "newb" level where one can play around with things that one barely understands but probably knows enough not to cause a major catastrophe? Enough knowledge to get an entry-level job? One year of experience in the field? Ten years? Major guru? God incarnate?

4
  • 2
    I'd like to point out that if there is actual concern to deal with the types of answers you've given an example of...they're basically never posted outside of spam, and when spammers do post answers like that you can just flag them as spam, so we don't need to amend VLQ to handle it.
    – Servy
    Commented May 31, 2018 at 18:08
  • 3
    Some tangential discussion on this from yesterday
    – Shog9
    Commented May 31, 2018 at 18:13
  • 2
    Your hypothetical "answer" reminds me of this one.
    – honk
    Commented May 31, 2018 at 18:32
  • This sounds more complicated than the existing system and an answer like that wouldn't stay on the site if a mod saw it, flagged or not.
    – user3956566
    Commented Jun 1, 2018 at 3:01
-1

Whether an answer is not even wrong sometimes requires understanding the question, but most answers that should be deleted don't fall into this: link-only answers, follow-up questions, thank-you notes... So while topic experts do sometimes have an edge, that's only in a small minority of cases.

Silver tag badge holders may be useful as fallbacks for disputed cases, but in the general case I don't see the point.

There's also the serious problem of all those questions that don't have any tag with enough silver tag badge holders who are also active reviewers. Are they relegated to a cesspool?

Sorry but this idea sounds a lot like you had a hole to fill, and you had a peg you wanted to get some use from, but that peg just doesn't have the right shape for the hole.

-7

One problem I see is that very often "very low quality" includes wildly wrong tags. Placing the moderation mostly in the hands of parties interested in totally unrelated tags doesn't look sensible.

2
  • 2
    very low quality flags are 95% done on answers. Answers don't have tags. Commented Jun 1, 2018 at 11:28
  • 2
    @Jean-FrançoisFabre The question is talking about tag badges though, so presumably they're talking about the tags of the question that the answer is posted under. Low quality questions tend to attract low quality answers, so when reviewing low quality answers you'd tend to find that the questions they're posted under are going to be of lower quality, on average. A sizable portion of them will be poorly tagged. I don't know if that would be a dealbreaker, but at a minimum it's a problem that will exist and will need means of mitigating.
    – Servy
    Commented Jun 1, 2018 at 13:07
-7

That's a terrific idea, that is in dire need of...

SCIENCE!

Before implementing this, why not asking the would-be-reviewers beforehand if they actually agree with the ultimate action taken? This is some thought about how the experiment could unfold, but maybe taking the list of flags handled by moderators and review queue, both declined and approved and present the test subject what actions they would take on the presented post (keep, edit, delete, or skip) and why. I would like a two pronged test with a group shown all the context and another that they have to click, but I'm unsure if there would be enough users to try that.

I think that would be more convincing along the way.

You must log in to answer this question.