83

It's been about 5 months since we started testing Triage. There've been a fair number of hiccups, and there's still plenty of work to be done refining the criteria, but at this point I think the process is working pretty well: with over 1800 questions reviewed every day on average (over 20% of all questions asked), these dedicated reviewers tirelessly sort a huge number of questions into categories that make them easier for the rest of us to deal with.

Some quick stats:

  • 50% of questions triaged as Looks OK actually get answered - while only 9% end up with a negative score.

  • 73% of questions triaged as Unsalvageable get closed and/or deleted, while only 2% end up with a positive score.

  • 29% of questions triaged as Should Be Improved get closed and/or deleted, and 33% get answered.

It's that last category I want to address... It is, as it was intended to be, a sort of limbo: no immediate action results from such categorization, but rather the question waits for some sort of intervention to save or damn it. Questions put there run the gamut from merely poorly-written to deeply-flawed. We do offer tools to help folks salvage such questions, but that still requires a lot of skill and care - many questions, particularly those where the original author is absent or uncooperative, simply won't make it.

David Fullerton had an analogy for this that I like quite a bit: he compared these questions to objects drifting down a river, with folks standing on the shore with poles dragging in what looked interesting to them.

Eventually, anything of value would be either claimed or washed out to sea. Right now, with the cooperation of a large number of reviewers in the Help and Improvement queue, roughly a third of these questions are hooked in by someone on shore...

...The rest should meet the Kraken.

I propose the following criteria for automatically deleting these questions, based on those for the automatic deletion of abandoned, closed questions:

  • Triaged as "Should Be Improved" more than 9 days ago
  • Score <= 0
  • Not closed
  • Not locked
  • No answers with a score > 0
  • No accepted answer
  • No edits in the past 9 days

If you're interested in seeing what this would catch, here's a SEDE query.

Play around with this a bit, if you would - I'm not seeing a lot of stuff worth keeping here, but it's likely that I'm missing something; if you've suggestions for better criteria for this, post 'em below.

14
  • 4
    This sounds fine. It uses the same logic as if it was closed, which is not too far from reality (since nobody answers them).
    – Braiam
    Commented May 6, 2015 at 3:33
  • 8
    consider editing to add more detailed explanation of why you consider answers to these questions not worthy of keeping. You probably have solid reasons for that, as you tend to be rather careful about deleting answered questions (overly careful to my taste:) but not all readers are aware of that, as indicated eg by this answer
    – gnat
    Commented May 6, 2015 at 11:19
  • 2
    Will the user be notified when their post is automatically deleted?
    – Travis J
    Commented May 6, 2015 at 20:27
  • @TravisJ if you are not paying attention to your own question, why should we?
    – Braiam
    Commented May 6, 2015 at 20:44
  • @Braiam - You should really learn to phrase your comments to be more general. I have a gold badge for asking good questions.
    – Travis J
    Commented May 6, 2015 at 20:49
  • 3
    As to why, in general, users should be notified. I think if we are trying to strive to offer some sort of redemption to these questions, and there are already comments suggesting changes, the last step may be a notification that their question was removed with perhaps some extra documentation to either visit or read. While this may not be perfect, it at least will be sent to a very targeted audience of users who desperately need to learn how to ask better questions and be more attentive.
    – Travis J
    Commented May 6, 2015 at 20:51
  • 1
    @TravisJ there are several already. Try asking a question in private browsing as new user, did you read that? you have to click the checkmark, that you "understand" what, how, and who of Stack Overflow. If they don't read then, there isn't too much redemption, there is?
    – Braiam
    Commented May 6, 2015 at 21:07
  • 1
    @Braiam - Again, why direct this at me? I am sure if I reciprocated this in turn it would devolve quickly. The information presented which you mention is prior to asking. This information would be different and could point to documentation on question bans and fixing already asked questions. It is already pretty understood that users avoid reading at all costs, so this might be a good opportunity to offer then something to read when they are actually potentially interested in why their question was deleted.
    – Travis J
    Commented May 6, 2015 at 21:12
  • 13
    That would be shutting the barn door after the horse has already bolted, @TravisJ. "Oh noes, my question is deleted a week after I'd forgotten about it completely. Shall I 1) edit it and hope it somehow gets undeleted? 2) post a new question containing the edits? 3) Move on with my life as I'd already done 9 days ago without this stupid ping." How 'bout instead we show folks who don't abandon their questions some sort of indicator that they should do some editing lest their question go the way of all flesh.
    – Shog9
    Commented May 6, 2015 at 21:35
  • @Shog9 - I was under the impression the new queues already include these indications in the form of comments. What other indication did you have in mind?
    – Travis J
    Commented May 6, 2015 at 21:39
  • 13
    @TravisJ is not you Travis J, but you the asker. I presumed that that would be obvious by now. BTW, english has many use of "you", one of them is "one".
    – Braiam
    Commented May 6, 2015 at 21:50
  • 2
    wrt indication, how about marking a question as put on hold by Community user (as unclear) after it ages away from H&I / Triage loop
    – gnat
    Commented May 6, 2015 at 21:51
  • 1
    @Shog9 - "The folks asking these sorts of questions don't even bother to read the comments - I have to point them out when they ask why they're banned from asking." meta.stackoverflow.com/questions/289315/… This was a comment from you, explicitly stating that you have to manually show users why they are banned once it happens. So stop chasing the horse all by yourself already. Let the system automate some of this process. Banned users do read, so take advantage of it.
    – Travis J
    Commented May 8, 2015 at 16:56

6 Answers 6

58

This is less about the automatic deletion (which I wholeheartedly agree with) and more about the idea of "Should be Improved" in general - early on in my Triaging, I was marking a lot of stuff as Should Be Improved. But it quickly dawned on me that "Should be Improved" was really for the rest of us, not for the author. However, more often than not, the improvements really needed to come from the author, either in the form of missing information or in changing the scope to meet site guidelines such as a narrowing of scope.

So I began to stop using Should Be Improved as much and went for using Unsalvagable more. However, that wording sounds far too harsh for what it actually means; if the question deserves a close vote, it should be marked as Unsalvagable. But this word directly goes at odds with the concept of having an On Hold grace period for these questions. It really ought to be Requires Author Intervention And/Or Unsalvagable, or, rather, a more brief version that means the same thing. Alternately, the help text should be clarified to say "if this question ought to receive a vote to close, choose Unsalvagable".

Here are some questions that really are Unsalvagable, yet the consensus did not agree, to see what I mean. 1 2 3 4 5 And some more controversial ones that really need more input from the author 1 2 3

I think these improvements will result in more questions appropriately getting closed instead of filling the Help and Improvement queue with things that really need author intervention.

tl;dr these stats prove that the question asker here is correct: Rephrase Triage help text for "Should be improved" and "Unsalvageable "

19
  • 4
    Rumors to the contrary aside, Unsalvageable does not mean "should be closed". Plenty of closed questions can be salvaged (and some are) - but there's an insane amount of overhead for that. Unsalvageable is what it sez on the tin: there is no hope.
    – Shog9
    Commented May 6, 2015 at 3:57
  • 23
    @Shog9 if that's the case, then why don't I get an opportunity in the UI to close vote questions that are clear CV candidates after clicking "Should Be Improved"? The only way for me to close vote these questions is to click back and then open the question separately.
    – durron597
    Commented May 6, 2015 at 4:28
  • 7
    @Shog9 In other words, I understand what you're saying is that there is no hope is the original intent of the triage queue, but the results is that we now have a need to generate an autodeletion mechanism for should be improved questions, especially because the answer rate on these questions is so low.
    – durron597
    Commented May 6, 2015 at 4:30
  • 25
    @Shog9 your comment directly contradicts the workflow of the triage queue. If "unsalvageable" means "there is no hope for this question", and "should be improved" means "it's an ok question, and we're sending it to the HIQ so that the community can polish it", what exactly should happen with the inbetween questions, e.g. everything which should be closed as unclear? Right now, these are under "unsalvageable", which is problematic as many users interpret that as "there is no hope" and thus choose "should be improved" for questions that cannot reasonably be fixed by the community.
    – l4mpi
    Commented May 6, 2015 at 12:07
  • 32
    @Shog9 - If "Should Be Improved" and "Unsalvageable" are independent of whether or not something should be closed, why does choosing "Should Be Improved" lead to invalidation of close votes? If you're going to tweak the Roomba to clean these up, why not remove that vote invalidation, too? That would let "Should Be Improved" questions that should be closed get closed, and help with the overall process. Whether or not to do the same with the currently-invalidated "very low quality" flags is another question.
    – Brad Larson Mod
    Commented May 6, 2015 at 15:25
  • 11
    Disputing the flags was an expedient way to avoid skewing the results here, @Brad. Combined with flag aging, we could drop that, at least for Should Be Improved. I'm actually thinking we should bite the bullet and mark VLQ flags as helpful from Triage when the result is either SBI or Unsalvageable.
    – Shog9
    Commented May 6, 2015 at 15:50
  • 2
    SBI does not at all mean "this is OK", @l4mpi. There's another option called "OK". There are entirely too many Unclear questions to be closed - even if we gave the top users in every tag insta-close abilities, they'd have to spend all day closing in some tags just to keep up. So let's cut to the chase: if the problem with a question is that it can't be answered without improvement, and it doesn't get improved, we should skip the closing ceremony and just get rid of it.
    – Shog9
    Commented May 6, 2015 at 15:56
  • 21
    @Shog9 But it's a waste of time to put something in the Help and Improvement queue that can't be improved without the author's help.
    – durron597
    Commented May 6, 2015 at 16:00
  • 14
    @Shog9 That's not what I'm saying at all. I don't feel the HI queue is a waste of my time, I think it's excellent! I'm saying that encouraging users to put questions like the one I linked into it from Triage wastes the time of those users that would be better spent revising questions that can actually be salvaged.
    – durron597
    Commented May 6, 2015 at 16:49
  • 16
    @Shog9 In fact, I'm saying just the opposite. I'm saying there's no hope for that question without original author intervention. The only way to fix it is to involve the original author. It is too broad and problem scope can only be narrowed by the original author adding more information to generate a question that can be answered / improved upon by the community. It is questions like these that the [on hold] status was designed for without making questions immediately go into [closed].
    – durron597
    Commented May 6, 2015 at 17:03
  • 5
    @Shog9 not really, no. The problem is that the edits need to come from the author, not from the community. I have created a new question suggesting the addition of a fourth button to Triage
    – durron597
    Commented May 6, 2015 at 17:37
  • 12
    @Shog9 I was unclear, what I meant is that Tim Post says "should be improved" means "a clear on-topic question is present, it just needs some help to be made clearer"; and you say "unsalvageable" means "there is no hope". There is a huge gap between those definitions, namely all questions that should be put on hold because they need improvement by OP. Based on the meta discussions about triage and HIQ as well as my experience, far too many people choose "should be improved" for all those questions as they think it includes improvements by OP.
    – l4mpi
    Commented May 6, 2015 at 19:20
  • 17
    @Shog9 Was my comment too ambiguous again or are you actively trying to ignore my point? I have no problem with Tim's definition of "should be improved", I have a problem with your (and his) definition of "unsalvageable" leaving a gap between it and SBI as well as being inconsistent with the workflow for unsalvageable, which currently leads to HIQ being flooded with a huge amount questions that cannot reasonably be improved by the community. Auto-deletion of those questions is better than doing nothing, but it only fights the symptoms, not the cause.
    – l4mpi
    Commented May 6, 2015 at 21:47
  • 1
    @Shog9 I think marking VLQ as helpful when Triage rules Unsalvageable makes perfect sense. However, not so with SBI. Consider a VLQ flag cast from HIQ, which sends the question back to Triage. If Triage ends with SBI, the flag was a waste of everyone's time.
    – user3717023
    Commented May 7, 2015 at 6:02
  • 1
    @l4mpi You are assuming HIQ labor has a cost meaningful to Shog9. There is a surplus of HIQ labor with nearly zero cost from the perspective of stack overflow. Sending them "useless" tasks is worth the (nearly zero cost) if it can generate even marginally better results. Sure, HIQ reviewers time is "wasted" (they are given posts that previous people could have diverted), but that is not a problem so long as there is a surplus of HIQ reviewers. Don't HIQ if you don't approve of the by-design stream of junk coming from Triage. Free labor is free labor, let someone who wants to do it. Commented Sep 1, 2015 at 3:40
21

As much as I like to roomba things, I think this will roomba too much.

With your proposal, it would delete 32k questions. Many which have answers.

I think it should be made a little tougher to roomba.

Instead of 9 days since "should be improved" and "no edits in last 9 days", switch it to 45 days since "should be improved" and "no edits in last 45 days". This will still roomba 22k questions or about 67% of your proposal.

There are currently no roombas that delete any questions that have answers, other than questions that are closed. So by deleting unclosed questions that have answers is taking a big step. I understand it's only 20% of questions that go into triage, and only the ones that were marked "should be improved" that were talking about here, but I think it's probably too strict at 9 days. So I suggest changing it to 45 days. This will give the questions time to settle in to Google search and time for people to find them though search and upvote if they found them helpful. 9 days for this is likely too short in my opinion.


If you'd be fine with two new roomba scripts instead of one, I'd go with

  • Triaged as "Should Be Improved" more than 9 days ago
  • Score <= 0
  • Not closed
  • Not locked
  • 0 answers Changed from "No answers with a score > 0"
  • No accepted answer
  • No edits in the past 9 days

So we can get rid of these unanswered ones much quicker.

And then do the 45 day script to catch the remaining ones.

The 9 day roomba would run every day. The 45 day would run once a week.

In addition, I don't mean keep all those questions that don't get edited/deleted in the HIQ for 45 days. Either just remove any questions from HIQ after they've been there for 9 days. Or remove them from HIQ based on your original proposal.

5
  • 1
    i.sstatic.net/MW9QR.gif
    – gnat
    Commented May 6, 2015 at 7:07
  • That's a good point @gnat I still think it's too overkill though.
    – CRABOLO
    Commented May 6, 2015 at 10:58
  • 2
    @gnat For a better world, with only hate/jealousy :P
    – nhahtdh
    Commented May 6, 2015 at 11:35
  • 11
    Not all answers have lasting value, so the mere presence of answer shouldn't necessarily stall cleaning things out.
    – user1345223
    Commented May 6, 2015 at 17:07
  • 2
    It might be good to also factor in the view count, so that questions with lots of views (500+) do not get autodeleted, as they seem to be useful (to googlers at least), even if the answer is not upvoted. Commented May 8, 2015 at 12:44
13

I suggest replacing

  • No edits in the past 9 days

by

  • No edits since exiting Triage

Rationale:

  1. Since the question has been edited, we can't be sure that the Triage outcomes reflects its current quality.

  2. For closed questions, disregarding edits older than 9 days makes sense, since they haven't managed to attract votes to reopen. But there is no such thing as un-Triaging, which is why this condition should not be copied verbatim.

0
12

Questions that require additional input from asker to become meaningfully answerable are considered close-worthy.

Prior to introduction of new queues, their "lifecycle" was expected to follow this: these would be closed and after a while, auto-deleted by old-fashioned 9-days roomba script - "No answers with a score > 0..." etc. It's hardly a mere coincidence that auto-deletion criteria proposed here are basically the same, except for those related to question being closed.

As far as I can tell, old roomba way has somewhat changed with new queues. Thing is, these introduced limited visibility of the questions ("take that, wall of crap"). As a result, triaged questions may just miss eyeballs of flaggers/voters that previously triggered close review.

In that sense, proposed change sounds more like simply adapting of an old roomba script to feature changes involved in new queues.

FWIW you may also test above reasoning prior to making a change. For that, instead of deletion, simply raise a system flag to push the question into close queue (you may want to also adjust its priority in the queue to account for lower visibility and delayed triggering of close review). Keep things that way until you get enough statistics to conclude whether reviewers tend to consider these close-worthy or not.


Note that if your concern is about how new queues impact roomba, this can be also approached from a different angle, without adjusting current criteria for auto deletion.

For that, you could "emulate" question closure that likely would happen prior to introduction of new queues (due to wider question exposure) and that eventually triggered auto-deletion back then.

For example, when a question ages away from Triage / H&I loop, it can be automatically put on hold (by Community user, probably as unclear - since reviewers couldn't make sense of it).

From this point, you would have state of the things essentially recovered to how it would be prior to new queues - protected from being answered, available to editing and reopen review and, of course, eligible to usual roomba deletion.


Another way to address this issue seem to be possible by implementing a recent proposal to let tag badge holders cleanup stuff like that:

show the viewer this option:

delete and request edits

Upon submission, the question will be immediately deleted...

6
  • 1
    The critical issue here is that the folks best equipped to identify and close these also tend to be the folks most annoyed by them. So if we can keep them out of sight until... Or unless they're fixed... That removes a huge burden from these folks.
    – Shog9
    Commented May 6, 2015 at 20:13
  • @Shog9 agree, that's the way I see it (and that's what I love about new queues). You only have to consider side effect that keeping them out of sight also makes them less (much less?) probable to be closed, as opposed to how it was before - so that new features get you a bunch of questions hanging in a limbo while in the past these would be closed and collected by roomba
    – gnat
    Commented May 6, 2015 at 20:20
  • 2
    Hence this FR! Got one more tweak up my sleeve too... Hoping to have this all wrapped up soon though.
    – Shog9
    Commented May 6, 2015 at 20:23
  • 2
    @gnat: the thought that these questions should be closed seems right. If a question is closed, it can still be improved and reopened, can it not?
    – serv-inc
    Commented May 7, 2015 at 11:12
  • 3
    @user1587329 exactly so. In my opinion, for potentially salvageable questions, primary purpose of closing should be making it easier to edit into better shape, followed, apparently, by reopening. "...while open, it can attract irrelevant answers which will make it difficult ... to further edit the question into the shape... protect the poorly worded good question from getting bad answers..."
    – gnat
    Commented May 7, 2015 at 12:26
  • 2
    ...probably worth noting that system is purposely designed to support reopening of edited questions, see Which edits push closed questions to the reopen review queue?
    – gnat
    Commented May 7, 2015 at 20:36
8

Why are almost 30% of reviews being deleted? Possibly because reviewers don't understand that hitting the "Should be improved" option escalates the question to higher moderation and could be adding to a throttle point.

My experience with Triage was a bit like "sink or swim". It wasn't until I failed an audit that I did some research on meta and found some good posts to help focus my responses.

It's an educational thing - when you join SO you go through a tutorial (skipped by some). So why not have something similar when a user hits enough rep to earn a privilege? Tell them how to do Triage. Explain that "Should be improved" means not just a notification to the OP but also escalation to the Help and Improvement queue.

This won't clean the queue but it could improve it over time.

3
  • 1
    Maybe every Nth "Should be improved" should pull the triager into H&I to fix it themselves, so they realize what clicking that button does. Commented May 7, 2015 at 1:44
  • @JeffreyBosboom there are users (me, for exampe) who can review the Triage queue but don't have enough rep for the Help&Improvement (it requires 2K)
    – AJPerez
    Commented May 7, 2015 at 14:10
  • 3
    I would love to see a (mandatory) tutorial for each review queue. The first time you open a new queue, you would be shown a few test questions (or answers). Something like the audits, but hand chosen by our moderators. You would have to choose the right action to take, and after that, you would see a message explaining why what you did was right/wrong. Once you have passed enough questions, you would be allowed to participate in that queue. Or something like that...
    – AJPerez
    Commented May 7, 2015 at 14:29
4

It may be worth including the activity of the original asker as well. I'm noticing a trend with the votes on a given question and a relation to when the user was last present to participate to clarify or to bring it on-topic.

I've modified the query to include an upper bound of when the user was last active on the site with respect to the question that was found, with a nominal max value of 72 hours. It adds a few more interesting questions into the mix for sure, but I feel like the query would be more accurate if I could query the OP's direct interaction with that specific question...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .