49

This was pointed out in an answer to the original topic thread, but could probably do with more extraction, as it can be viewed as quite subjective whether a question qualifies for "Looks Okay" vs "Needs Improvement" or "Unsalvageable" and "Needs Improvement."

Specifically, the gray areas here marked by Vs:

Looks OK---V---Needs Improvement---V---Unsalvageable

There are questions that appear obvious, and then there's the gray area where we end up with 3-2 distributions in favor of OK/Improvement or Improvement/Unsalvageable.

Edit - I'd like to also point out that in medical triage, there are very defined categories so that people will not get stuck on picking between colors. If you don't meet a single criteria, you get put into another category.

I'm not sure how effective triage would be if a large portion of these questions are decided based on a tie breaker of one person in a group of five or so people. In many cases that I've seen, most of the LQPRQ posts are unanimous, or with one possible dissenter. In many triage cases, I'm seeing very close splits on the Good Enough or Bad Enough line, probably because, even with the provided definitions, these haven't been fleshed out to the point that we can establish how much improvement is required versus too much improvement needed to be salvageable.

Obviously, since this queue is still in its infancy, there is no defined consensus as to what to do in regards to these questions. Should we round up? If it's borderline unsalvageable, that indicates it might still be fixed. And whether it looks "good enough" also feels subjective. Thoughts? Should we "round up" and give the benefit of the doubt? Round down and say "any hesitance indicates that it probably needs another look?" Round every other and say "Anarchy?"

I'll provide two examples of these situations that I encountered today:

Exhibit 1 - A 3-2 split on Needs Improvement/Looks OK

https://stackoverflow.com/review/triage/6387075

The question itself looks well-formed, but would benefit much more from a smaller code example. When we consider the two categories:

Those that will likely be well-received by the community and obtain answers with no further structured review.

Those for which additional guidance or revision might result in a positive reception and useful answers.

We still see a gray area, resulting in the disparity, it seems between the 5 of us.

Exhibit 2 - A 3-2 Split on Needs Improvement/Unsalvageable

https://stackoverflow.com/review/triage/6387339

It's a pretty bad question, but whether it is doomed to fail or not appears to lean towards "We can fix it." Determining the limit would probably have made this an easier choice between the two.

10
  • 4
    Skip?
    – gnat
    Commented Dec 4, 2014 at 18:06
  • 6
    @gnat Well, it's not a question of "I don't know what to do at all." I have a pretty good idea it fits into one of the two categories. But unlike other queues, it's significantly more subjective where one of those categories ends and the other begins.
    – Compass
    Commented Dec 4, 2014 at 18:12
  • 1
    @gnat actually, if I were to propose a change, perhaps the following. Two types of queues: Looks Okay vs Needs Improvement. Questions that get 3 OK clear the queue. Questions that get 3 Needs Improvement enter the NI/Unsalvageable queue. This would allow to indicate the general consensus so far and make it a fair dichotomy where a Skip is actually justified once it reaches the correct queue, rather than no input at all.
    – Compass
    Commented Dec 4, 2014 at 18:20
  • 1
    that sounds interesting. One nitpick - people will complain about Needs Improvement in the first queue, because they would think it leaves no place for Unsalvageable ("WTF that's total garbage but I only have options it's okay and can be improved?"). To avoid a cognitive dissonance like this, I'd use Doesn't Look OK as a second choice
    – gnat
    Commented Dec 4, 2014 at 18:25
  • 2
    @gnat Yes I think that would be better. Rather than trying to solve all the issues in one go, determine whether an issue exists, and then handle the issue.
    – Compass
    Commented Dec 4, 2014 at 18:29
  • 1
    FWIW, I'd consider the first exhibit "unsalvageable" as well - not because it's impossible to answer, but because it's impossible or at least way too complicated for anybody but OP to edit it into an acceptable question that meets the SO quality standards. Seriously, the problem description is "shows the error Error to Inserting into database at the end of the code" - anybody who clicks "looks OK" on that should be suspendend for a week IMO.
    – l4mpi
    Commented Dec 5, 2014 at 10:34
  • @l4mpi Unsalvageable, as far as I've interpreted it, is "No amount of work by anyone will make this question work", or "The amount of work required to make this a good question would be more than making a new question."
    – Compass
    Commented Dec 5, 2014 at 14:24
  • 2
    @Compass Closing is under the "unsalvageable" option, thus I interpret it as "should be closed (or insta-deleted)". And my threshold for closing questions is way lower than "No amount of work can fix this"...
    – l4mpi
    Commented Dec 5, 2014 at 14:50
  • I'd say it would make sense to round to the middle. If some people think it's salvageable and others done, it's probably salvageable. If some people think it needs work and others don't, it probably needs work.
    – neminem
    Commented Dec 5, 2014 at 23:20
  • I'd say your example #1 between "looks ok" and "should be improved" is a clear "should be improved" (I agree example #2 is more borderline, ultimately it should be flagged as duplicate) The question in #1 is an acceptable question. The question can be deciphered and answered as written. That doesn't mean its a good question; the person answering has to read unnecessarily long code segments and figure out most of what the OP is trying to do. Which, the question could clearly be improved to help the answerer not have to do as much. This is literally what the "should be improved" queue means.
    – aestrivex
    Commented Jun 19, 2015 at 15:09

2 Answers 2

47

There are no hard borders between these options. At one extreme lies spam and utter nonsense; at the other lies a novel, clearly-written question that already has an amazingly useful answer. But in between... Well, there you have to accept that you're looking at sand, and even the prettiest sand doesn't guarantee a pearl.

That's the hard truth: we've put together a lot of guidance on what makes for a good question, but the best question is always the one that gets the answer you need - and it's amazingly hard to predict which one that'll be.

So instead, we look for things that might make questions more appealing to answerers and easier to find for others with the same problems. Is it...

  • ...About programming?
  • ...Reasonably-scoped?
  • ...Clearly written?
  • ...Detailed enough to identify the problem, with a descriptive title?
  • ...Nicely formatted?
  • ...With relevant tags?

If so, I'd say it Looks OK. If it's lacking all of these, it's probably Unsalvageable. Everything else Needs some Improvement - but if the need isn't great, it could still be OK, while if it is severe the chances of it being salvaged are still pretty slim. So I don't think there can be a hard rule here; a lot of the time, you're gonna have to go with your gut.

...But since we both like flowcharts, I made one for you:

Question Triage Algorithm flowchart

...Just don't take this too seriously - you're better off using your own experience coupled with the guidelines given on the review page. If you're ever unsure, Skip and let someone else decide.

Background

It might help you to understand that this three-paths method of reviewing potentially-poor questions has already existed... For two years.

Your job is to read these posts and:

  • decide if a post does in fact Look Good
  • Edit the post to make it better
  • Delete the post entirely

Those options correspond roughly to the options in the new Triage queue. The problem is... Very few people choose the middle option:

Low-quality 
question review
Action Name        Reviews PctTotal 
------------------ ------- -------- 
Close              211554  44.31 %  
Delete             74      0.02 %   
Edit               6738    1.41 %   
Looks Good         74211   15.54 %  
Not Sure           172293  36.09 %  
Recommend Close    12489   2.62 %   
Recommend Deletion 54      0.01 %*   

Why is that a problem? Because there's an awful lot of stuff that needs some improvement, and yet a healthy % of it was ending up marked Good (where it then went on to annoy people and generate more flags) or Close (where it contributed further to the close review backlog.) Everyone's familiar with the false-positive "on the homepage, annoying people" problem... But in some ways, false-negatives are worse: because we were putting these questions in two parallel review queues (and sometimes the moderator queue as well!), they stood a better than average chance of being closed even if someone was willing to fix them - meanwhile, they sucked resources away from poor questions that managed to avoid the LQ queue, including false-positives. Ever wonder why there are so many posts here with folks complaining about their Very Low Quality flags being declined? Because, in spite of the fact that that flag was supposed to be reserved for the worst of the worst posts, it turned out that voting to close and flagging was a great way to get a question closed faster. Any question. Even questions that, realistically, weren't all that bad! Questions that were a bit ugly, short, or simple, but still answerable and most critically potentially useful...

Now... You might be thinking, "What's the harm? Closing faster is good, right? Who cares about a few false-positives? Plenty of sand on the beach..."

Truth is, there hasn't been a whole lot of harm, apart from some very annoyed moderators. See, we weren't really feeding that many questions into LQ review; our heuristics just weren't good enough to even identify the vast majority of problematic questions. When you're only processing about 3% of all questions asked, there's not that much harm you can do...

...But 3% doesn't do much good either. Go count the crappy questions on the homepage or your favorite tag page right now, and imagine you were shown maybe 2 fewer. Not really a dramatic difference, is it... We needed a system that would identify a lot more problematic questions.

So we built one. Actually, we built several - and we're not done yet. Right now, we're feeding roughly 22% of all questions asked on Stack Overflow into Triage review, and our goal is to run as many questions through it as y'all can handle in a timely manner.

But that comes at a cost: the new systems aren't perfect either. In fact, they have really high false-positive rates. We're hoping to get that down, but when you're talking 2K+ questions per day even a small error-rate adds up to a lot of questions... And what happens when you feed one error-prone system into another one that's already prone to making the same sorts of errors?

So we needed a third path. One that would allow y'all to correct the errors made by the automated systems instead of compounding them. An option for some of the reasonable but, uh, less-shiny questions that wasn't "Looks OK", and an option for some of the uglier but still possibly-useful questions that wasn't "Close/Delete".

We needed a grey area. And this is it...

14
  • 5
    Feature Request: Instead of 3 discrete options ("looks ok", "needs improvement", "unsalvagable") how about an analog slider? :D:D:D
    – Mysticial
    Commented Dec 4, 2014 at 21:51
  • 10
    Only if we can make the slider logarithmic.
    – Shog9 Mod
    Commented Dec 4, 2014 at 21:52
  • 1
    I apologize in advance for duplicating some of what you've written here in the answer I'm about to post, and have been working on for... longer than I'd like to admit.
    – Air
    Commented Dec 4, 2014 at 21:53
  • @AirThomas: You are forgiven... if you also have some new thoughts and perspectives. Commented Dec 4, 2014 at 21:57
  • Did you forget the footnote for Recommend Deletion 54 0.01 %* ? [there's an asterisk there].
    – user3717023
    Commented Dec 4, 2014 at 22:18
  • It's not that important, but... Recommend Deletion isn't supposed to be an option for questions. At some point, someone figured out that it was, and used it until we made it not work anymore. Note that Delete is a valid option, but only for moderators. @Raff
    – Shog9 Mod
    Commented Dec 4, 2014 at 22:21
  • 1
    You know... The more I think about what you've said in this answer, the more I believe any objection I have to being limited to three options, or using "should be improved" as a catch-all, is strictly growing pains. False alarm, everybody, no MS paint diagrams and enthusiastic rambling about the Likert scale today.
    – Air
    Commented Dec 4, 2014 at 22:34
  • @AirThomas you forgot Optimist and Pessimist. With Optimist rounding up, and Pessimist rounding down. Also Anarchist where you switch between Optimist and Pessimist every other time to keep your votes balanced.
    – Compass
    Commented Dec 5, 2014 at 0:15
  • 8
    I second @AirThomas. Using "Needs Improvement" as a catch-all isn't good enough. There needs to be a distinction between "by its author" (= put the question on hold) and "by the community" (= send to some edit queue to fix grammar, formatting, style, etc.). Commented Dec 5, 2014 at 13:07
  • 3
    I'm going to tattoo the flowcharts in my left arm Commented Dec 5, 2014 at 13:59
  • @Denis Actually, I was saying, never mind! It's not an issue! Since that wasn't clear, I'll be the wafflingest waffle that ever waffled and post the answer after all. Diagrams ahoy!
    – Air
    Commented Dec 5, 2014 at 17:15
  • /OT - Can someone explain to me the story behind that picture?
    – Compass
    Commented Dec 5, 2014 at 17:15
  • 2
    @Compass The picture links to the WP page, where it is captioned with a link that leads to the Montparnasse derailment page
    – Air
    Commented Dec 5, 2014 at 17:22
  • 1
    I think a lot of the confusion is because "Needs Improvement" explicitly says it's also used for when clarification from the author can fix the post, but the on hold/close system was also designed for that, yet close reasons are in the Unsalvageable button. Basically, the guidance for triage conflicts with the guidance for closing questions. Seems to me that "needs editing from author" should be moved to Unsalvageable and "other people can improve this" should be "Needs Improvement" - and your flow chart backs this up. Maybe update the triage review guidance?
    – Troyen
    Commented Dec 8, 2014 at 8:41
30

I think the new queue is great. Fantastic, even! I had some thoughts about the implementation, which I was going to post in the feedback thread, but this seems like a more appropriate place. (The incoming wall of text is not because this is a horrible, world-ending issue that kills the crab, but because I find the subject interesting.)

So here are the options as they currently stand:

Looks OK Should Be Improved Unsalvageable

I see this as roughly equivalent to a three-point scale assessing the overall quality of the question.

When I was in college I had a fascinating conversation with a psychology lecturer who taught survey design; one of the things she said was that people have an inherent bias away from either end of the scale. So when you want to assess whether people feel positively or negatively about something, a three-point scale tends to produce a strong bias toward the neutral response.

I don't think that's necessarily a problem in terms of where the question ultimately ends up – these questions have already "failed" the Q-score screening and Shog9 has been fairly clear (1, 2) that the expectation is for "Should Be Improved" to be a catch-all for borderline scenarios. But one of my strong impressions using this queue, which I think is echoed by some of the other feedback that's been given, is that it's not as fast as it could be. Sometimes you get a question that just really doesn't seem to fit on the three-point scale, but you do feel able to quickly and accurately assess it and you don't want to skip it. And even if you do skip, it takes some amount of extra time to decide to skip.

One of the other points from this conversation with the lecturer was that it can be easier for people to respond to questions asked on a five-point scale versus a three-point scale.1 We intuitively understand that lines have endpoints, a midpoint and regions in between, and we're comfortable mapping this concept to a five-point scale.2 Here's an illustration, using a Likert scale:

Five-point scale

When choosing to use a three-point scale, the naive survey designer may expect that we'll simplify our model by eliminating either the in-between regions or the endpoints. Either way, you get essentially the same result:

Three-point scale

Instead, what we tend to do is cling to that comfortable, finite-continuous model and map it to the three-point scale. But it's not as easy to do this when we have fewer options to map to:

Three- to five-point mapping

Now we have to spend more time thinking about the scale instead of the question. According to this instructor, generally speaking, we don't want to be fence-sitters, but even more than that, we don't want to be extremists. We act as though we're being personally judged, even when the responses are anonymous, and we tend to be more concerned about nuance when taking a survey than we expect people will be when we're designing surveys because we have different motives. The survey designer ultimately wants to answer a yes or no question, while the respondent wants to express their personal opinion, and maybe also give "good" answers. In the review queues, there's definitely an incentive to figure out the "right" answer – go against the majority too often and you get the whip. Failed audits! Drama on meta! Accusations of robo-reviewing!

Three- to five-point mapping

So the survey designer spends a bunch of time thinking about their survey, and comes up with a three-point scale that they understand, and expect other people to understand. Because they're soliciting opinions, they might want people to be very opinionated; more likely they've just developed their own "feel" for the scale and haven't given it much thought. All that response style bias will smooth itself out if the sample size is large enough, right? No big deal.

What she suggested to me was that a clever thing to do is to design the survey with more options, to maximize how comfortable people are expressing an opinion one way or the other when they're not truly ambivalent; and then map the responses after the fact back down to your desired three-point scale. In other words, drop the strongly/somewhat qualification and look only at whether people agreed, disagreed, or were neutral. Everyone has seen this happening, whether they know it or not, in PR copy that says things like "X% of customers were satisfied with the service they received." Odds are, those customers were asked to rate service on a scale with more than three points.

This was fascinating to me, because it's so counter-intuitive to think that people would take less time and be more comfortable choosing between more options.3 You do lose some apparent measure of the intensity of the response by doing this but, practically speaking, you shouldn't expect to get a consistent, stable measure of response intensity from this type of scale in the first place. Nor would the three-point scale you started with have measured intensity at all.

The analogy only goes so far; review queues aren't opinion surveys and the actions available to us aren't responses on a Likert scale.4 Also, I should point out that there's nothing special about a five-point scale – it's just the example that was used in the original conversation. We could be talking about any scale with more than three points (although even vs. odd has its own implications).

So here's the point of all this blah, blah, blah. In testing the Triage queue, I've had the same feeling that you and davidism have expressed, that the options we're given may be too limited. I've felt like I'm being asked to rate questions on a three-point scale from "Awesome" to "Average" to "Horrible" but my first impression often doesn't fall into any of those buckets. As a consequence, while I can evaluate items very quickly in the triage queue, I find myself sometimes spending an unreasonable amount of time translating my evaluation into one of the three available options. There's some overhead involved in that mapping step, that I find makes reviewing in this queue a little less comfortable than it has the potential to be.

Like you, I don't think "just skip it" is a great solution here. It takes about as long sometimes to fail to decide between the given options as it would take to choose one, when the question is borderline.

There's a good chance that this is just growing pains; as we get more comfortable with the triage queue, we'll all get better at making quick and painless decisions. But it's worth thinking about whether expanding the options could make the queue function better from the reviewer's perspective. One of the uncertainties about using "Should Be Improved" as a broad catch-all is that there's a difference in how we perceive and approach questions that need the author's intervention to fly, versus those that just need some love from a capable editor. And there's a difference in how we handle them, ultimately, based on whether the author puts in the work or not. I read this comment by Shog9 as saying that, since questions that are quickly closed rarely escape deletion, there's not a significant enough difference in expected outcome to justify distinguishing between "unsalvageable" and "author must edit" – but this isn't necessarily about changing the outcome for the author of the question. It's about the usability of the queue for the reviewer.

So, here's an example of the sort of alternate interface that some people have requested:

Looks OK Should Be Improved Needs Revision By Author Unsalvageable

Behind the scenes, this doesn't need to change where the post actually goes. "Looks OK" and "Unsalvageable" will probably still lead to the homepage and a flag/close review queue, respectively. "Needs Revision By Author" and "Should Be Improved" could both go into the helper queue, to begin with; but as we get data about how people are actually using the triage queue, it might make sense to divert "Needs Revision By Author" to flag/close review instead. Or send "Should Be Improved" to the homepage to sink or swim.

Would these four options be better than the three current options? Would any four options be better? I don't know. This is a question that requires data – and a clear metric for "better" – to answer. It may or may not be worth investigating; that's up to the good folks who Make Everything Work around here to decide.

Anyway, I hope this was at least moderately interesting food for thought. I went back and forth a few times about whether to even post it after Shog9's answer, because he's not wrong – the three options are a great idea. Personally, I think the four options are a good competing idea, and I'd be interested to see whether people use them differently.

I'll spare you the annotated bibliography for this post; suffice it to say that Google will bury you in literature about N-point scales, should you so desire.


1 Anyone who has studied survey design will start squirming in their seat when they read this; at the very least, I'm opening a huge can of worms by even bringing up the relative merits of different scales. There's tons of research on this subject and no clear consensus; some studies have concluded a seven-point scale is best, or a two-point scale, or even a 100-point or analog scale. My layperson's take on the subject is that there are different implications of different scales; keep reading, and I'll get to the point.

2 In hindsight, there may have been an unspoken cultural context to this conversation. Different cultures can have fascinatingly different perspectives and ways of thinking; in any case, it was a relatively casual conversation, so take it for what it's worth. When I sent her an email to confirm my recollection of the conversation, I also asked about this; she responded:

I have no idea if this is 'American' / Western or 'Human'. What an interesting question. We think about Response Sets and Heuristics as being 'humans think this way' -- but so much of the psychological research has been done on American undergraduates, it is possible that these are cultural not innate.

3 On the other hand, maybe it's not that counter-intuitive; you wouldn't ask, "What's your favorite color?" and provide only cyan, magenta, yellow and black as acceptable responses.

4 "Should Be Improved" is the middle option in terms of how the interface is laid out, and also in terms of judging the question on a scale of overall quality. But in terms of taking action on the review item, "Looks OK" could be considered the middle because it doesn't lead to further review, and is in that respect "neutral." And what if there's no middle option at all? Bye-bye, Likert scale analogy. It was fun while it lasted.

3
  • 2
    I will finish reading this, but you still haven't added the Optimist, Pessimist, and Anarchist voting styles q.q
    – Compass
    Commented Dec 5, 2014 at 17:15
  • @Compass The point was less to enumerate/categorize all styles of response, than to illustrate that there are many ways to slice this pie. You could consider those subsets of "Nuanced"
    – Air
    Commented Dec 5, 2014 at 17:18
  • 2
    Dear heaven.... This is sooooooooo true. It makes so much sense. I find myself wavering on what to actually pick so often. Part of the problem, too, is that the sample size is actually very small for reviews. None of them take more than 5 agreeing reviews, right? You could take it a step further, I imagine. Give us more choices and then base the action on a total score. As in have an "Unsalvageable" and a "Looks Good" threshold, with inbetween representing the "Need Improvement" area. Additionally, speeding up the review process would allow for a larger reviewer sample; wouldn't it?
    – jpmc26
    Commented Dec 6, 2014 at 3:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .