332

tl;dr: there's a new review queue. It'll be getting somewhere around 1-2 questions per minute. The only thing they have in common is that the system is unsure of what to do with them. Some are great, some are awful, some are in-between. We need you to help the system decide which category these questions belong in. If this works, we'll be building a whole lot more stuff on top of it: better questions on the home page, more help for promising users, less of a close-queue backlog... So let's test it!

We need your help separating questions into categories.

Background

Stack Overflow gets a huge number of questions per day. The best get answered. The worst get closed and deleted. However, there are a huge number of questions that aren't bad enough to be removed but... still aren't great. Many of these still get answered, and could be turned into exemplary questions... if anyone cared enough to fix them. But whether or not they get answered or edited is largely a matter of luck. In fact, luck has become entirely too much of a factor for even well-asked questions - the volume is too great to easily sort through, and many users depend on either the homepage's automatic filtering or complicated tag filters.

For a long time, we've been trying to find a more automated way to categorize questions when they're asked rather than requiring each and every question to be moderated. It turns out this is really hard. Ben Collins took another crack at this recently, and the results are pretty good... But not perfect. The algorithm still needs help. And that's where you come in:

The goal for Triage

A prepared reviewer at work

The term “triage” originated in battlefield medicine, describing a process for quickly sorting the wounded into three groups:

  1. Those who are likely to live, regardless of what care they receive;
  2. Those who are likely to die, regardless of what care they receive;
  3. Those for whom immediate care might make a positive difference in outcome.

The goal is to make the most of limited resources. And that's what we'll be doing with Question Triage, sorting questions into three groups:

  1. Those that will likely be well-received by the community and obtain answers with no further structured review.
  2. Those that will likely be poorly-received by the community, closed or deleted regardless of how much assistance they’re given.
  3. Those for which additional guidance or revision might result in a positive reception and useful answers.

Behind the scenes, a "quality score" is calculated for each question based on an automated analysis of the content. Those that score well are sent immediately to the homepage; those that score poorly will now be sent to Triage. From there, they'll go to one of three places based on human input:

  1. The homepage, where they can be answered
  2. The close or moderator flag queue where they can be reviewed and eventually deleted
  3. A new "Help and Improvement" queue where they can be edited

The intended lifecycle for a question will look something like this:

a flowchart, of sorts

However, first we need to make sure that works! In particular, before we start filtering questions from the home page or building a queue for helpful editors, we need to make sure the two pieces we've built already actually do what they're intended to.

So right now, there's no visibility restrictions or helper queue. After a few days, I'll come back with the results:

  • If it looks like things are working, we'll start dropping questions that don't "Look OK" from the home page.
  • If it's not working, we'll fix them. Change the guidance, the number of reviews required to complete a task, etc.

Once we're all happy with both the behavior of Triage and the appearance of the home page, we'll roll out a Helper/Editor workflow.

Go, review, & let us know how it works!

60
  • 11
    Is the Q-score that's compared to the threshold based entirely on an analysis of the question text, or does it incorporate stats about the user as well (whether they have a history of asking good/bad questions)?
    – Servy
    Commented Dec 3, 2014 at 22:38
  • 60
    It would be nice if we could directly edit a question ourselves and then click "Looks Okay" once we've improved it. Right now "Should be improved" simply whisks it away to a new queue when we could have just taken care of it right then and there.
    – JTG
    Commented Dec 3, 2014 at 22:41
  • 5
    @JTG There's the link to the question for editing, but it interrupts the flow of reviewing.
    – Celeo
    Commented Dec 3, 2014 at 22:42
  • 32
    @JTG Keep in mind the idea of Triage is for the actual triaging to be very fast. If you want to be spending time fixing up posts you should simply be in the NI queue. The entire goal of the triage queue is just to get the posts in the right place.
    – Servy
    Commented Dec 3, 2014 at 22:43
  • 3
    How many votes for a particular action in the Triage queue does it take for a given action to be taken?
    – Servy
    Commented Dec 3, 2014 at 22:47
  • 25
    If the question author needs to edit (to add information), does the post need improvement or is it unsalvageable (off-topic -> unclear what you're asking)? It's not really "unsalvageable", but putting it in a "needs improvement" queue won't help because the author won't see it there. Commented Dec 3, 2014 at 23:08
  • 5
    I also notice that twice after a review I've been given a breakdown my assignment and others' assignments of the question, with user names. What is that and is it supposed to show after every review?
    – Ajean
    Commented Dec 3, 2014 at 23:17
  • 18
    I think the guidance on "Unsalvageable" should be changed. An action such as closure doesn't (under my understanding) mean its unsalvageable, just that it needs a lot of work before it can be answered. Without the ability to comment, the user will just have his question randomly closed without even getting info from the NI queue. Commented Dec 3, 2014 at 23:19
  • 5
    Looks good, I would like to be able to tell a few more options like "needs formatting" "needs to show effort" and a "awfull incomplete/uncomprehensible (but still can be improved)"
    – eckes
    Commented Dec 4, 2014 at 0:19
  • 9
    This looks pretty great but I have one major problem with it, pretty much summarized in this answer. Not letting those of us who can and/or care to fix it right now, when possible, is terribly inefficient, so bad that I really am not looking forward to using this queue. I shouldn't have to open it in a new tab or hunt it down in another queue if I can fix the issue here and now. Not letting me or someone else is just a terrible idea IMHO.
    – Seth
    Commented Dec 4, 2014 at 1:35
  • 14
    @Seth If you want to spend your time actually fixing posts, rather than determining which posts need fixing, then you just shouldn't be using this queue and you shouldn't feel bad about that in the slightest. If you feel your time is better spent in the NI queue actually editing the known-problematic posts, then just do that. I'm sure that will be the queue with far more demand for quality reviewers than this one, as it's simply more difficult and time consuming work. The whole point of having a bunch of review queues is to allow people to do the type of review work that suits them best.
    – Servy
    Commented Dec 4, 2014 at 2:44
  • 8
    This post needs M*A*S*H metaphors. Because reasons.
    – user50049
    Commented Dec 4, 2014 at 3:22
  • 8
    @Seth You stated that the queue is very inefficient at doing something that it wasn't designed to do. Yes, that's true, strictly speaking, but that's because you're trying to use the queue in a way it wasn't designed to be used. Making it better at doing what you want it to do makes it less efficient at doing what it is designed to do, namely sorting questions into the correct buckets. If what you want to be doing is editing questions then you should be in the NI queue. I'm sure you'll find that, when released, that it's very efficient at accomplishing that goal.
    – Servy
    Commented Dec 4, 2014 at 3:59
  • 24
    I'm assuming that you intend for us to do this while drunk. I accept your mission.
    – user1228
    Commented Dec 4, 2014 at 16:12
  • 3
    And if you choose those options @Simon, someone may even try to salvage those questions... in something like 13% of cases.. Realistically though, if you close the question then the odds of it being reopened vs. being deleted are very slim. No reason not to give folks one more chance, but if it isn't a trivial fix then it's probably not gonna happen. If it is a trivial fix, consider "Needs Improvement" instead...
    – Shog9
    Commented Dec 5, 2014 at 0:31

25 Answers 25

123
  • When an item is marked as needs improvement and gets edited, it should go back to triage, not to the homepage.

    Lots of people like making small trivial edits. Perhaps someone will property format the code, or add a new tag, or fix a few of the problems. This may help, but it won't necessarily mean that after the edit the question is completely done. It should go back to triage to verify that it doesn't need any further editing

  • Since Triage is effectively how the VLQ flags will be handled for questions, the VLQ flag on an item currently sitting in "needs improvement" shouldn't push it back to Triage. It should just do nothing, since it's already in the queue that handles those flags. It shouldn't push it out of the needs improvement queue.

    A flag for closure (or maybe 2+ flags/votes?) should push it directly to the Close queue, and pull it out of the Triage/Needs Improvement section

  • It'd be nice to be able to see, when viewing a question, if it's in the triage queue, and any past Triage actions taken on it. It doesn't need to be prominent (perhaps as entries in the revision history?), and it should likely have some sort of rep requirement, but it'd be nice to see.

9
  • 13
    This is only for questions. And if it works, it'll replace the LQ review queue for questions, since, quite frankly, the LQ review queue does not work very well for questions. The flag for disputing triage results will be VLQ.
    – Shog9
    Commented Dec 3, 2014 at 22:26
  • @Shog9 okay, that makes a lot of sense; I'll edit the last point accordingly
    – Servy
    Commented Dec 3, 2014 at 22:27
  • @Shog9 So the third point indeed becomes effectively moot, that was me misunderstanding your intent, the second point has been re-worked accordingly, as that workflow rule still seems off.
    – Servy
    Commented Dec 3, 2014 at 22:33
  • The first point is especially strong since we have "no improvement" instead of "not enough". Commented Dec 3, 2014 at 22:39
  • @Deduplicator Very true, although I would expect that most (if not all, depending on what the rep requirement is set at, Shog, any comment on this) users looking at items in the NI queue to have full editing privileges.
    – Servy
    Commented Dec 3, 2014 at 22:40
  • @Shog9 BTW, on that earlier point, maybe the Low Quality Posts review queue should be renamed to Low Quality Answers when all of this is sorted out, if we're pulling questions out of it? Minor point, I know.
    – Servy
    Commented Dec 3, 2014 at 22:42
  • 1
    @Servy: Better Very Low Quality Answers, to emphasize that bad does not equate bad enough to kill. Commented Dec 3, 2014 at 22:45
  • The rep threshold is the basic access review queues privilege.
    – Shog9
    Commented Dec 4, 2014 at 4:19
  • 1
    I strongly agree with Servy's first point. This is mostly why recently I've just given up on looking at suggested edits anymore. Most of the time it's a half-a*s job and I'm always clicking Improve. Lots of people when editing don't edit everything Spelling, Grammar, Formatting, Tags, Breaking Walls of text..etc
    – gideon
    Commented Dec 30, 2014 at 7:16
86

It's unclear to me what the cutoff point or difference is between "Should Be Improved" and "Unsalvageable". When I think something should be improved, usually it's because there's an applicable close reason. However, the close reasons are located under "Unsalvageable".

Right now, "Should Be Improved" isn't clear on who should be doing the improving. An editor? Then there should be an edit button. The author? Then there should be close reasons as well as comments to guide the improvement.

"Unsalvageable" should also include close reasons, since some "questions", such as "give me teh codez", can't be improved, but it shouldn't be the only route to the close vote option.

Another option is to break out the choices into a few more buttons that have clearer instructions. This separates closing questions because they can be improved by the author from truly bad questions that we all know are inappropriate for the site. "Good Enough", "Needs Editing by Anyone", "Close, Author Needs to Edit", and "Unsalvageable, Should Be Deleted".

9
  • 15
    Have to agree with this. I find myself gravitating towards "Should be Improved" or "Skip" for almost every question. Some questions are close to unsalvageable but could be improved. Whereas some questions are pretty close to "good", but still could benefit from improvement. I find a big variation in question quality between 2 different questions which I've flagged as "should be improved", yet neither is totally unsalvageable. Commented Dec 4, 2014 at 2:30
  • Yes, I think that definition will have to be refined and more clearly specified. I stumbled over the queue before coming here, and clicked on it just to try it out. The first question I saw was quite bad (way too little detail), but I didn't click "unsalvageable", because it wasn't by the definition of the word. It could totally be salvaged by adding a ton of information. Commented Dec 4, 2014 at 8:07
  • 45
    In my opinion, if a post is unsalvageable, it shouldn't be closed, it should be deleted. I think there should be 4 buttons: "looks ok", "needs improvement (from anyone)", "close (because OP must edit)", "unsalvageable - this needs to be deleted".
    – gunr2171
    Commented Dec 4, 2014 at 13:43
  • 1
    This is one of the questions I was in the gray area about. I voted UN: (stackoverflow.com/review/triage/6387183). Should I have voted SBI?
    – Compass
    Commented Dec 4, 2014 at 14:40
  • @Compass Since the meaning of upvoting your comment would be ambiguous: I think you voted correctly.
    – Air
    Commented Dec 4, 2014 at 16:43
  • 5
    Closing is effectively a precursor for deletion. In particular, questions closed quickly are very likely to be deleted automatically. Note that "unsalvageable" also covers Spam and Offensive cases, @gunr2171
    – Shog9
    Commented Dec 4, 2014 at 22:49
  • 2
    I think the fundamental misconception here lies in the word "unsalvageable" - what is actually meant is that the post "requires moderation" (as it was put in the tavern). similarly the wording for "looks ok" could use an improvement to differentiate it from "VLQRQ looks ok" that could also use a better wording.
    – Vogel612
    Commented Dec 5, 2014 at 10:35
  • 1
    Upvote, the new helper-que proves (to me) that this would be needed. I think that questions that require improvement by the asker FIRST should not be in the que for anyone to edit (especially if the reviewers in the triage-que already spent time to come to this conclusion). Once the Asker has provided the required additional info, the post should be sent back to the triage-que (where we loop the same logic: 'needs improvement by op', 'needs improvement by anyone', 'is ok', 'un-salvageable', etc).
    – GitaarLAB
    Commented Mar 6, 2015 at 20:18
  • "looks ok" "needs improvement" "has problems" (leads to chooser) "unsalvageable" (just delete, or a deletion-related chooser) "skip" Commented Sep 8, 2015 at 20:11
61

The upvote dialog is just nonsense; it should be removed.

What's the rationale about "we strongly encourage you to upvote it"? Just because a question meets the site's standards, it doesn't necessarily mean it is a good question.

Furthermore, it is very likely that the reviewer lacks the technical knowledge needed to determine whether the question is good or not, because they don't understand the actual content.

Stack Overflow is a Q&A site about programming, not about rhetoric.

To receive an upvote, a post should meet the site's standards and contain good technical content, which gets the upvoting individual curious and interested and/or benefits the community as whole. In particular, questions that seem unique and have not been asked many times before deserve upvotes.

This upvote dialog also kind of suggests that "we are so used at horribly poor questions on this site, that if you come across something of mediocre or above quality, immediately upvote it no matter the contents!"

3
  • 3
    We're discussing changes to this (at minimum, I should be explaining why we're encouraging upvotes in this scenario). FWIW, roughly 40% of reviewers shown this option have upvoted thus far.
    – Shog9
    Commented Dec 4, 2014 at 17:31
  • 1
    We've updated the guidance to clarify why that UI even exists, and also changed how we're emphasizing the availability of voting in that scenario (no more redundant buttons).
    – Shog9
    Commented Dec 4, 2014 at 19:28
  • 2
    I agree with the upvote prompting being nonsense. Most of the questions I vote "Looks OK" on are not ones that I would upvote normally. For example, I will vote "Looks OK" for a question that is on topic, well-reasoned and properly formatted but that could be answered by reading the relevant documentation. So the question certainly doesn't need to be improved, but I'm not going to upvote a lazy asker. In full disclosure, I tend to look to SO for stuff I could look to documentation for (e.g. how to do most complex things in git :)).
    – eebbesen
    Commented Jan 8, 2015 at 3:36
38

Given that the questions in this queue are in that murky area of "not sure if it's a bad question or not", then I would think that only users who have good knowledge of the question's field will be able to make a call.

So, can the queue be filtered so that we only see questions that have a tag matching one of our favourites? Otherwise users will (should) be hitting the Skip button most of the time.

11
  • 5
    Even if those aren't the only questions a reviewer sees, those are the ones that should float to the top.
    – Sneftel
    Commented Dec 3, 2014 at 23:17
  • 10
    I just tried working the new queue, and most questions were "skip" because intelligent evaluation required knowledge of some package, language, or system I don't know about. Commented Dec 3, 2014 at 23:38
  • 4
    I'm not sure about that. When I went through the queue, I was looking for things like well structured sentences, proper use of grammar, high level ideas, proof of attempt, code samples and/or diagrams, etc.' Commented Dec 4, 2014 at 0:38
  • 1
    @Qix If it's general criteria like that, then I agree that you wouldn't need to be a SME to make a call.
    – DeanOC
    Commented Dec 4, 2014 at 0:42
  • 4
    If a question asks for debugging help but doesn't provide code, the error message/incorrect behavior or the desired behavior, it doesn't really matter what language it's tagged with. Commented Dec 4, 2014 at 0:52
  • 3
    @Qix What about when they provide some information, but you're not sure if they provided enough information because you aren't familiar with that technology?
    – Troyen
    Commented Dec 4, 2014 at 1:14
  • 2
    @JeffreyBosboom What would you do with this question? I had to skip because I don't know if that's enough info for people in that tag.
    – Troyen
    Commented Dec 4, 2014 at 1:16
  • 2
    Skip is always a reasonable choice. I figured we'd want filtering here eventually, but currently this is just a proof of concept - if you were filtering, there's a good chance you'd spend a lot of time waiting for items in your tag to show up.
    – Shog9
    Commented Dec 4, 2014 at 4:17
  • Taking a look just now, it seemed a lot of the questions were really terrible. A few did have sensible amounts (by volume) of information about things that I didn't have a clue about, so I skipped them, but not that many of the questions in this queue look like they need specialized knowledge
    – chrisb2244
    Commented Dec 4, 2014 at 7:06
  • @Shogi - having reviewed half a dozern the filter should be to ignore some tags not necessarily to include them - I have more tags that would interest me than those that just turn me off - e.g. out of 6 reviews 5 were PHP - if that ration continues then I wan't bother - most I think are rubbish but of the others I don't know if they are reasonable PHP questions
    – mmmmmm
    Commented Dec 5, 2014 at 16:34
  • 1
    When you're looking at a PHP question, the odds are strongly against them being reasonable, @mark.
    – Shog9
    Commented Dec 5, 2014 at 23:23
31

Not having seen the "Needs Improvement" queue, this request may already have been taken care of.

However, my very first review in this queue was a "Needs Improvement" question, but not one that can be fixed by an external editor. In this case, as in many Stack Overflow questions, the requisite code was missing.

The problem statement was clear, and the scope narrow, just this one (very important) piece of information is missing! An external editor can't add it, and I couldn't add a comment.

Could we allow comments, or perhaps a pro-forma one like that in VLQ ("Your post needs improvement, your question will not get visibility until you...") or something like that? I get that we are trying to triage quickly but if the focus is on other users editing in the "Needs Improvement" queue, I can see many questions being stuck in limbo.

Also, the "Unsalvageable" guidance seems a bit off. Closing a question just means it needs a lot of improvement, not that it can't be salvaged at all (unless we are lying to ourselves about the whole "On Hold" thing). Without the ability to comment, a user who wishes to VTC will likely cause the question to be closed without ever getting a "You need to do..." comment (since it wouldn't hit the NI queue).

Note, there is no "Add comment" button on the review page for me. See this screenshot:

Enter image description here

12
  • 2
    The current plan for Needs Improvement would fit that case well. One of the possible main actions we expect there is a "Comment" option, generally for cases where additional information is needed or would help a lot, but the OP needs to be told what's missing. (I'm not against having some ability to comment in triage, but one of the key ideas here was to ask people to bucket without the distraction and slowdown that comes from feeling like you need to take the needed action yourself...)
    – Jaydles Staff
    Commented Dec 3, 2014 at 22:57
  • @Jaydles Okay, problem solved :) Commented Dec 3, 2014 at 22:58
  • Un-deleting so other people can see that this is taken care of. Commented Dec 3, 2014 at 23:13
  • 1
    Is this answer intended to cover the same ground as your comment above? It doesn't really read that way to me, but I share the concern of your comment and want to avoid posting a duplicate answer. Specifically, I was very surprised that the flag dialog came up when I hit the "Unsalvageable" button and not the "Needs Improvement" button.
    – jscs
    Commented Dec 3, 2014 at 23:46
  • @JoshCaswell I posted this before my comments above. I deleted it (since I felt it was responded to/taken care of) and undeleted when I saw your comment (since others obviously share my concern). Sorry for the confusion! It does not cover the unsalvageable guidance piece. I suppose I should edit to include that... Commented Dec 3, 2014 at 23:48
  • Gotcha; I'll hold off on posting then.
    – jscs
    Commented Dec 3, 2014 at 23:50
  • @JoshCaswell Just edited. I'd be happy to take any input from you, or you can post an answer to cover areas I haven't :) Commented Dec 3, 2014 at 23:50
  • Note that nothing stops you from adding a comment before picking an action here. I don't know if there's a place for canned comments, but if you have something useful to say, say it!
    – Shog9
    Commented Dec 5, 2014 at 0:13
  • @Shog9 Are you referring to adding a comment by going to the linked post? The "Add comment" button is not available. Commented Dec 5, 2014 at 0:18
  • You don't see "add a comment" below the question? I see it...
    – Shog9
    Commented Dec 5, 2014 at 0:20
  • @Shog9 Added screenshot showing no add comment link Commented Dec 5, 2014 at 0:23
  • Dalgas is fixing.
    – Shog9
    Commented Dec 5, 2014 at 0:40
26

The "Unsalvageable" button seems to work exactly like the normal "flag" link on the post itself. If you have a review item open and it has already been closed by the time you get to choosing "Unsalvageable", you get the flag dialog minus the close options.

The same thing, with the addition of the VLQ option, appears if I myself have already voted to close outside the queue.

This is weird, or at least unexpected, behavior. I'd figure on getting a message along the lines of

This has already been handled/closed. If more needs to be done, visit the post and flag it.

Similar to what happens when you finish a suggested edit review too late. Presenting the dialog lacking the option to do what I'd typically do in this queue, without explanation, is confusing.

2
  • Seems the close reasons have been added (under "for another reason"), this might have been updated. Though I don't like all the clicks it takes to mark something as "unsalvageable". It should be a one button press that says "this should be closed", or another top-level flag reason.
    – gunr2171
    Commented Dec 4, 2014 at 1:00
  • 3
    Note that this only comes up when you've had the review item open for a while and it's already been closed by others, @gunr2171.
    – jscs
    Commented Dec 4, 2014 at 1:26
23

I am missing the "edit" & "leave a comment" buttons in particular.

Often posts are okay, except that the code isn't in a code block. This is so easily fixed and there's almost never a reason not to do it when you spot it. This is often all that's needed before it "Looks OK".

Similarly, leaving a comment is often useful; half the time this can be as simple "We can't debug code we don't see, you need to post your code :-)"; why not just leave a comment when you're at the question anyway?
I know this is probably not what "Triage" is intended for, but I already spent time reading & comprehending the question, so why not immediatly add a simple comment with a suggestion on how to improve it while you're here anyway, instead of having someone else read & comprehend the question again?

But in general, looks cool :-) Looking forwards to the results.

7
  • 7
    Yes I agree. Sure triage is meant to be fast. Those who don't have time and/or just want to be fast can blow through it, but let those who have the time and/or actually care fix it right now. Absolutely zero point in making us open them in a new tab or hunt them down in another queue if we could have fixed it on the spot. Do leave all the options available after editing though, sometimes a question needs an edit and more improvement, maybe even a comment too.
    – Seth
    Commented Dec 4, 2014 at 1:21
  • 7
    We need the ability to comment in order to let the user know what they need to improve Commented Dec 4, 2014 at 1:58
  • 3
    I understand the intent is to not do anything, but to just push it through to the right queue. However, if there is a question that hits me for some reason, and I really want to do something about it (comment, edit, etc.), then what will be the process? Will there always be a link so you can go to the question in a different "view"?, or are triage questions only available in the triage queue? I think it's important for there to be some way to take action--not in the triage queue directly, but some way to break out of the queue for a question that you really want to do something about. Commented Dec 4, 2014 at 3:25
  • @aliteralmind Well in all other queue's there is the questions title that will take you directly to the question.. I suppose it will be the same here.
    – Seth
    Commented Dec 4, 2014 at 3:45
  • @aliteralmind There's a link to the question, and you can still do whatever there; I did that yesterday a few times, it's just not very fast. Commented Dec 4, 2014 at 8:37
  • I'm not getting the final form here. A link to a question in the triage queue CURRENTLY goes to the question as it exists in the main displayed-to-everyone view, but this is only its beta form. Ultimately, these questions will only be in the triage queue, right? If that's the case, then where will these links go to? What view/queue will that editable question be in? Commented Dec 4, 2014 at 13:01
  • 1
    We're not gonna make questions impossible to view, @aliteralmind - we're just not going to push them in front of nearly as many people. If you give someone a link, or go trawling through the new questions lists, you should still be able to view a question until it is deleted.
    – Shog9
    Commented Dec 5, 2014 at 15:03
20

Please rename the unsalvageable option

After doing a few reviews and observing the queue, it looks like the label "unsalvageable" is being confused with the VLQ flag's terminology of salvaging.

Can it be renamed to something which doesn't coincide with any pre-existing term on the site, like "Needs moderation"?

9
  • 11
    Looks OK | Needs a Doctor | Unlikely to Live
    – user3717023
    Commented Dec 5, 2014 at 13:33
  • What do you mean confused? They mean the same thing...
    – Shog9
    Commented Dec 5, 2014 at 15:00
  • 1
    @Shog: If a post needs closing as POB or tool-recommendation, then it is off-topic and hence I need to choose unsalvageable->close. In this case, it isn't gibberish as defined in the vlq flag's unsalvageable definition. In some reviews, users are selecting "Should be improved" instead of closing the post as "unclear", because it isn't gibberish. Commented Dec 5, 2014 at 15:34
  • 1
    @Shog9 Someone said: if you're flagging things as VLQ you don't honestly expect to make the mods reflexively reach for the delete button (while throwing up a little bit), you're probably doing it wrong. Is this the standard for Unsalvageable in the triage?
    – user3717023
    Commented Dec 5, 2014 at 15:39
  • That's the standard I've been using, @Warm. Well, that & duplicates - those are always a weird exception. YMMV, of course - the big problem we've had with VLQ has been that folks have very different ideas of what is/isn't salvageable. But yeah - get rid of the worst stuff ASAP, hide the bad but not doomed stuff until/unless it gets fixed: that's the point of Triage.
    – Shog9
    Commented Dec 5, 2014 at 15:43
  • 3
    @Shog9 How does the concept of on-hold fit into this? It was supposed to mean that the state of the patient is not yet terminal. Yet the path to on-hold goes through Unsalvageable.
    – user3717023
    Commented Dec 5, 2014 at 15:47
  • 4
    @Shog: But please rename the "unsalvageable". Users are getting confused. I proposed this after revisiting the reviews I did earlier. Except Mysticial and I, other reviewers seem to be using "needs improvement" as a synonym for on-hold (where the OP needs to add more information to improve the post, and an editor can't do anything to salvage it). Commented Dec 5, 2014 at 15:50
  • 2
    @Infinite Because that's what the queue instructions say: "Should Be Improved for questions that would benefit from further revision by the author or others". Yet the close votes are on the "Unsalvageable" option.
    – Troyen
    Commented Dec 5, 2014 at 23:56
  • 2
    I agree with this comment but would propose: Looks OK | Needs an Edit | Flag This Item Commented Dec 15, 2014 at 14:38
18

As things stand, I'd be willing to wager that this experiment will fail miserably. Primarily for four reasons.

First and foremost, a pig with lipstick is still a pig. Users don't like reading SO's front page once they've decided it's an ocean of trash. I'm therefor at a loss as to why they'd suddenly want to go through the very worst of the trash they're actively avoiding, for the mere reason that it's presented in a slightly different format -- and without the appeasing satisfaction of rabidly casting down-votes to boot.

Secondly, and as a result of the this, potential reviewers have absolutely no incentive to dive in this cesspool.

Thirdly, it's evident in the queue's history that quality standards are very different from one reviewer to the next. Some questions, which I'd rate as objectively "Get out of my face" unsalvageable, apparently look OK for other reviewers.

Lastly, and in stark contrast with the low quality post queue where mindlessly closing typically is the obviously correct answer (even when you know next to nothing of what's being asked), these questions often need to be read, and sometimes even understood, in order to make an educated assessment of how bad they are.

Only show what the reviewer is comfortable evaluating

This last reason is easy enough to fix: only show posts tagged with either of one of the reviewer's favorite tags or one of the tags in which the reviewer has a bronze badge.

Give more incentive to weed through the trash

With respect to the second and third reasons, I'd like to suggest that any "Unsalvageable" or "Needs improvement" vote cast on a question that ultimately gets published as "Looks OK" without modifications should automatically count as a down-vote.

If that is not acceptable, than at the very least give the opportunity to cast a downvote then and there.

If it is acceptable, a further incentive to make reviewers exhaust their review quotas would be to make it so that this downvote does not count against daily vote limits.

(As a side note, the same could be an incentive for users to visit the Close-Vote queue more often.)

Stop the trash at the gate

With respect to the first reason, much more must to be done in order to ensure that posters don't flood the site with junk to begin with imho, so as to stop the trash at the city gates rather than in the town center when it's all over your face.

In the flow chart, I believe extra steps are needed prior to new questions getting sent to the queue. The point would be to make it impeccably clear upfront that a question will not even get published, let alone answered, if it is too low quality.

Very low score questions

If a question's score is very low, it should get rejected outright with a notice:

Your question raised too many low qualify flags. Left as is, it will not even be reviewed for consideration.

Please take some time to review our guides on writing a good question if you haven't done so already, and amend your question accordingly.

It doesn't matter of there are false positives here. Authors of good questions with a very low score will expand on theirs a bit more and move on with their day. What matters is catching as many genuine positives as possible; this "very low score" needs to be high enough that posters of undesirable questions find it user hostile.

Duplicates

With respect to potential duplicates, some javascript should fill in a hidden field that increments a counter when OP visits URLs the system suspects are duplicates -- suspected dups where OP didn't even bother to read prior art should get a more significant penalty. Ideally, OP should be required to list at least one of these URLs within the question, and explain why the problem is different.

Low scored questions

If a question's score is better simply low enough, the form should stall the submission and notify the poster of potential problems:

Your question raised several of our low quality flags. If you leave things as is and re-submit this form, the question will be pushed to a triage queue.

The triage queue is where reviewers will decide whether you question fits our quality standards or not. When not, they may prompt you to improve your question or delete it outright, before it even appears on the site.

Before re-submitting this form, please take some time to review our guides on writing a good question if you haven't done so already, and amend your question if you feel it is necessary.

Offer an alternative to freeform questions

In each case, and perhaps for all questions or for some other set of rather low quality questions, the submission form should change from freeform to structured.

Currently, it is freeform and it stems from the database schema:

Title: [                   ]
[                          ]
[                          ]
[                          ]
Tags: [                    ]

It should get turned into something like this instead, in order to guide the user into writing his question:

Title: [                   ]
Introduce your problem in two or three lines:
[                          ]
[                          ]
[                          ]
Give additional details on your problem, if appropriate:
[                          ]
[                          ]
[                          ]
Describe how to reproduce your problem, if appropriate:
[                          ]
[                          ]
[                          ]
...
Tags: [                    ]

Add a quality meter

While we're at it, we could additionally expose some kind of question quality meter as a question gets typed:

Question Quality: [####                    ] (What's this?)
11
  • I disagree with the real-time quality meter (too easy to game the system for people who want to bypass it) and the duplicates suggestions. All other points I agree with.
    – gunr2171
    Commented Dec 4, 2014 at 17:21
  • @gunr2171: Imho, a quality meter would serve its purpose even if a user could potentially game it. I'd imagine it's less of a hassle to try to improve a question than it is to try to reverse-engineer and work around a meter. Commented Dec 4, 2014 at 17:34
  • "Add a quality meter" -- FWIW there is already a dedicated feature request for this: Add a “Magic 8-Ball” feature to the Ask a Question page
    – gnat
    Commented Dec 4, 2014 at 17:55
  • 2
    Regarding blocking "Very low score questions" - we've been doing this for years. It's not enough. It's not just good authors that expand and squeak by... Ironically, it's some of the worst authors who are the most motivated to get their questions posted by any means necessary (even if they never respond to feedback after that).
    – Shog9
    Commented Dec 4, 2014 at 18:23
  • Regarding duplicates: we already try to identify these and present them to askers. A huge number of questions don't get asked because of this... Looking into identifying folks who don't bother checking for dups as an additional signal.
    – Shog9
    Commented Dec 4, 2014 at 18:25
  • 6
    Better ask form is an extremely good idea. Quality meter is likely to be counter-productive: we're basing the score on various factors that correlate with low-quality questions, but eliminating those from a post do not necessarily make it good. However, we already provide some general guidance on writing a good question when a post scores exceptionally low, and could probably do more there.
    – Shog9
    Commented Dec 4, 2014 at 18:28
  • 1
    @Shog9: I foresee a slight risk, when monitoring whether a user clicks suggested related questions, that the suggested questions' titles reveal all that is needed (to a knowledgeable user) to dismiss them as irrelevant. But then, this knowledgeable user, I'd expect, will generally output a high enough quality that it doesn't land in a triage queue. If you've the tools handy, studying in-page metrics such as the presence of a long pause might also give a good indicator that a user is doing his research and at least considering the possibility that he's entering a dup. Commented Dec 4, 2014 at 18:46
  • "it's some of the worst authors who are the most motivated to get their questions posted by any means necessary" - I'm not entirely surprised, seeing how unsophisticated end-users with vague problems tend to be the most time consuming and insistent users support desks need to deal with. As an aside, and I hate to be the one asking this since it could be politically incorrect, but do we have any kind of metrics on whether these authors are concentrated in a specific geographical area or IP address ranges? Commented Dec 4, 2014 at 18:54
  • 4
    Some networks are worse than others, for various reasons. We had to ban an entire university at one point... Geographical areas are hit and miss. There are places that are particularly bad for things like spam and plagiarism, but again there tends to be a pretty high false-positive rate (also, geolocation is inexact at best). We are using known-to-be-problematic subnets to augment the content analysis right now though... Stay away from Comcast in Los Angeles.
    – Shog9
    Commented Dec 4, 2014 at 19:17
  • @Shog9: Doesn't that mean that net will plummet even further? (And I really hope that university could restore its reputation somewhat... Anyway, only users below a rep-threshold are blocked when coming from a blocked ip, right?) Commented Dec 4, 2014 at 23:20
  • Anyone can get unblocked if they demonstrate a willingness to stop doing whatever got them blocked in the first place, @Deduplicator. Also, there are different forms of "blocks" that affect different users and have different lengths.
    – Shog9
    Commented Dec 4, 2014 at 23:49
6

Per my observations, triage appears to be making a noticeable impact on close votes queue. This seems to be intended:

There were (and still are) a huge number of false-positives flooding the close queue because we were essentially giving folks only two options for handling these in the VLQ queue. Hence the value of that "grey area"...

Since triage queue was introduced, it looks like close queue started stronger prioritizing for certain kind questions. Top of the queue (items presented first to reviewer) seems to be much more filled with questions that are: 1) clear cut close-worthy, 2) recent (often posted within 1-2 hours) and 3) not yet answered.

  • Above was observed with my favorite filter setting (too broad). I also briefly tested this with filters off, the effect appears to be present there too but it was just less comfortable so I quickly dropped back to filtered review.

To me this change is beneficial, because mentioned priorities fit my own review preferences. As a result, I skip much less than before.

For reviewers with different preferences (eg older questions, or trickier cases etc), this may mean they would have to skip more to get to lower priority queue items.

2
  • ...have to admit, it feels pretty rewarding to vote close questions currently brought to top of the queue
    – gnat
    Commented Dec 16, 2014 at 15:03
  • (upon further thinking) regarding unfiltered review, I didn't like that it forced me to frequently flip between too broad and vague debugging questions. Per se, each of these kinds was easy to handle, but this only made context switching more noticeable and painful. I don't know, maybe one can get used to that; this time it somehow felt better than in the past
    – gnat
    Commented Dec 17, 2014 at 8:06
5

I have several questions and comments:

First, I hope this experiment works out well.

How should we handle questions that belong at another SE site? "Unsalvageable" (apparently?) puts the question into a mod queue, which creates useless work. "Needs improvement" and "looks OK" trigger the wrong flow. I'm marking them unsalvageable and closing them as "belongs elsewhere"; I hope that triggers the right flow.

Also, and this is quite minor: Can you make the "Unsalvageable" -> "wrong site" flow fewer clicks long?

"Needs improvement" seems to cast a pretty wide net. Should I use it for questions that are slightly ambiguous but may be answerable? Should I use it for questions that appear to have the kernel of an interesting question buried inside a mountain of crap, or do those get whacked with "unsalvageable"?

I recognise that it will be very, very hard to articulate a set of general guidelines that actually separate the lower fringes of "looks OK" from "needs improvement" and the lower fringes of "needs improvement" from "unsalvageable," but I've never gotten text classification to do anything useful without rigorously defining categories. I also recognise that defining categories based on something other than how crappy a given question is might lead to useless results.

What should I do with poorly-researched questions?

5
  • There was a long, mostly unproductive comment discussion here that has been moved to chat if anyone is interested in continuing it.
    – Shog9
    Commented Dec 4, 2014 at 4:04
  • 3
    Needs Improvement covers a vast area. There are a lot of questions that could stand to be improved; which ones need it depends on your perspective. This is intentional - the option is intended to provide a sort of guard band between the home page and the close / moderator queues. Note that while most unsalvageable questions should be closed, not all questions that can be closed are unsalvageable - you should make a decision based on your impression of the question, and then choose from the options available to you in the flag dialog.
    – Shog9
    Commented Dec 4, 2014 at 4:08
  • 1
    "What should I do with poorly-researched questions?" Same as always, down vote, close as duplicate if possible.
    – Lundin
    Commented Dec 4, 2014 at 7:52
  • @Lundin: That's not an option within this flow. I'd like to be able to "unsalvageable"->"this question does not break any rules, but is crap regardless" or "needs improvement"->"please make your question less hopelessly muddled; I think the kernel of it is XYZ."
    – tmyklebu
    Commented Dec 4, 2014 at 16:12
  • 1
    The purpose of this review queue seems to be the same as the low quality one: ensure that the question meets the site standards, without judging the quality of the content itself. To realize that a question is poorly researched, you most often need technical knowledge of the topic, which is usually not required to do these reviews. In the case where you have such knowledge and spot something bad, it is probably better to go to the question manually, out of the review queue, then take moderation actions from there.
    – Lundin
    Commented Dec 5, 2014 at 7:41
5

I think these queue items should require more reviewers.

When I tried it out, I found there were a number of questions where I wasn't sure which category to put them in. Optimism can be the difference between NI and U, and pessimism the difference between LO and NI. The one that I saw the results on passed with 3 "Looks OK"s and 2 "Needs Improvement"s. I guess it was questionable whether it was actually OK or not.

If it had been 3 "Needs Improvement"s and 2 "Unsalvageable"s, those people would have got their flags disputed as well.

It just seems to me that, since these are already unclear/borderline questions (albeit as decided by the system), it would be better to have more pairs of eyes on it.

You keep saying that the whole point of this queue is for it to be really fast, right? I think that since it's a fast queue you could require twice as many reviewers and not have it be a problem.

Edit: I've now seen 4-5 questions where I was the last vote on them and they all had 3:2 results. This has made me more sure that 5 reviewers is not enough for what are essentially already borderline questions.

7
  • 1
    the way to review fast (and responsibly!) is to never hesitate to Skip. There are tens thousands reviewers in this queue, someone will pick up what you skipped
    – gnat
    Commented Dec 4, 2014 at 14:35
  • @gnat Absolutely, but not everyone skips when they should. Commented Dec 4, 2014 at 15:23
  • 2
    @gnat Also unfortunately that doesn't really solve the problem I was bringing up, which is that the very nature of triage means lots of these questions will be borderline and the dividing line will be down to personal opinion. We basically have to guess which questions can survive on their own without intervention, which will die regardless of intervention, and which the intervention will make a difference for. And, sure, it can be an educated guess based on your knowledge of that area of expertise and on your experience with SO, but it's still inherently fairly subjective. Commented Dec 4, 2014 at 15:33
  • 2
    [contd.] Not necessarily an inherently bad thing, but I thought this particular queue might benefit from more eyes. That's all. Commented Dec 4, 2014 at 15:34
  • 2
    I think I understand. I still believe though that core problem is reviewers aren't "trained" to fallback to Skip when they feel it's getting too hard. I for one don't avoid difficult reviews mind you, I only limit their amount to keep it comfortable to me (so that I keep clear mind and good understanding of what I'm doing). If other reviewers would do so, that would be enough to handle borderline stuff. Think of it, 2-3 "difficult" reviews (4-5 in familiar tags) per session done by hundreds... thousands reviewers, that would be quite a lot
    – gnat
    Commented Dec 4, 2014 at 15:50
  • @gnat I definitely agree that reviewers not being "trained" to use Skip is a problem. And I kind of see what you're saying about it being related to this. I don't think it would completely solve the problem, though, because even rested reviewers with fresh eyes would find some of these hard to sort, I think. However, it might alleviate it a bit. Commented Dec 4, 2014 at 15:55
  • 1
    It has to be fast because we're throwing a ridiculous number of questions into it. That said, it's also important that the results are accurate - so we may adjust the number of reviewers required over time. At present, between 3 and 7 reviewers are required to achieve a 3-review majority; that's a lot of reviews, and some tasks are ending up in the queue for a considerable amount of time.
    – Shog9
    Commented Dec 4, 2014 at 23:59
4

What reputation cap is needed for it to become visible? (It is visible for me right now; I just want to know.)

It's not listed under the privilege that unlocks late answers and first post queues nor that for suggested edits and very low quality queues. (If it is at the edit privilege, that privilege should probably be split, since it is summarized as "edit" but is actually "edit and also access 3 new review queues).

4
  • 1
    I have 1,316 rep and I can review it, so it is not at the edit privilege.
    – NobodyNada
    Commented Dec 4, 2014 at 0:07
  • 7
    Mine just showed me the message: "You need at least 500 reputation to review triage questions." When this was first posted there was no message but I didn't see any questions show up.
    – LJW
    Commented Dec 4, 2014 at 0:07
  • I only have access to Late Answers and First Questions, so I would imagine it's the same level as that (500). Commented Dec 4, 2014 at 15:35
  • 1
    It's the same privilege as Late Answers and First Posts. I'll update that topic eventually.
    – Shog9
    Commented Dec 5, 2014 at 0:09
4

At first, I thought that this wasn't the best idea. But the thought of using this to improve the site at least made me not hate it.

Then, I started doing some spot-checks on what people thought "looked OK". Now I'm worried.

Bear in mind that I'm not picking on any of the askers here.

For example, this question. If anyone had looked around a bit, they would have observed that it was a pretty straightforward duplicate of this question.

And yet, it's been marked "Looks OK".

enter image description here

So this is what scares me a little bit about the system. If we rely on people that don't really take the time to review this to improve the algorithm, we're not going to get much improvement out of it.

That said, I'd recommend/encourage that the algorithm evaluate a user's participation within that tag, or overall reputation, before they take these evaluations seriously.

Or, better yet, if the question is found to be of poor quality anyway due to it being closed, deleted, or otherwise disapproved upon by the community, those points of data shouldn't count.

The idea is to remove garbage from the site. Let's make sure it stays removed.

2
  • 5
    Duplicates are hard; there's really no substitute for having someone with a lot of history in the topic look at them. I've closed a couple through triage, but I suspect this queue won't do much for them.
    – Shog9
    Commented Dec 5, 2014 at 6:15
  • 5
    I would have voted "looks OK" in your example. The question actually looks OK: on-topic, clear, concise, well formatted, answerable, acceptable tags... it's not garbage. Sure, if I identify it as a duplicate I would flag it, but I don't think that's the main aim of the Triage queue, there are much worse problems than just asking a duplicate question...
    – AJPerez
    Commented Dec 5, 2014 at 9:40
4

Triage offers a great variety of options of categorizing unsalvageable posts. I think it would be a great help to get some kind of feedback after the community decided on the fate of a post.

Simply color-coding the review audits where the community disagreed with my decision or my suggested reason for closing would be helpful to learn from mistakes and misconceptions and thus improve future audits.

2
4

Duplicate vote has a duplicate in the Unsalvageable dialog:

enter image description here

Both do the same.

Sorry, if it's been already reported or if you decided to redesign the dialog.

3
  • 1
    As the linked question shows, that actually has nothing to do with the Triage queue, which just shows the normal flag dialog.
    – l4mpi
    Commented Dec 10, 2014 at 10:46
  • 2
    @l4mpi, that's what I understand, but the problem is that the navigation through this (flag) dialog is not well arranged hence I'm reporting that the Triage queue offers duplicate close votes on two places.
    – TLama
    Commented Dec 10, 2014 at 11:06
  • As I mention in the answer to the meta equivalent question this is just a short cut to the same dialog.
    – Mark Hurd
    Commented Dec 13, 2014 at 6:15
3

I have found a couple of weird things in this queue. In the history page, specifically.

After reviewing a few questions, I wanted to compare my decissions with the rest of the reviewers'... to make sure that I'm doing it well. So I've clicked on History and opened one of the review links... such as https://stackoverflow.com/review/triage/6383238. I don't know how these are called, I'm going to refer to them as "review pages".

  • For that particular question, I'm the one who made the final vote, so I was presented with the "vote up" dialog (which I did, because it looked like an interesting question, and I was strongly encouraged to do so :). Now, when I open the "review page", the "vote up" button is enabled again, and if pressed, it removes the upvote. Shouldn't it be disabled if I already used it, instead of behaving exactly like the up arrow?

  • For questions that I skipped, and which are still in the queue (i.e. no consensus reached), the "review page" shows the review buttons again (looks OK, should be improved...). I did not try, but I guess it would allow me to vote. It shoudln't behave that way, I already skipped the review because I didn't know what to do with that question. Wouldn't it be better to show the other reviewers' actions? It's what it does once the review is complete (consensus has ben reached).

Other than that... I also miss the "Edit" button, as many people already stated. "Should be improved" is ok, but it would be better if we had also the option "Can be improved, let me do it".

3
  • this "pressing the vote up button removes the upvote" also happens when manually voting up the post while in the "A consensus has been reached" screen
    – Vogel612
    Commented Dec 4, 2014 at 14:56
  • 2
    For skipped reviews we explicitly do allow you to go revisit them in case you change your mind and want to review a task. This also works via the back button. The "vote up" dialog has been removed in favor of new copy and a call to action if you believe the post is worthy of an upvote. Feedback is appreciated! Commented Dec 4, 2014 at 19:22
  • Yes, removing the "vote up" button totally solved the first problem xD. As for the skipped reviews, I'm not sure that I like that second chance to change my mind, but if that's by design, fine then. It was a really minor complain, anyway. Thanks for providing feedback on the feedback!! :)
    – AJPerez
    Commented Dec 5, 2014 at 9:00
3

Some discussion in the comments of Triage gray areas - what should we do when the question is borderline? between me and gnat seems to have extracted that the three buttons may be incidentally making it more difficult to reach consensus.

A proposed option of having two separate triages might be possible.

Triage 1 - Looks Okay vs Does Not Look Okay

Posts that reach consensus on Look Okay are cleared. Posts that reach consensus on not looking okay are sent to Triage 2.

Triage 2 - Needs Improvement vs Unsalvageable.

Same results as before.

This allows for more users to get their opinions on a single question, and establishes the precedent early on whether something is wrong with a post. Then, a post with a high Skip count would be indicative that a post seems to be in a gray area in Triage 2.

A Skip in the current system doesn't indicate very much as it could be bordering between Okay and NI, or NI and UN.

5
  • Consider that this queue already borders on being pure overhead - adding a second queue purely for the purpose of classification starts to get a bit ridiculous.
    – Shog9
    Commented Dec 4, 2014 at 19:10
  • @Shog9 couldn't it be part of the same queue? Like how First Post is both Questions and Answers, and LQPRQ is both Questions and Answers.
    – Compass
    Commented Dec 4, 2014 at 19:11
  • In practice, LQ review only works well for answers. And FP doesn't work all that well for anything. Review queues tend to work best when they're asking for very specific decisions on similar items: this causes less confusion and allows the system to compensate for reviewer error.
    – Shog9
    Commented Dec 4, 2014 at 19:19
  • 2
    There's a separate answer by @davidism which suggests adding an extra button, which I believe would solve this. Its merit lies in that it allows to distinguish between completely unsalvageable mess (delete...) vs serious editing by author needed (close...) vs some editing by anyone needed (low quality queue...) vs OK (publish...). Commented Dec 4, 2014 at 20:41
  • Distinguishing "needs editing by anyone" and "needs editing by author (on hold)" is particularly important. I have clicked NI when the question should have been closed because I thought that's what I should do. Commented Dec 5, 2014 at 20:34
3

Suggested improvement:

Users with access to VLQ queue, should not put the VLQ flag, this adds an extra step in the process of deletion (or not) of the question.

If the user is trusted to put a more accurate flag, it seems more expedient to be done directly on the Triage queue.

2
3

Why am I taken to a "Thanks" dialog when I'm the third person to submit Looks OK?

I'm not entirely sure if this affects anything other than Looks OK, but I usually when selecting that you get taken straight to the next review. However when you're the third person to chose this on a review, you instead get taken to a thanks dialog:

Example

I appreciate that I'm being thanked, but if I'm to be thanked then so should everyone else who voted (without having to manually get back to the review).

I feel this dialog is a bit unnecessary, and would rather just be taken straight to the next review.

3
  • 1
    The "thanks" page also gives you the opportunity to vote on the question. There currently are or soon will be automatic repercussions for the question associated with the other two outcomes, but the only further action that makes any sense for "Looks OK" is a vote - and that shouldn't be automatic. So we give one person the opportunity to vote (to prevent vote inflation); if you don't feel the question warrants it, just move on to the next review.
    – Shog9
    Commented Dec 9, 2014 at 16:36
  • 2
    @Shog9 makes sense. However the question's title within the review process is already a direct link to the question - if I felt a question was worthy of an upvote or downvote I could already get to it by clicking on that link. This thanks page was a bit confusing to see as it made me feel like I'd done something wrong (failed an audit, for instance), as I'd expected to be taken directly to the next review. Commented Dec 9, 2014 at 16:41
  • 1
    Interesting; I wonder what we could do to make that less jarring.
    – Shog9
    Commented Dec 9, 2014 at 16:45
3

Talking of "Skip"...

I did my first rounds of reviewing today and triaged about a dozen or so. I came here to meta because I was unhappy with not being able to say "don't know".

And that was because I work with a 1200x1980 monitor (that is: 1200 wide) where the screen ends before the "Skip" button. I only recognized there was such a thing as skipping when I saw the screenshot in the question on the present page.

Please rethink the web design in this respect.

(Yes, I had the same problem with some other button in stackexchange before; can't remember which one.)

Edit 2015-01-20

This is a partial (but complete to-the-right) screenshot of my full-screen Firefox 31.3.0 window (Windows 7):

SO 1200-wide but cut off on the right

3
  • 1
    Screenshot please? Should be waaay narrower than 1200.
    – Shog9
    Commented Jan 16, 2015 at 20:46
  • 1
    The only way I can reproduce this is to zoom in quite a bit. At 1200, there should be some fairly generous margins on the page, and your screenshot cuts off not only /review but a fair bit of the Ask Question button and search bar.
    – Shog9
    Commented Feb 6, 2015 at 21:01
  • @Shog9 You are right, that appears to be the explanation: My browser view was magnified. I had not been aware of that. Still, I think layouts that do not grow beyond the screen width when magnified are usually preferable. Commented Feb 9, 2015 at 14:19
3

I'm wondering if "Looks OK" should need a majority of three votes instead of only an absolute of three votes.

Usually when my "Looks OK" vote is the third, the thank you dialog looks something like this:

enter image description here

It is in my experience rare that you will only see three "Looks OK" votes there. I guess worst case would be two votes "Unsalvagable", two votes "Should Be Improved" and three votes "Looks OK". The question will go to the homepage although only the minority of voters think that the question is ok.

So I'm wondering if the triage quality could be improved by requiring the "Looks OK" votes to have a majority of three votes over all other votes. In my example that would mean two more "Looks OK" votes.

Update

I decided to explore this thought a bit further and get to know SEDE in the process. The result of this exploration is Triage Review Decisiveness. As suspected, a decisive result where one option has 3 more votes as the other options combined is more the exception than the norm. The percentage of indecisive results seems to hover around 70% for all three options.

As a side result, surprisingly (for me), the "Should Be Improved" option seems to be slightly less favored than "Looks OK".

Originally I also wanted to weigh the votes by the reputation of the user, but I found no way to get the user for a review result.

2
  • 1
    Is the ReviewCount column the count of that outcome? If so, why doesn't it sum up to around 565,262, the current total number of triage reviews? Commented Mar 8, 2015 at 16:31
  • @Jeffrey Bosboom: Thanks for the feedback. There are several factors why the sum of the ReviewCount column of my query differs from the "reviews all-time" in the review stats (stackoverflow.com/review/triage/stats). First of all my query was restricted to the reviews of the last 7 days. I removed that restriction now. Second, "reviews all-time" seems to count the number of triage review votes, while my query only counts the review tasks. Third, my query only counts completed reviews.
    – FH-Inway
    Commented Mar 8, 2015 at 20:25
2

Something to consider (not necessarily pitching it as a good idea) is allowing us to manually enter into the triage a specific question - much like how we can with close votes, as well as seeing pending edits.

This is mainly because we can do it for every other existing queue.

However, here's why I do think it's a good idea:

Close votes and flags are both absolute votes for closing something. The On Hold system tried to remedy this but alas it ended up being a comfort blanket instead of a real solution.

In order for the Triage queue to be this solution, I feel like we should be able to manually place questions into the triage queue in order to give them a second chance.

For instance, a user posts a question that may be able to be salvaged; instead of voting to close and instantly discouraging them, as a user, I would be able to enter into the Triage their question (with an initial vote to improve) and give them a chance to not only save their question but to see how an external editor would have asked it.

To me, that would be a much better lesson in how to work the site - especially since this is a trial run for a potential replacement of current queues on the site.


triage example

5
  • 3
    I think when it is fully implemented, the question won't even appear on the homepage if it's in triage. So you wouldn't be able to get to the question in the first place without a direct link. I'm unsure if they will also be hidden from the per-tag question lists.
    – Mysticial
    Commented Dec 4, 2014 at 0:49
  • To an extreme, what if we removed close voting on a post directly; you could only send to triage, and triage would vote to close. It would take more time to close a question, but it would give the user possibly more chances to fix their content before being hit wit "on hold".
    – gunr2171
    Commented Dec 4, 2014 at 0:52
  • 1
    It seems like VLQ flags on questions will put a question back in triage, according to Shog. Though VLQ currently means "delete now", not really what you mean when you want to put a question back in triage. Commented Dec 4, 2014 at 0:54
  • @Deduplicator: Huh, I've never seen a question closed as very low quality. I always use the 'closed for another reason' reasons when flagging for closure. Commented Dec 4, 2014 at 1:03
  • If a user is discrouraged by closure of a bad question, they're not fit for participating on SE anyways. Furthermore, I doubt that most people would learn to ask good questions by having other people fix their problems for free; I'm way more inclined to believe that they learn by having to invest effort themselves to get any results. So if you see a bad question, please don't worry about discouraging the OP by closing it - keeping it open would be discouraging to all the people who don't want to see SO being overrun by crappy questions.
    – l4mpi
    Commented Dec 4, 2014 at 8:11
1

Whenever I have reviewed, and the consensus is "Looks Ok", I get a weird bouncing animation on the voting dialog (sorry for the bad quality):

bouncing vote buttons

This is quite annoying (it's like <marquee> and <blink>!). Can it be removed?

5
  • 6
    Not as annoying as that animated gif. I spent like 10 minutes tweaking the bounce rate there; you couldn't even bother to keep it in frame for the whole animation?
    – Shog9
    Commented Dec 4, 2014 at 22:19
  • @Shog9 Trust me, it was a hard capture process. I had to use some Microsoft software thingy, then transcode it to a WMV. I then uploaded it to some website which converts WMV's to a gif. It probably lost a good amount of quality along the way.
    – hichris123
    Commented Dec 4, 2014 at 22:22
  • 5
    Those jumping votes are absolutely cute!!! <3 No need to remove this animation...
    – nicael
    Commented Dec 4, 2014 at 22:38
  • 4
    I fixed your gif. The new one loops once every 10 seconds, but the actual animation does not repeat.
    – Shog9
    Commented Dec 4, 2014 at 22:44
  • @Shog9 It was that annoying? Sorry. ;)
    – hichris123
    Commented Dec 4, 2014 at 22:46
1

Flashing triage question text, suggested improvement

When you triage a question it's text will be greyed out while the system looks for a possible next question. The text will briefly flash back into non-greyed before being deleted if there is no next item to review.
If there is a next item to review and the internet (or SO servers) are fast in responding, I never noticed anything, but today the internet seems slow and the old message gets presented, with the refreshed header, only to be replaced after 1-3 seconds by the text of the new question to triage. The latter is quite confusing.
You probably can fix this by completely removing the innerHTML of the greyed out text before starting to load the next page (in both cases: new Q for triage available or not available).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .