523

Prologue

Since late 2019, I have been investigating why posts are ending up in the Help & Improvement (H&I) review queue when it should have been closed in Triage. Unfortunately, due to the poor choice of wording of the Triage action Requires Editing button, more work is created for everyone:

  • new questions don't get closed fast enough
  • poor questions flood the H&I review, and users are not able to edit them, so it's basically a skipfest
  • I have to develop my own tools to manually handle such H&I reviews, by being able to manually close the question in H&I
  • I have to visit each question's post timeline to fetch the Triage review that pushed the question into H&I (happens when 3 users vote for Requires Editing)
  • I have to apply manual bans and pray hard that the users chance upon the review ban message, WHICH IS NOT ALREADY APPARENT AND THEY GO ON TO DO THE SAME THING AGAIN AND AGAIN.

Apparently, Triage has been this way since March 2016:

Which has been made worse with the granting of review privileges to at least 53,000 new users recently, an understatement. (60 answers, score of -501 on MSO)


Current Status

Currently on Stack Overflow there are 2894 banned reviewers. (this is not the norm if you are wondering - just a year ago it averaged 200 users at any one time)

  • Out of which only 118 are automated bans for failing review audits (~5%)

  • 2118 review banned users are manually banned for clicking Requires Editing in Triage, when the question should have been closed instead

  • Out of 2894 currently banned reviewers, 2767 are banned from Triage for some reason (combined audits and manual) 96%!!!

  • I am currently adding manual bans every day at an average rate of 200 per day, and even more bad reviews are NOT being caught as I am the only moderator who is handling this issue

  • Also, at this rate we are going to have 4000 banned reviewers around April 1st, and probably 6-7 thousand by end-April

See live Google spreadsheet for full stats and trend graphs.


Quick Wins?

This Triage issue, if unable to be fixed without further analysis and investigation, could still be alleviated by implementing the following:

1. Make it more obvious that you're review banned

  • so that users can be alerted to the reason and avoid making the same mistake
  • perhaps push a notification to the user's global inbox?
  • (and 15 linked questions)

2. Retain a link in the top bar review queue menu for users in review suspension

  • same as above, currently the only way to read the review ban message is to go to stackoverflow.com/review, but there is no link to this page if you're currently review-banned.

3. Raise the review privilege from 500 to 1000

  • due to the free rep users got from the doubling of question rep from 5 to 10, we have at least 53,000 new users who suddenly flood the review queues without knowing how to review

4. Implement a compulsory tutorial/guide when users gain the privilege to perform reviews

5. Change the text of the action buttons from "Requires Editing" and "Unsalvageable" to avoid confusion

  • to "Needs an editor" and "Should be closed..." respectively

Further steps

Join us in the public discussion room where users are getting banned live, for daily updates (stats/graphs), and escalation plans on the situation:

>> Bad Stack Overflow Reviews

22
  • 8
    Samuel, is H&I performing useful work? Maybe removing that queue and the "requires editing" button would be enough.
    – yivi
    Commented Mar 9, 2020 at 9:33
  • 2
    @yivi If the triage queue worked as it's supposed to, then potentially it would be, the only reason it's not performing useful work is because triage doesn't work Commented Mar 9, 2020 at 9:34
  • 2
    @Nick Not necessarily. H&I might be a useless queue regardless of triage, however well-intentioned. It's not only about bad posts being sent there, but if are enough users doing useful work there. Stats could be gathered about reception/peformance of not-closed questions that went through the queue, for example.
    – yivi
    Commented Mar 9, 2020 at 9:36
  • 5
    I'm not sure what you're proposing can be solved with review bans. A fundamental overhaul of triage and H&I is required
    – Magisch
    Commented Mar 9, 2020 at 9:50
  • 6
    @Magisch yes, I can't solve Triage with review bans, but I can temporarily prevent bad reviews from reviewers who vote for unsalvageable questions to get into H&I which creates more work for the rest of us.
    – Samuel Liew Mod
    Commented Mar 9, 2020 at 10:39
  • 7
    This really does need fixed, I only found what I was doing wrong from the ban message with a link explaining how I was voting wrong in triage after my second ban
    – Taazar
    Commented Mar 9, 2020 at 11:07
  • 2
    This thread speaks to my heart! Thanks for voicing my concerns, much more concisely than I could. Commented Mar 9, 2020 at 13:40
  • 5
    The only option buttons presented in H&I are Edit & Skip. There is also the "question is very low quality" link but nobody uses that ever in fear of declined flags.
    – Samuel Liew Mod
    Commented Mar 9, 2020 at 13:49
  • 4
    Thanks to this post I now know I'm review banned. Apparently for making the wrong decision on one question. Bans appear to be anonymous, give no chance for feedback and no notification to the user who is banned.The ban comment just pointed me back to the review with no further information. I know when I'm not wanted. I'm done reviewing, commenting and answering.
    – chrisis
    Commented Mar 9, 2020 at 14:46
  • 3
    @chrisis In this situation, trying to learn from what's happened is more useful than getting angry. Also, the ban is not really punishment, since it's only impeding you from performing free work, temporarily. The idea is to catch your attention, to give you the opportunity to learn what you did wrong, so that if in the future you want to continue working for free you can do it better. Since you are doing this because you want to contribute, it makes sense giving you the opportunity to learn how to do it more effectively.
    – yivi
    Commented Mar 9, 2020 at 16:06
  • 3
    @chrisis If you disagree with a question closure, that's a different issue. You can raise that in a different meta question, but please be prepared for disagreement, and be open to the idea that maybe you could be wrong about the your interpretation of the site's rules.
    – yivi
    Commented Mar 9, 2020 at 16:06
  • 1
    Totally agree to @yivi here. Why do we need that queue at all? Simple formatting issues (e.g. code as text) get fixed by the tag watchers, everything else is either minor or it "requires editing by the OP", which no queue can fix. Commented Mar 9, 2020 at 18:07
  • 1
    I'm curious. Have the numbers of banned reviewers gotten back to the levels of last year? I guess the changes made by the CM team are in place by now? Commented Jul 10, 2020 at 11:52
  • 2
    over here after my first review ban too. honestly, from the time I got the reviewing privilege to the time I actually got around to doing any, I forgot any (if any?) instruction telling me that Requires Editing did not mean/include by the author. And Unsalvageable means only the author could improve it (that would make salvageable, then, no?). I apologize for the number I expect I dumped into the wrong queue. I think the simplest thing is up front UI fix. change the button name/label to make it obvious. Add 1-liners under the button to make it clear. something. no process change needed.
    – Nick J
    Commented Aug 24, 2020 at 19:37
  • 2
    I concur with many of the complaints that have already been said. I wanted to see if I could contribute a new perspective here after my recent ban experience. I ended up writing my response elsewhere. But I'd like to share it here too, so here is the link. Rather than enumerating the many problems with the current queue system (as others have already done a good job at covering), I wanted to see if I could address what I think lies at the heart of the issue.
    – peacetype
    Commented Sep 30, 2020 at 0:28

27 Answers 27

109

Thanks everyone for your suggestions and feedback. We plan on addressing this issue as part of our upcoming review queue overhaul project, and will be taking everything written here into account.


Updates on Meta Stack Exchange:

Improving the Review Queues - Project overview

Announcing tweaks to the Triage Queue

Workflow changes for review queues

12
  • 15
    Great news, is there an ETA on that project, or just the general 6 to 8?
    – Erik A
    Commented Mar 24, 2020 at 9:55
  • 14
    We are hoping to include it in the roadmap for Q2 that we are working on right now, but nothing certain until we have finalized that (at which point the public roadmap for Community Q2 will be posted on MSE)
    – Yaakov Ellis StaffMod
    Commented Mar 24, 2020 at 10:02
  • do you have reasons to think that reviews other than triage need (urgent?) overhaul? (to avoid misunderstanding, technical debt would count as a solid reason to me in this case)
    – gnat
    Commented Mar 25, 2020 at 7:10
  • 10
    @gnat obviously triage has it's own set of pressing concerns. But all of the review queues need some love. We are planning a full overhaul on the backend to support doing things with queues that we have not been able to do thus far (due to technical limitations of the current implementation - there is your technical debt). Our UX Research team has spent many hours going through feedback and speaking with users about the ways in which they use review queues. So we hope to address both the urgent items such as this, as well as more general issues relating to all queues.
    – Yaakov Ellis StaffMod
    Commented Mar 25, 2020 at 7:46
  • 7
    Haven't seen status planned on anything like this in a very long time. Looking forward to it Yaakov! The queues are such a core system to the ecosystem here.
    – Travis J
    Commented Mar 25, 2020 at 15:07
  • 4
    @YaakovEllis: While we're fixing the review system, can we make the results of the reviews match the button descriptions that were clicked? If the button says "Looks OK," the review result should also say "Looks OK." See this review, where the user clicked "No Action Needed," but the review result says "Reviewed." Commented Mar 26, 2020 at 13:52
  • Yaakov, is making the pages responsive also part of the intended overhaul?
    – Luuklag
    Commented Mar 26, 2020 at 20:02
  • What is sad, is that I reviewed some of the help info for triage, etc. recently before processing some items in the queue, and was real careful (at least I thought I was!) and got banned last week I think! I wonder if I set off a automatic 5% audit (I only remember getting "caught" once recently on an audit, and passed a few others), or if someone was not impressed with my votes. Commented Apr 1, 2020 at 15:26
  • @Yaakov in the short term is it possible to add extra text to the Triage page description as suggested here?
    – rtx13
    Commented Apr 1, 2020 at 17:54
  • 3
    PLEASE don't raise the review privilege to 1000. I was over 500 even before we changed the question vote rep from 5 to 10. Commented Apr 4, 2020 at 14:42
  • 7
    I would also review the ban time. I have been reviewing for months with no issues and suddenly, because of a "Requires Editing" instead of "Unsalvageable", I see myself banned for 4 months.
    – Luke
    Commented Apr 16, 2020 at 8:14
  • 4
    I don't participate in the review process at all. I tried it once, I approved a question which was written in poor English so the moderator hadn't understood it, but I did, and I got such bad feedback for doing this that I never tried again. Commented Apr 28, 2020 at 7:01
237

Triage is a flawed concept in this context. In the real world, triage is used to quickly classify subjects and leave their treatment (if required) to someone dedicated to that specific purpose. But, when the subjects are simply posts on the Internet, there is no reason for hurry. No poorly formulated question will die if not treated fast enough. Also, no-one is sitting and waiting for the arrival of new posts requiring editing. Volunteers come by in the H&I queue when they have time.

Triage and H&I should be merged into a single queue. "Requires editing" should be replaced by "Edit" and the reviewers, if they feel a post "requires editing", should be welcome to improve it themselves.

The way it is now leads to situations where the Reviewer in the Triage queue knows how the question should be edited, clicks "Requires editing", and the question lands in someone else's hands in the H&I queue. This Someone Else cannot read the original Reviewer's mind, doesn't know how to edit the question, gets p***ed off and bans the Reviewer (possibly with a wrong justification).

The Reviewer, not being able to understand the reason for the ban, decides to refrain from further reviewing.

Other than that, I agree with the OP's proposal 4: Implement a compulsory tutorial/guide when users gain a potentially damaging privilege (not only review).

11
  • 37
    Merge these queues == remove H&I. Looks good to me.
    – yivi
    Commented Mar 10, 2020 at 12:11
  • 21
    "No poorly formulated question will die if not treated fast enough." Ovation! I also don't understand the urgency for salvaging a post in a single step; sometimes the same goals can be reached with more steps: if I correct the formatting, an expert could be encouraged to take a look, and to ask the OP for that specific clarification that will result in an answerable question. Another important topic is the expertise in the question evaluation: if only an expert can undetstand if a question is clear enough why am I not allowed to filter triage questions by selecting the tag I am expert in? Commented Mar 10, 2020 at 22:31
  • 25
    No poorly formulated question will die if not treated fast enough. Unfortunately, poor questions do often attract answers (which are often complete guesses and/or not helpful due to the question being unclear). Obviously duplicate questions have the same issue - some people try to post the well-known answer before the question gets closed just so they can have a chance of getting rep, and these sorts of reposted answers aren't helpful either. There is a strong public interest in closing questions that should be closed ASAP... but this is a much larger problem than Triage. Commented Mar 10, 2020 at 23:14
  • 1
    My only problem with this is I believe we grant users the ability to use Triage before they can edit. So I'm not sure how effective that idea is. It will still just ultimately end up in another queue.
    – user10957435
    Commented Mar 11, 2020 at 2:21
  • 2
    @CertainPerformance: My proposed model makes editing faster at the expense of slower flagging of poor questions. But, the additional benefit of my model is that no unsalvageable questions block an edit queue. In total, the performance of the site improves.
    – Igor F.
    Commented Mar 11, 2020 at 9:39
  • 1
    "No poorly formulated question will die if not treated fast enough" actually, triage was meant to do the opposite, get poorly formulated questions out of the view of people. Basically, shove them under the rug. "Looks ok" questions rarely aren't "ok"
    – Braiam
    Commented Mar 11, 2020 at 15:24
  • 4
    @CertainPerformance: The fix for that would be to temporarily disallow answers to new questions until they've passed "triage" (or something roughly similar; call it "new question review" or something). Or replacing the current inefficient and non-gamified dupe closing system with something else — e.g. turning suggested duplicates into a special kind of answers that could be voted on and possibly accepted. Or maybe both. But rapid closing under the current system is basically an unwinnable race; the only SE site I know of that's gotten even close is Role-playing Games, and only by heroic ♦ mod efforts. Commented Mar 16, 2020 at 15:43
  • 1
    "Volunteers come by in the H&I queue" never. It's terrible. An absolute crap hole.
    – S.S. Anne
    Commented Mar 17, 2020 at 21:30
  • 2
    I'm siding with @IlmariKaronen on this one. Personally, I've never used "requires editing"; If I think a post needs editing and what should be edited, I go edit the post so that the changes go into a change review queue, just like happens for anyone else who is able to edit a post. Otherwise, I'm effectively passing the buck to someone else, IMO. Commented Mar 31, 2020 at 8:50
  • 8
    Right. Ban me from reviewing for a month for an honest mistake? Don't bother lifting my ban, I won't waste any more time reviewing. Now both staff and moderators are throwing their weight around. Nothing lasts forever. I wonder if Stack Exchange has jumped the shark? You may want to think through how your (otherwise excellent) "be nice" policy applies to long-time volunteers.
    – O. Jones
    Commented Apr 5, 2020 at 10:48
  • Should the merged queue be avalable with 500 or 2000 reputation? If it is available with 500, there should be a suggest edit button, right? What if multiple users making the queue suggest different edits?
    – dan1st
    Commented Apr 14, 2020 at 9:50
116

There's a lot of focus on the "requires editing" option. I think there's some adjustment that we could make there to the description but if we're looking at a quick fix while we're considering other options, I'd actually like to look more closely at the help text for both requires editing and unsalvageable:

  • Requires Editing for questions that you can make clear and answerable by editing
  • Unsalvageable for questions that cannot or should not be answered and must therefore be removed from the site

As I've been hearing about this problem, what I keep thinking is that there are two different types of editing that's needed.

  • Editing for grammar, spelling, and other cleanup that can be done by any user with the interest to do so.
  • Editing by the author to add additional information and details so that the question becomes answerable.

So, in a sense, we're failing to clarify who needs to edit something to be a correct usage of "requires editing". Yes, the explanation says "that you can make clear" but that's easy to misinterpret and the alternative choice is... somewhat bleak.

"Unsalvageable"... is... kinda harsh. And I can imagine that someone who is looking through these reviews might be hesitant to describe a question looking for help that way - particularly someone new to review. So, the bigger change I'm suggesting we make is to update the terminology:

Incomplete for questions that should be closed because they need more detail from the asker to be answerable or are off topic for this site and should be removed.

I'm not 100% convinced this is perfect so feel free to suggest other terms or descriptions that you feel explain the use case more accurately. My main goals here are to give users a less scary or judgemental term that will make it more likely they'll actually use it. Bonus points here for introducing the concept of closure and when questions should be closed.

We can also update requires editing to something more like:

Requires Editing for on topic, complete, answerable questions that may need minor editing to clean up formatting, grammar or other minor issues.

Again, there may be better phrasing here but I hope you can see where I'm going with this.

24
  • 13
    What about also addressing the issue that users who are review banned are not notified about it, and the poor UI (when banned) doesn't help them get to the page where the review suspension message is displayed? This will help all reviews in general and not just Triage.
    – Samuel Liew Mod
    Commented Mar 11, 2020 at 3:30
  • 3
    Creating notifications is ... apparently harder than one might imagine. I don't think that's a bad solution - to be clear. The suggestion here is merely a quick copy change we could likely do without a ton of work - though we'll need to make sure we update any help documentation to match.
    – Catija Staff
    Commented Mar 11, 2020 at 3:36
  • 24
    "Incomplete" doesn't fully capture questions that should be closed because they are opinion-based or wildly off-topic (i.e.: not a programming question). Incomplete suggests that the question can still be salvaged by the question asker, so this still doesn't differentiate itself from "Requires Editing". May I suggest "Should be closed..." and "Needs an editor" instead?
    – Samuel Liew Mod
    Commented Mar 11, 2020 at 3:47
  • 2
    "in a sense, we're failing to clarify who needs to edit something to be a correct usage of "requires editing"" perfectly captures the issue here. I have been requesting, and seeing requests, to address that, and I think your "requires editing" approach will definitely help.
    – Travis J
    Commented Mar 11, 2020 at 6:18
  • 3
    Perhaps, instead of Incomplete it should simply be Off-Topic. A question that is unclear, too broad, opinion based, are as off-topic as tool recomendations, etc. it could still read along the lines of "Incomplete for questions that should be closed because they require more details or focus from the question author or they are off-topic for this site and should be removed."
    – Thom A
    Commented Mar 11, 2020 at 11:26
  • 5
    @Larnu “questions that are lacking detail or too broad aren’t “off topic” if they’re about programming within the scope of the site. I... know people may broadly use the terminology that way but it’s something I actively work against because it’s confusing to people. “I asked a question about programming and they told me it’s off topic... so what is on topic?!!!???” We should avoid this ambiguity as much as possible.
    – Catija Staff
    Commented Mar 11, 2020 at 12:00
  • 3
    Perhaps Unsalvagable needs to be 2 options then, @Catija . If Off-Topic and Incomplete are both closure reasons, but for very different reasons that do not overlap, they should be treated as such. Unless having a wide ("ugly") button with Incomplete/Off-Topic is seen as a better solution. A question that is Off-Topic can certainly be complete (especially if it might need to just be migrated), and based on your above comment the reverse is true too.
    – Thom A
    Commented Mar 11, 2020 at 12:07
  • 1
    @Larnu Except this isn't closing. The two buttons would do, essentially, the same exact thing, so that seems like a bad UI. That's why Sam's suggesting "Sound be closed"... I feel like that's a bit long and also that it requires knowing what should be closed... which a lot of people at this point don't know. I'm not sure that this is the right time to teach someone what that means. We need a comprehensive rework of review that helps teach people as they gain privileges... It's a good goal, it'll just take a while to build.
    – Catija Staff
    Commented Mar 11, 2020 at 13:05
  • 1
    "Except this isn't closing." Then why would my actions in the review in triage close a question if it doesn't close? That doesn't make sense. Of this isn't about closing, then voting Unsalvageable shouldn't close the question; regardless of the reputation of the user.
    – Thom A
    Commented Mar 11, 2020 at 13:13
  • 1
    @Larnu It opens the flag dialogue, right? So it's... a huge breadth of different things... it could be spam, it could be rude, it could just need to have the code written out instead of in an image... it could be a question that belongs on another site... so, I guess what I should have said is "it isn't only closing". For users with under 3k, they're not voting to close, they're just flagging. I don't know what the mentality was when we chose "unsalvageable". It's hard to find a brief way to encompass so much. To that end, "incomplete" doesn't cover the red flag usage, either.
    – Catija Staff
    Commented Mar 11, 2020 at 13:28
  • 2
    Does the phrase "Requires Editing" make sense? I think the problem I have with that has always been, nothing requires editing the way it's describing. If I can edit someone else's question to make it more readable, then it did not require editing, because I was already able to discern the meaning of the question. If it required editing, then I would not be able to discern the meaning. The only required editing would be from the OP. I think the phrase should be closer to something like "Would benefit from others' editing" or "Can be improved by others", but I don't know the perfect phrase Commented Mar 11, 2020 at 18:15
  • 30
    I'd prefer "Requires Cleanup" over "Requires Editing", because it makes clear what sort of editing is expected.
    – Mark
    Commented Mar 11, 2020 at 20:59
  • 5
    I still say skip H&I entirely and throw up an edit screen for the reviewer. If they think it's fixable by a third party edit, make them do it right then and there. I bet most of them pick another button when they realize that
    – Machavity Mod
    Commented Mar 12, 2020 at 14:52
  • 8
    How about "Requires minor cleanup" and "Requires fundamental changes [to be on-topic]"?
    – deceze Mod
    Commented Mar 12, 2020 at 15:21
  • 2
    @catija The main issue I haven't seen mentioned: if a question has major formatting issues (obviously requires editing) BUT IS ALSO unsalvageable. In that case, choosing "requires editing" can lead to a ban (problem is, at first sight the question has obvious formatting issues, and that's blatantly more obvious than the unsalvageable reasons.) So it could be fixed by an indication:"choose require editing ONLY IF question is not unsalvageable". (I think this is leading to most suspensions atm...)
    – bad_coder
    Commented Mar 13, 2020 at 6:05
106

Getting a ban is a meh way to notify the user about the rules: it doesn’t leave a good impression, some damage has already been done, and it requires moderator time.

Earlier is better.

Clicking “Requires Editing” should give a pop up for the user to explain what people in the H&I queue should do, selected from a list of the most common options plus maybe a free form box that leaves a comment. This should make it clear that it’s being sent to one’s peers in the other queue and not the OP.

The message would then be shown in the H&I queue, because I feel that queue currently seems to require some mind reading. Two birds.

7
  • 23
    Samuel already knows that the bans are not the ideal way to deal with this, and he's certainly not advocating to continue this way. But it's the only tool in the moderators hands at the moment. The can't retool the queues, can't change the user interface, etc.
    – yivi
    Commented Mar 9, 2020 at 17:01
  • 2
    @yivi Then moderators need new tools. If the standards are high for us then they should be even higher for the mods.
    – Forklift17
    Commented Mar 13, 2020 at 5:19
  • 9
    @Forklift17 and who's going to provide the new tools? Isn't that's what this featured post I made is for: to bring attention to this issue?
    – Samuel Liew Mod
    Commented Mar 15, 2020 at 5:41
  • 4
    Totally true, a ban is annoying - especially when your own impression is, that the post could have been successfully edited. Commented Mar 21, 2020 at 20:42
  • 18
    It really doesn't matter whether the wording is clarified. Banning good-faith reviewers discourages them from ever contributing again, and further degrades the quality of SO. Review is a minefield that I'm tired of trying to cross. This is a culture problem, not a technical one -- no amount of UI gimmicks are going to help this. Commented Mar 28, 2020 at 23:59
  • 1
    banning (by the OP) made me to find this thread and read the rules and to find out what I misunderstood. Now I understand the concept better, and hope to provide better quality participation in the future, but man, this felt like a kick in the nuts. I am banned for a week, and must honestly tell you this makes me feel like my contribution is quite worthless if I can be banned without a warning on the spot.
    – ForestG
    Commented Aug 13, 2020 at 12:50
  • 1
    It's not only meh, it's completely discouraging. I still don't know why I got banned for a whole (!) year? Did I misunderstand the system? Made a mistake? Not even a DM "Hey are you aware of..."
    – swaechter
    Commented Apr 22, 2021 at 9:48
60

I burned out reviewing on Triage. It was like trying to review the Wild Wild West. No sense of direction, anything goes (Or does it? Who knows?). I believe that if I had had a strong clear sense of purpose for how to approach the reviews in that queue, like I feel I do for Close/LQ/Suggested-Edits/etc.... I would have reviewed thousands by now and kept going. Further, I think many others would have completed far more useful appropriate reviews and the high-minded idea for what that queue was meant for (I do remember the discussion long ago) would far more likely have been met. I'm probably not going back, atleast not soon, but I hope this suggested implementation of Triage can be sculpted into shape and meet its potential.

Your idea of a ban notification is a good one not only for this queue, but for the site overall.

53

As someone who is currently banned from reviewing because of this exact issue I thought I would give my reasoning and thoughts on a solution.


My Experience as a Reviewer Who Got Banned

Triage question It has since been removed, but I will post it anyway as moderators can still view it.

A user posted the following question, which for obvious reasons required triage.

I am a beginner in python and I really wanted to make a auto clicker bot, my code is correct but It does not work, I already installed pynput using pycharm. Are there any other solutions?

The comments on this post from the included the usual.

  1. Welcome to stackoverflow. Honestly, you will probably not get many answers if you do not show the relevant parts of your code. No one will know what to do with "my code is correct but it does not work." You'll get much more help if you edit and show the code

  2. Please post your code (simplified to the smallest form that demonstrates your problem, and what results you are seeing when you run it.

  3. If you posted at least the error message in your question, we might be able to determine if it's because you don't have pynput installed

As a new reviewer, I was presented this question and marked it as Requires Editing, which after reading this meta discussion I now know was wrong. However, let me give you my reasoning at the time.

The tooltip you get for Requires Editing has the following text.

Requires Editing for questions that you can make clear and answerable by editing.

I believed the YOU in this case referred to the original poster of the question, as I thought that should the question be marked as Requires Editing they would see the same tooltip I had.

I marked the question as Requires Editing because the question context was appropriate for the community it was posted to, and if the original poster edited the question in response to the comments already existing on their question, they would have an acceptable question.


Takeaway

Root Cause: As the reviewer, I believed Requires Editing meant that the original poster should edit the question to make it acceptable.

So what would have helped me in this case to mark the question correctly? I think the following tooltip would have.

Requires Editing for questions that can be answered as-is, but require editing to make it easier to find and understand.

This definition makes it clear that Requires Editing should be selected if there is enough information already present to work towords a solution, but the question requires editing to make it presentable and more understandable. This definition also follows the same format as the existing Looks OK option.

Looks OK for questions that can be found, understood and answered as-is

7
  • 9
    Thanks for reading the Triage guide as well as sharing your thoughts on the matter. I've lifted your review ban early. (My advice is to try to stay away from Triage at the moment though)
    – Samuel Liew Mod
    Commented Mar 18, 2020 at 1:35
  • 5
    This got me too. It's not clear what the "you" in Requires Editing means. First time I'm looking at the Triage queue. I usually fix some typos in the edit reviews, but now that's blocked too.
    – Asti
    Commented Mar 27, 2020 at 11:17
  • 4
    I also thought the Requires Editing sent the question back to the OP for further improvement. Going through the Triage Guide now @SamuelLiew thanks for the info.
    – santamanno
    Commented Mar 28, 2020 at 11:21
  • 8
    To me, "needs editing" means someone needs to edit it. Banning people based on some fine distinction pretty much underscores the reputation of SO as a minefield of unfriendly humorless pedantry. Commented Mar 29, 2020 at 0:03
  • 2
    I'm in the same position here. While new to reviewing, I received a 4 month ban until August for a "requires editing" first offense. This seems a bit harsh to me. I was sincerely trying to do my best. I read the triage guide and would appreciate a lessened ban of possible. Thanks! Commented Apr 13, 2020 at 20:13
  • 1
    Same here; similar misunderstanding; banned from triage after (I believe) first offence (maybe there are more which would fall under the same category/misunderstanding) - I feel that a lengthy ban after the first offence is a bit harsh; it actually will discourage me from going into the triage and help. Too bad because I liked helping/sorting queues (but not if it's a detriment to me for honest mistakes) - oh well, moving on and learn to ignore these things
    – blurfus
    Commented Apr 20, 2020 at 16:43
  • 1
    I made the same mistake several times but that's a long time ago (haven't really reviewed much since, at least not in triage). I was also under the impression that you was more a one. Basically having a "OK as-is", "Not OK as-is but can be salvaged", "Cannot be salvaged" choice, which intuitively does make sense, although in the wrong way. (Also, an actual 2nd-person you wouldn't even make sense here, because I don't get the option to edit there. and how does it matter to others that I think I personally could fix it, maybe because I know what acronym XYZ means that was used?)
    – CherryDT
    Commented Apr 24, 2020 at 9:36
52

Adding written instructions to guide the people that don't read written instructions won't work. So, here's a suggestion for changing the "Requires editing" button's functionality I have been considering posting for a while:

When the reviewer clicks "Requires editing", bring up a dialog similar to the flag/unsalvageable dialog box. Provide two options (plus a cancel-button for fat-fingered individuals like yours truly):

  • The original poster should add details -> Flag as "Needs details or clarity"
  • Requires editing for language, spelling or formatting -> Move to H&I

I used the Developer Console to make a mock-up of this dialog:

Mock-up of the dialog described above

4
  • 1
    Not every post which currently should be treated as "unsalveagable" should be closed as "Needs details or clarity". All other flag reasons, including spam, fall under that button now. You'd lack a whole lot of options when changing the button to just "Needs details".
    – Adriaan
    Commented Mar 12, 2020 at 10:48
  • 20
    @Adriaan I'm not suggesting the "Unsalvageable" button get changed, I'm suggesting that an additional step be added to the "Requires editing" button.
    – Haem
    Commented Mar 12, 2020 at 12:56
  • 3
    I fell for this, reviewing "needs editing" because the author should add some information, while "unsalveagable" sounded way too hard. It sounded like an interesting question to me - I wanted the author to add information, and this UI would have naturally pushed me towards the correct thing, while currently I intuitively go for the wrong thing.
    – alex berne
    Commented Mar 20, 2020 at 14:26
  • 2
    I like this idea, because it does a preliminary analysis and suggests review in a systematic way, and doesn't slap the original poster for not knowing the rules. It guides them to make better postings. Commented Mar 23, 2020 at 2:42
27

Triage bans are a good way to offend users.

Sometimes, I worked on the close or reopen queues.
Today, I used the triage queue the first time.
And I got banned for clicking at the Requires Editing button.

This is offending, you don't want my help: okay

I will never use any of the queues again.
I will not vote nor close any questions.

First time since 280 consecutive days (missing only a few days in the past 9 years), I logged off.

It's okay, now I have more time for other things.

13
  • 14
    You're making work for everyone else to clean up after you. If you stop doing that, it's a good thing. Commented Mar 24, 2020 at 7:43
  • 3
    I think you mean queue bans are a good way to get the attention of users Commented Mar 24, 2020 at 7:45
  • 9
    Imagine a person tries to volunteer to a meat-space organization. And during their volunteering they end up hindering the organization work instead of helping it, doing things wrong and creating more work for other volunteers. One would expect this person would be told off, right? That way they can learn that they are not contributing effectively and can change their approach. Well, it's exactly the same with review. Nothing to get offended about. Just see what you did wrong, do it differently in the future, and everybody wins.
    – yivi
    Commented Mar 24, 2020 at 7:47
  • 25
    @yivi Imagine someone volunteers for a meat-space organization, is asked to file things that are "red" or "purple", files something colored puce in the wrong box, and is immediately fired. Commented Mar 29, 2020 at 16:16
  • 1
    That could be weird, depending on the severity of the action. Good thing no one is fired here, @Chuck. Simply temporarily suspended so they can find out what they did wrong and change their course of action.
    – yivi
    Commented Mar 29, 2020 at 16:42
  • 13
    @yivi Is there any point in the temporary suspension though? Why not just a notification bringing to their attention what they did wrong in the review? Otherwise we're going around in a circle - let me explain. A person gets review banned and understand where they went wrong, but decide not to review again because they feel the help they've provided in other reviews has been thrown back in their face. Other users, without the understanding of what's wrong start reviewing and the same happens. Where as if the user with the understanding wasn't banned, we wouldn't have the problem! Commented Mar 31, 2020 at 17:07
  • @The There is no other way to notify users of this at the present time, mods have to work with the tools available. They are requesting better tools, though. Notwithstanding that, a notification can be easily ignored/missed; and the user who received may continue reviewing badly. Reviewing badly is worse than not reviewing.
    – yivi
    Commented Mar 31, 2020 at 17:09
  • 3
    The fact that mods have to manually clean up the H&I queue at all indicates a larger problem with this process. The heart of the matter is not user behavior, it's a flaw in the way the queues operate. Banning reviewers is ineffective and always will be (although it's quite effective at alienating users) because it treats the symptom and not the problem. Add a mechanism to H&I to systematically remove hopeless posts (ie. make it part of the mechanics of how H&I works) and this problem goes away.
    – Z4-tier
    Commented Apr 15, 2020 at 3:06
  • 9
    @RobertLongson obviously, there are better options that don't involve great programmers leaving the website. I doubt that someone with 65.5k rep is that bad at sorting posts, and if it is the case then surely that person can be explained how to improve.
    – Nakx
    Commented Apr 24, 2020 at 7:38
  • @Nakx They clearly are bad at sorting posts because they've not read the instructions. So yes they need to be told what they are doing wrong and I'm confident once they realise that, they will do it right. Having high rep means correlates with being able to ask and answer, not necessarily review without understanding which button to press in each circumstance. Commented Apr 24, 2020 at 7:45
  • 9
    @RobertLongson Banning someone without any warning is not in any way an appropriate method to notify them of an honest mistake.
    – BrtH
    Commented Apr 28, 2020 at 16:12
  • 1
    @BrtH and that's precisely why this question exists, it suggests a number of mechanisms to educate rather than ban people. Commented Apr 28, 2020 at 16:17
  • 4
    @RobertLongson Your wording made it seem to me that you agree with the current policy of banning users. My apology if that is not the case.
    – BrtH
    Commented Apr 28, 2020 at 16:19
23

Can we just remove the Requires Editing button in Triage in the meanwhile?

It's so confusing and more often than not it's not useful.


Also in Help & Improvement more often than not I want to downvote it or vote to close. Since it's not on-topic but there are only edit/skip/VLQ option there.

10
  • 12
    The only way a post gets into the Help & Improvement queue is if it gets a "Requires Editing" result in Triage (see this answer for how the queues work). Removing that button from Triage effectively shuts down H&I. From what you've written, it doesn't sound like your intent is to shut down H&I. Are you aware that's what would result from the implementation of your suggestion?
    – Makyen Mod
    Commented Mar 10, 2020 at 6:48
  • 2
    Well, I agree with this just because I believe H&I seems to be rather useless queue, the wrong solution to the problem. So let's get rid of button and queue.
    – yivi
    Commented Mar 10, 2020 at 7:14
  • 4
    @yivi, Then what's the point of Triage? That can all be taken through the queues that existed previously, and we just accept the fact that questions that had a chance for improvement and might have become great questions just vanish into oblivion (along with the contributor) like before.
    – ouflak
    Commented Mar 10, 2020 at 7:42
  • 3
    @ouflak If the "point of Triage" is to push question into H&I so we can discover potential great questions that need polishing, we should first question if H&I is actually fulfilling that purpose, or it's a good way to go about this. I think that most edits and polishing are performed by tag watchers while they are looking for stuff to answer, for example. Triage would be good not to find pearl but to help in throwing away sand (which makes easier finding pearls in the regular course of business)
    – yivi
    Commented Mar 10, 2020 at 8:03
  • @yivi, Yeah the discussion was quite some time ago, so my memory might be fuzzy on this. But I thought the motivation was trying to comb through potentially good questions (and contributors) to get them to the standard that a question on SE should be to generate good answers and useful information. This was to be done by some means. These queues just happened to be one attempt to make that happen. We can abandon the queues. But the idea is still a worthy one. And I agree with all of those who say (and said) we have to do something or else face the site dying.
    – ouflak
    Commented Mar 10, 2020 at 8:11
  • 1
    @Makyen Yes, I aware of that. As yivi said most edits come not from H&I queue but from tag watchers, which explain the low number in the screenshot. VLQ in H&I just return the post into the Triage which are pointless if on Triage it would be send back into H&I.
    – Mukyuu
    Commented Mar 10, 2020 at 8:54
  • 16
    Shutting down H&I is a feature, not a bug, @Makyen...
    – Cody Gray Mod
    Commented Mar 10, 2020 at 9:22
  • 4
    @CodyGray I was trying to not take a position as to shutting H&I down being a feature or a bug. It just appeared that Mukyuu both wanted to, effectively, shut down H&I (without saying shut it down) and wanted changes to H&I so people can downvote and/or vote to close in H&I. Making changes to H&I would be useless if it was shut down. Thus, it appeared this answer was wanting two conflicting things, which makes the OP's intent unclear, which could be clarified with an edit changing the text to say something like "if you're not going to shut H&I down, then could we have these changes: ... .
    – Makyen Mod
    Commented Mar 10, 2020 at 10:13
  • 4
    @Makyen Less code to maintain, and remove a queue that nobody can use? Kill two birds with one stone!
    – S.S. Anne
    Commented Mar 10, 2020 at 11:25
  • I wrote a ~10 line user script for greasemonkey that just hides the Requires Editing button. Now i never have to worry about this problem (and sure, "that's not how the system is meant to work!" may be true, but the system. is. already. broken)
    – Z4-tier
    Commented Apr 14, 2020 at 16:29
18

This is probably controversial but I would like to see Triage and Low Quality Posts merged and have Help and Improvement, First Posts, and Late Answers removed/merged as well.

Triage and Low Quality Posts are incredibly similar from the point of view of a reviewer. You determine if the post is of sufficient quality and if it isn't vote to close it. If it looks ok but needs formatting changes you hit edit otherwise you say it looks ok. If you are unsure you skip. That makes perfect sense to me.

The only real difference between these queues is that you can't actually edit the post in the Triage queue. This seems dumb to me. The only time you should click requires editing is when the post has formatting issues: any issues with the question itself that only the asker can fix must be handled with the close votes. If you can identify formatting issues, you can fix them, so it makes no sense that it gets put onto another queue when the current reviewer could have handled it already.

Having separate queues for Triage, First Posts, Low Quality Posts, Help and Improvements, and Late Answers seems a bit pointless to me. Most of these queues have very few posts in them especially on other stack exchange sites and they all do very similar things. Late Answers and First Posts both allow reviewers to vote and comment on posts as well as editing and flagging, and Help and Improvement allows editing and commenting.

Why separate all of these queues when the review process is pretty much the same for all of them? In all of them the goal is to improve posts that can be edited and provide guidance to the poster where necessary. Surely it would be better to have one queue that encompassed all of those requirements and allowed votes, comments, editing, and voting to close if needed with the goal of rewarding good content with upvotes, improving content that could be improved with edits, and filtering out content which is unsuitable with close votes.

I personally think that the following queues make sense:

  • Close votes - to confirm that posts should be closed.
  • Reopen votes - to confirm that posts should be reopened.
  • Suggested edits - to validate edits by users.
  • Triage - to handle posts from new users, low quality posts, late answers, and anything else that is potentially poor quality.

The new triage queue would have "Looks OK", "Edit", "Recommend Deletion", and "Skip". Vote to close would put it onto the close vote queue. Edit would allow you to edit the post inline just like the Low Quality Posts queue as this would make it absolutely clear that you should only click edit if you can edit the post. Voting and comments could be enabled as well so reviewers could leave feedback on the post as needed and can upvote good posts.

I feel this would allow us reviewers more freedom to handle a post in the way that we know it should be handled rather than in the way the review queue forces us too. I'm always annoyed when I want to upvote or comment on a post in a queue which doesn't allow for those actions or when I want to edit a post to fix the obvious formatting errors but all I can do is mark it as requires editing and hope someone else does it. The improved triage queue would be focused on improving already good content via editing and direct feedback (with votes and comments) and would filter out bad posts using close votes.

6
  • 3
    Not disagreeing, but shouldn't there be a "Needs to be deleted" button? Vote to close works only on questions. LQP doesn't close but delete.
    – BDL
    Commented Mar 11, 2020 at 10:18
  • 1
    Good point. Should probably be "Recommend deletion" instead then? Or it could change based on whether it's a question or answer? Commented Mar 11, 2020 at 10:21
  • I agree, except low quality posts should still be a separate queue. It serves a completely different purpose. First posts, triage and help & improvement should be merged.
    – Dharman Mod
    Commented Mar 12, 2020 at 1:18
  • I don't see how LQP serves a completely different purpose. It has the same options as Triage and pretty much everything in it could be removed. IMO the triage queue should be the queue where everything goes to determine whether it needs editing or deleting so a reviewer knows that their goal is to improve the quality of the post or recommend it be removed from the site. You don't need to learn how each individual queue works as you have all the tools you normally have on any post on the site and anyone that understands how SO works would understand how the Triage queue worked. Commented Mar 12, 2020 at 9:37
  • I also think flagging and close vote systems should be adjusted to make things a lot less confusing.The flagging options need to remove the duplicate and closing options and should just be used to either report inappropriate content or flag the post as problematic (to put it on triage queue). I then feel the close votes should be opened up to everyone on the site but should only count towards the voting requirements if you have enough rep. Instead of duplicating the close into flags, keep it where it should be and handle it differently behind the scenes based on what privileges the user has. Commented Mar 12, 2020 at 9:52
  • I don't think "Recommend Deletion" is very good. "Recommend Closure" would better reflect what happens. I believe there's a natural reluctance to nuking stuff, just because it doesn't quite meet the mark.
    – Scratte
    Commented Mar 18, 2020 at 23:35
16

I recently took time to study triage items and as far as I could tell, the system can (or rather, should) be tweaked (with reasonably small development effort) to help inexperienced reviewers do more useful work.

I checked about 350 triage questions in the last 10 days (mostly using skip because I prefer to invest my close votes elsewhere) and what struck me the most was how few questions felt like they were in the need of "Requires Editing".

These were mostly cases with reasonably sensible text and glitches with code formatting. These really strongly felt like in the need in H&I review, however these were very very rare.

My guess is this is because most such cases are handled outside of review, by users who organically watch questions in their tags (these were probably originally planned to be hidden in tag pages but since this feature implementation was abandoned midway triaged questions are visible in tag pages, I just re-checked that).

For the sake of completeness, there was another (also rather infrequent) category of questions that I would prefer to pass to H&I, these that were overall okay but might be improved by removing greetings and salutations. But frankly, for these it somehow felt... tolerable even if they would be slipped through triage by skips and looks-ok votes - probably because these were mostly the questions that really looked okay otherwise.


In my studies however things changed when I figured that most reviewers will be likely less (much less) experienced than me. I tried to imagine how it would feel for a user who maybe hasn't even cast a close flag yet.

And that perspective changed things to basically opposite. I wrote above that "requires editing" was needed very rarely - forget about it: for an inexperienced reviewer it looks like the right way out in very very many cases.

Time and time again I made myself notes that particular review would look like "requires editing" for an inexperienced user even though someone like me would pick another option in a heartbeat.

It was like, everytime I made an effort to forget about my tens of thousands of close votes and imagine how it would look like for a newbie reviewer, it very very often ended in "hey, this looks like requires editing, isn't it".


Given what I learned above I think it would be better to disable Requires Editing until user completes 20 (better, 50) reviews. As I wrote above, this option is needed very rarely and system would better change to teach newbie reviewers use more frequently needed options before opening the one that they would (incorrectly) use instead.

Technically, this could be implemented by simply deleting or graying out "Requires Editing" button for users who didn't reach the required milestone. However I personally would prefer for it to stay enabled but (for non-audit reviews) pop-up a modal window explaining to reviewer why it is not available to them and when they are expected to get it enabled.

6
  • 8
    I'm not sure 50 is enough. I think the Triage queue is better suited for experienced reviewers. It appears to be a queue where things can be determined quickly, but in reality, very few posts can for inexperienced reviewers. Even with my 300 reviews, I cannot quickly make an accurate judgement of a post. Unless it's a very clear case. The trouble is that every post looks a like another corner case.
    – Scratte
    Commented Mar 20, 2020 at 14:04
  • @Scratte I wouldn't be surprised if more that 50 will turn out optimal although I wouldn't bet on that. I expect milestone reviews amount to be a configurable parameter that can be changed if needed so I'd rather prefer to try it out with whatever value company developers pick, and worry about possibly better settings later
    – gnat
    Commented Mar 20, 2020 at 14:11
  • 2
    When I used to review, I reviewed 600+ posts in Triage and I don't feel I was using it properly even then. I didn't ever trip over an audit that I can remember, and never got banned. I'm not hopeful that just temporarily disabling the button is a solution.
    – hat
    Commented Mar 20, 2020 at 14:17
  • I primarily addressed things that can be done quick and cheap given that this discussion asks for "urgent fix". Triage probably can be improved futher but this was the only thing that stood out in that regard above al other things that I considered
    – gnat
    Commented Mar 20, 2020 at 14:27
  • 1
    How about instead of 50 reviews as the threshold for enabling the requires editing button, the threshold would be tied to the reviewer having actually successfully performed some edits?
    – Stephen Rauch Mod
    Commented Mar 20, 2020 at 23:25
  • @StephenRauch I think your idea has merit, I considered something similar myself when drafting this answer (dropped it in favor of simpler way only because this question asks for urgent fix). I expect that we will eventually find and use more efficient criteria than plain count of prior reviews, it will just take time
    – gnat
    Commented Mar 21, 2020 at 7:23
16

I think one of the main issues here is that people are encouraged to review/triage without giving them a good enough introduction on how to do it properly.

My story

I personally wanted to try out the triage feature and it seemed clear enough to me:

  • Requires editing: the post needs to be edited for more information (by the author)
  • Unsalvageable: the post is spam or not related to the site

Unfortunately, this is not how it is supposed to work at all, and I only found that out when I got review banned.

Additionally, I did not even figure out I was review banned until I manually went to https://stackoverflow.com/review after researching why I have a "bug" that it shows me new reviews are available (the red dot) but then it only shows "There are no review queues available to you". I kept wondering why that bug is happening all the time, and why it keeps showing me the red dot.

I finally went to meta stackoverflow to find out this is a review ban (which most people probably won't even end up researching). And even after I found this out, it took me some more researching to figure out how to find out which review caused the ban (I initially looked into the "Reviews" tab in "All actions" on my profile, there is nothing related to bans/problems with reviews there).

The improvements suggested here should solve the problem of users not knowing when they were banned.

Edit: I also noticed that the red dot keeps re-appearing although there is no way to access the review queues. It would be better if it goes away after you have seen the ban reason.

My suggestion

I totally agree with the suggestions put forward here, and I'd like to add that from a UX perspective it would be better if you:

  • Still encourage people to try reviewing out, but give a tutorial first.
  • Stage 1: Ideally, use a couple of example posts (where no real reviewing happens!), so that users can practice triaging. If they do it wrongly, tell them what they did wrong and how to improve. This would be an automatic process.
  • Stage 2: After users have practiced on enough posts, start showing real posts that others have already reviewed as well. Only let them do a handful of posts here so that moderators can check if they did it correctly, if not, let moderators send them a message on what they did wrongly. This is a manual process, but because each user can only review a handful of posts at first, it will reduce the amount of work.
  • Stage 3: Only after users have finished the tutorial (practicing on example posts and real posts), let them do triaging for real. This is where all users start, at the moment. If they still do it wrongly, you can resort to bans at this stage, of course.

I am sorry that I caused more work by wrongly triaging items, but I thought it was clear what to do, but obviously that was my wrong and naive interpretation of the wording.

4
  • Your initial understanding of what's expected from a reviewer doesn't seem to be far off. What went wrong?
    – yivi
    Commented Mar 24, 2020 at 12:38
  • 9
    @yivi I triaged some posts as "Requires editing" as I thought "Unsalvageable" is a bit too harsh and that the posts could be useful if the author adds more information on their issue. They weren't spam or questions like "is X or Y better?", and I did not want to turn away new users by saying their post is "Unsalvageable". It turns out that some of them should have been triaged as "Unsalvageable", which I now understand after reading the triaging guides, but this was not initially clear to me.
    – omnidan
    Commented Mar 24, 2020 at 12:41
  • 1
    So, how do you figure out which review caused the ban? Commented Apr 10, 2020 at 13:58
  • 2
    @MarkStewart you have to manually go to https://stackoverflow.com/review, there is no way to access it directly from the interface as far as I know...
    – omnidan
    Commented Apr 10, 2020 at 14:05
15

Funny, I always thought I'd get review-banned for clicking "Unsalvagable" instead of "Needs Editing". Turns out it was the other vague and subjective distinction that I should have chosen all this time.

This is chasing away all the people that SO needs the most.

5
  • 10
    People reviewing incorrectly means we need lots more people to clean up after them. While we're not trying to chase them away we really don't "need them the most" while they continue to do that. Commented Mar 29, 2020 at 7:25
  • 14
    If the workflow of SO is such that it takes "a lot more work", then maybe the workflow needs improvement. Stop taking out your frustrations on well-meaning reviewers. Don't bother replying -- I'm done with SO's internal culture. Commented Mar 29, 2020 at 16:19
  • 8
    Hence the existence of this question. Commented Mar 29, 2020 at 16:22
  • 5
    Good thing they applied the ban then, so you could be made aware of the proper way to continue reviewing. Now that you know what's what, you'll be able to continue reviewing, surer in the knowledge that you are actually helping, and much less likely to be review suspended in the future.
    – yivi
    Commented Mar 29, 2020 at 16:44
  • 8
    Subjective is the key word here. And apparently if someone else's subjective choice disagrees with yours... banned! Seems one should just not waste time or effort reviewing things.
    – Tyler V
    Commented Apr 11, 2020 at 3:16
12

There's something that is really bothering me about this post so I feel like I must post this.

You have stated the numbers: 118 bans from failing review audits and 2118 manually banned. You also mention that you personally ban an average of 200 reviewers per day. You also mentioned that you are the only moderator who is handling this issue.

This tells me that you are personally responsible for a significant portion of the 2118 people who were manually banned. Now personally I think moderators are all smart and hard working people (your dedication to this cause does outline this as well), so having this in mind I think the fact that you are the only moderator who manually adds so many bans per day means that there really is no consensus about whether or not there actually is a problem. If there was consensus then those 200 (or more) would uniformly come from multiple moderators if others also felt that the bans were warranted.

I don't think you can make the argument that the triage queue needs fixing based on the number of reviewers banned. This is because you have personally ensured the number of reviewers banned follows your argument instead of the other way around. In addition, given the number of bans it is now impossible for anyone to audit each ban and determine whether or not they were really warranted and are genuine or might be over-reacting or unwarranted. The number of bans seems to be central to your premise but however is not admissible because of this reason. The link to the search for triage related issues has more complains about unfair bans that it does about the triage actually need fixing which might indicate that the number of bans are high because of over-banning.

Now don't take this as a personal attack against you as a person, I don't know you nor do I have anything personal against you. But it exactly because I don't know you that I feel that this is a critique that needs to be made. Now I'm sure this will not really get much attention and will probably also get a lot of negative feedback but I think it would be a mistake to not take it into consideration.

2
  • 5
    many triage bans made by Sam were thoroughly checked in chat room linked from this questions and in multiple meta threads related to particular cases of bans. Amount of mistakes discovered during these checks seems to be very small so it looks reasonable to assume that most of his bans are correct. As for him being the only mod doing this permanently, per my reading of some details about this it is just a (lucky for us) coincidence that currently he has enough time to put on that and other mods would do the same if they had enough time
    – gnat
    Commented Apr 1, 2020 at 13:16
  • 3
    Precisely. What the triage ban system serves as is an excellent way of driving users from the site for wrong-think. Commented Apr 16, 2020 at 2:13
12

I have reviewed a few hundred triages over the past years and it's the first time that I got review banned. I see why this happened and I will continue review posts after the ban ends. However, here are a few things I want to point out from the perspective of a not-so-hardcore SO user:

  1. I usually only enter the review queue when there's the red overflow indicator. The only reason why I'm doing triage is because I want to help out the community that has basically taught everything about coding and allowed me to grow from a complete beginner to the most-respected coder in my school.
  2. When I saw the red indicator this morning and the dropdown says "no review queues available" I genuinely thought there's a bug in SO and stumbled into the ban message when I tried to figure out what happened with the intention of filing a bug report.
  3. I never knew there was a Help and Improvement queue and that choosing Requires Editing will send the post to a different queue until now.
  4. I read the entire review queue guideline document several times over the past years. When I just started working on the triage queue I always chose Unsalvageable for problematic posts. Then when I looked at my flagging history I found that about half of the posts I flagged for closing were disputed as Requires Editing, and that made me wonder if I've been too critical and started sorting questions to Requires Editing category (which was what lead to this ban). I admit I should have gone back to the guidelines when I had that question but I didn't.
  5. For posts that actually fit the definition of Requires Editing I usually just edit them when I see them instead of flagging (so this means I'll probably never choose Requires Editing in the future.

I think one possibly easy-to-implement improvement is to make reviewers who do not have access to Help and Improvement queue aware of the existence of H&I. Maybe add to the review queue dropdown something like "this action will send the post to a different queue so choose it carefully" to the explanation for Requires Editing. For me the problem isn't that I didn't read but I never knew that. As I mentioned earlier, I only review posts because I wanted to help alleviate the stress of the community and I would have never chosen "sending this post to another queue that's already overflowing" if I knew that's what would happen. To the best of my knowledge this is not documented anywhere (I can't confirm it now because of the ban). I think many reviewers with the same intentions will be benefitted a lot if this is made obvious to them.

4
  • If you review incorrectly you aren't helping the community, you're hurting it because we need to clean up after you and that requires several people to sort out each mistake you make (or one moderator but they are in short supply). That's why we need to stop you and get you to do it right. Now that you know what to do that's great, the ban worked in this case despite being a blunt instrument. Requires editing is a rare thing to want to select, it means the post was almost good enough and a small tweak to formatting will fix it. A mod may see this answer and unban you if you're still banned. Commented Apr 21, 2020 at 23:11
  • 6
    @RobertLongson I get that. I'm just saying one quick fix for the problem is to make it obvious that there is a help and improvement queue and, as other people stated, phrase it better so people know this button will not send a post back to OP.
    – Mia
    Commented Apr 21, 2020 at 23:21
  • Indeed, that's the entire reason this question exists. Your comment is one of many steps towards a solution. Commented Apr 21, 2020 at 23:24
  • 2
    Thanks for the feedback, the review ban is now lifted
    – Samuel Liew Mod
    Commented Apr 21, 2020 at 23:41
12

I got recently banned for a long period of time and thought this was the first time, here is why:

  1. The icon that mentions you are banned looks like you are notified there is a need to help the review queue. I kept clicking on it and wouldn't understand why nothing was showing up, so as soon as the queues reappeared I reviewed all I could thinking that would help. At least that is what I now assume it means. enter image description here
  2. When you click on the list, it says the review queue is unavailable to you, not that you have been banned. enter image description here
  3. I just wanted to help, but it is hard to judge posts outside of your field, I have been banned on triage of posts from languages I never use. I suggest the review queues could filter posts and sort them so that you review the posts from the tags you are watching first.

Two other things I have in mind:

  1. Giving a badge for reaching a certain amounts of edits, reviews etc is probably not a good idea if people pursue these as goals, rather than try to improve quality.

  2. Banning people who try to help is highly discouraging and a long and painful tutorial would be more helpful (e.g. having to sort out previously sorted posts until >95% correct choices are made). If you really want to help people see the ban, banning them for several months is not useful and feels patronizing. Rather, users should be banned for the next n days where they have connected to the website, with a notification showing at the top of the threads.

1
  • 3
    Thanks for the feedback and response, I have lifted your review suspension.
    – Samuel Liew Mod
    Commented Apr 24, 2020 at 7:45
10

I reached 500 rep yesterday and was given the Review Queue privilege. I was also banned from the Review Queue for use of the deadly Requires Editing button in Triage the same day.

Something is Wrong!

I was shocked to see that I had been banned, and when I read the reason, I began to understand why people hate SE. It was something to the effect of: "You clicked "Requires Editing" when you should have clicked "Unsalvageable". This question has been closed. Come back in May." I was thrilled.

In my mind, Requires Editing means that the question needs work. Unsalvageable means that the question is total garbage and needs to be closed.

The exact question I was banned for was: Configure and costomize Apparmor Linux Debian. As soon as I read the question, I realized that it was off-topic for Stack Overflow and need some grammar fixes, but it could be Migrated to a different SE site. My reasons for selecting Requires Editing are as follows:

  • I cannot vote to migrate
  • BUT: The post was not so bad that it couldn't be answered on a different site.

There was even a comment suggesting that the post be moved to a different site. As I did not understand exactly how Migration works, I thought that Requires Editing was an acceptable option. Funnily enough, on that very question was a banner like this:
edits are possible
With a link to... GUESS WHAT? An edit function. The edit option is still there! Hence why it Requires Editing.

Long story short:

Yes, there is a problem with the way Triage Editing and bans are right now. There needs to be some better instruction and wording on the buttons. For starters, it would be helpful if in the Review Queue icon, instead of saying "There are no review queues available to you", it said that you have been banned and why. Just so that you remember to stay out of triage when you come back, and so that you learn your lesson. It would also be handy to have warnings for one-time offenders rather than automatic bans.

5
  • 4
    If the way to salvage a question is to migrate it, "requires editing" is always a bad choice no matter how you understand the button. The other points you bring up, have been already brought up in many answers, and the question itself. Also, there is already an ongoing project to improve the review queues experience. In the meantime, the review suspension is there simply as a way to inform users that they are not reviewing correctly, so they can be aware of it and change course in the future.
    – yivi
    Commented Apr 28, 2020 at 15:48
  • 1
    Once a mod sees this, it's likely your review suspension will be lifted.
    – yivi
    Commented Apr 28, 2020 at 15:51
  • 1
    @yivi Thank you. I know I'm not really adding much to the topic, mostly expressing my frustration at a very quick ban. Commented Apr 28, 2020 at 16:12
  • 1
    @Noah the review suspension is lifted now... Thanks for reviewing and for your thoughts.
    – Jon Clements Mod
    Commented Apr 28, 2020 at 16:58
  • 1
    @JonClements Thank you. I'll keep out of Triage until this issue is resolved. ;) Commented Apr 28, 2020 at 16:59
9

I have to apply manual bans and pray hard that the users chance upon the review ban message, WHICH IS NOT ALREADY APPARENT AND THEY GO ON TO DO THE SAME THING AGAIN AND AGAIN.

I was given 1 manual ban by you, and it helped me pay closer attention.

The fact no one has mentioned is: a lot of reviews on the Triage queue are remarkably nuanced. You have to take quite a few factors/criteria into consideration to effectively review (not counting posts that by their content, are borderline).

And it also requires "practice". When I got back from a 4 day suspension it seemed I had forgotten half the guidelines triage reviewing requires. (IMHO this contributes toward explaining why some users almost immediately fall into the same mistakes.)

3
  • 3
    One? You probably didn't see the first one on Jan 7 and got away lucky with a short 4-days duration the second time round on Feb 17
    – Samuel Liew Mod
    Commented Mar 12, 2020 at 4:19
  • @SamuelLiew are you sure it were only 2?
    – bad_coder
    Commented Mar 12, 2020 at 5:22
  • 5
    And one review audit ban. Three in total but you were mentioning manual bans.
    – Samuel Liew Mod
    Commented Mar 12, 2020 at 5:50
6

Interim Solution Proposal

I'd like to propose an interim solution that should require relatively little effort but result in reduced incorrect triage. I haven't read all of the other answers in this and related questions and I apologize if this suggestion is redundant.

Could you please modify just the text of the existing landing page for Triage to include this additional message:

NOTE: If you are new to Triage, it is imperative that you review the appropriate documentation in order to make correct triage choices. Making incorrect choices generates unnecessary downstream activity and may lead to a temporary suspension of Triage privileges.

This should have an immediate positive impact for all involved.

Thanks.

6

I have just been review banned and quite frankly I felt a little dispirited. After taking some time to read all the MetaSO posts, questions and answers (like this) related to Triage, I agree with why the ban is being applied. But I think there's a disagreement with the workflow the ban is being placed.

I understood this is a common issue and is being considered to make changes to. So on that note i have few points that I would like to add that i feel should be considered.

Getting a ban can mean two things,

  1. The User has done something wrong.
  2. The moderator doesn't agree with the user.

Now, there needs to be different workflows to identify these issues and address them. I feel that even if a moderator is proposing for user to be banned, there need to be an automated process to check the following:

1. Is this the first time the user is being banned?

  • If yes, then the ban should be converted to an automated ban for a couple of days or a week with a warning/notification to the user. And the ban should start from the time the user logs in first after the ban is passed.
  • The user should also be provided with a set of web-links to posts or tutorials on the guidelines.

2. For the specific triage the user is being banned, what is the majority vote?

  • If what the user has chosen is not the majority, it means that the user has done something wrong. And the ban will be justifiable.
  • If the majority is the same as what user has chosen, it means that there's a disagreement between the moderator and the majority users. This needs to be addressed further, like giving the user an option to justify themselves why they think it Requires Editing.
1
  • 5
    The majority opinion doesn't matter here. The site rules do. Moderators are elected to enforce those. Reviewers are also expected to enforce them. When reviewers make decisions that are in conflict with the site's question requirements, then those reviewers' privileges will be suspended.
    – Cody Gray Mod
    Commented Apr 12, 2020 at 3:14
6

A lot of posts are focusing on the Triage queue, but that is really only one piece of the larger problem. The real issue here is that posts are getting stuck in H&I and there is no "by design" mechanism to remove those that may not actually belong there. So there are 2 options:

  • Make it impossible for posts to be incorrectly sent from Triage to H&I (not realistic)
  • Add a mechanism to H&I that allows hopeless posts to be removed (easily implemented)

The only way to implement the first option is to completely remove the Requires Editing button from Triage, but doing that obviates the need for H&I since it's the only ingress point for that queue. No amount of instruction / warning / review-ban / rep requirement is going to be able to completely prevent this problem.1

So that leaves the second option: add a check valve to H&I so these posts can be removed without moderator intervention. How about just adding Unsalvageable as an option in H&I, similar to Triage? There are lots of good ways to implement it that would both remove the burden from the moderators and improve the quality of the reviews without punishing reviewers (send it back to Triage... notify the original reviewers... send it to users with high scores for the applied tags... make Sam Liew fix it (wait not that one)).


1: I am not implying that these options should therefore not be considered; they will absolutely help. What I am asserting is that no combination of "soft interventions" will ever be 100% successful here.

5

EDIT: I see you've actually taken my suggestion and added to your list (sort of. IMO, you've made it even better). My only problem is it makes this post kind of useless ;). That's certainly a problem I can deal with, though. I might delete this later if I feel like I have nothing more to add.


Good suggestions. I have an addition to your list that I think will also be a quick win:

  1. Change the "Requires Editing" button language.

Now's the part where I would normally make the case that the "Requires Editing" button is ineffective with that wording, but you've already done an excellent job of that already by linking to the meta posts that I would be linking to anyway, so I won't repeat it here.

What I will say here is that I'm sure the person who came up with it had good intentions. I don't fault them at all. It's just that it didn't go far enough to differentiate between requires editing from the OP vs. requires editing from someone else. I don't think the problem here is with the button itself, it's just the final wording has made it fail its job. Ideally, the button would have some language that makes clear its intended use and doesn't require a guide to use properly. It should be self-evident.

I don't think a fix here has to be hard here. I propose making the button say "Has formatting or grammar issues" instead. Okay, that's a bit of a mouthful to put on a button, so maybe something like "Has grammar/formatting issues" would work instead. But you get the idea. Just something that makes it more clear to users jumping into it what it's supposed to do.

The good news is I would think that this sort of window dressing wouldn't be hard to implement (it's just the wording on a single button), and it would go a long way to make the button self-evident to use. So that's why I think it's an easy, quick win.

2
  • 2
    I was thinking about something like that too, but it would also be essential to somehow communicate that the question is also on-topic and answerable. Otherwise, people may will click the button for unsalvageable questions which are also terribly formatted (which are very common). Commented Mar 11, 2020 at 2:33
  • Excellent point I didn't consider. Might have to give that one a think and see if there is something better I can suggest.
    – user10957435
    Commented Mar 11, 2020 at 2:34
5

Given this announcement on Meta Stack Exchange they're going to make serious work on some of the points:

2. Retain a link in the top bar review queue menu for users in review suspension

In the event of a suspension, a user will be notified in the Review Queue dropdown. A post notification will be available with reason and time remaining on the Review Queue main page, along with guidance for continued learning in this area. We’re also proposing softening the language by renaming “review bans” to “review suspensions.”

(source)

4. Implement a compulsory tutorial/guide when users gain the privilege to perform reviews

Users will find an onboarding message on the homepage with general information about using the Review Queues. As they enter each new queue, they will be presented with a modal with unique instructions on how to best contribute to that queue.

(source)

'unique instructions' implies Triage will get its own specific guide.

5. Change the text of the action buttons from "Requires Editing" and "Unsalvageable" to avoid confusion

We want to remove the Requires editing action and replace it with two new actions. Selecting Improve readability will send the post to Help & Improvement. Posts that can only be salvaged by the original poster (Add clarity) will be sent to the Close vote queue.

(source)

3
  • 2
    FINALLY! WE DID IT! Commented Apr 23, 2020 at 20:48
  • @zixuan: Did what? Commented Apr 28, 2020 at 17:43
  • The compulsory tutorial/guide could be gamified or even be a game, like in some computer games where the first missions act as training/learning the rules/system. Commented Apr 28, 2020 at 17:43
4

Adding the 'Edit' option for Triage that is normally accessible when viewing questions and answers, including from review 'First Posts' and 'Late Answers,' I think could help alleviate the problem.

As 'Requires Editing' should be clicked when the reviewer thinks that the post can be appropriate with editting, the reviewer may be able to make the edits himself. However, there is no option for him to edit the post itself unless he wishes to search for the post manually before or after he triages. Having to manually find the post in triage, which isn't as easy as with the other reviews, is offputting. It can also safe effort from the moderators if reviewers can make or suggest the edit themselves.

It can also help reviewers decide if the post should be selected as 'Required Editing' or 'Unsalvageable' if they can actually attempt to edit the post into something that would warrant 'Ok.' This could be seen as testing the 'Required for Editing' requirement with a hands-on method instead of simply reading. For example, this recent post which I had previously (and genuinely) thought could be fixed with editing from someone other than the person asking it (I thought the information in the question and the code could have been used to clarify the text that should have more clearly explained the question). However, when going back to it afterwards the next day following a review ban, and attempting to edit it manually, I realised I couldn't adequately edit it, and a 'Unsalvageable' or 'Skip' was more appropriate.

3

I have a hard time believing that implementing a required tutorial would be any less work than adding a reason for Requires Editing that auto-generates a comment. (As I proposed previously.) Auto-generating a comment would be the quick and dirty (but potentially effective) way of capturing the reason, without having to build a dedicated interface for it. Having to choose a reason for Requires Editing would likely reduce misuse significant.

3
  • 6
    That would not help matters in any significant way, IMO. The problem is that users are misusing the "requires editing" button. How would auto-posting a comment fix this? Or you meant that selecting from these list of reasons would have more than one entry that would convert the action from "requires" editing to "unsalvageable"?
    – yivi
    Commented Mar 10, 2020 at 13:38
  • 5
    Also, questions that need polishing can require it for a variety of reasons. Syntax, grammar, code formatting, links, etc, etc. If several apply, would a "choose your reason" would make you select one of many. Maybe you could add to your answer a draft of what these "reasons" would be, and what would happen when chosen?
    – yivi
    Commented Mar 10, 2020 at 13:41
  • 1
    @yivi Yes, selecting a reason would reduce misuse since they would get a better understanding of what the button is supposed to accomplish, and it need not restrict anyone to a single choice. The reason also need not affect any existing behaviors of the site; the generated comment would just be for recording the reviewer's findings so someone else could more quickly identify the problems. I detailed most of that in the linked answer.
    – jpmc26
    Commented Mar 11, 2020 at 5:35
3

I'm not sure if it's late to respond here. But the confusion that 'Requires editing' creates is who is supposed to edit the post. It states, anyone on the site should be able to edit the question to make it more acceptable.

Now let's take an example, a new user posts 'screenshot' of his code. Am I supposed to select 'Looks OK' or 'Requires Editing'. But if selected the 'requires editing' option, that basically means someone on the site just typed it out from the picture. Can't mark it as 'unsalvagable' because rest of the question format is fine.

In my opinion, we should have the following 4:

'(1) Looks OK:' Yes, when there is no issue.

'(2) Requires Edit:(From User)' This is to be selected when the person who asked the hasn't provided complete information regarding the question. For instance, missing logcat, or the error the user is facing. When a reviewer marks it as 'Requires Edit' there will be an option for the reviewer to type a short message (lets say 100 chars) of what is missing and this message is sent to inbox of the question asker so that he can add the required info. This privilege ofcourse can be assigned to high reputation [trusted user in short] people. If the reputation isn't high enough the message should be reviewed by moderator anytime.

Well guess what this could just become a new review-queue for high reputation people [5k+ maybe?]. Review personal Messages sent queue.

'(3) Instant Edit:' Now this is where anyone on the site can edit the question/answer correcting some minor issues like typing errors, replacing image instead of links, formatting text (I've seen many where the code isn't formatted well and the error is embedded in quotes). This option will open the editor for the post and the reviewer can directly make the changes.

'(4) Unsalvagable:' This should remain the same and should be used only when the question is asking for ready-made stuff like "Hey how do I make a chat app?" and not asking for a specific issue, the language is not English and when it seems even asker cannot add anything to question that will make the question sensible.

[Note: Requires Edit (from user) should be disabled if reviewing an answer. I don't think answers would be incomplete and miss significant details.]

This will sort of convert 'TRIAGE' to 'QuadAge', but sounds good update. 🤩 Yeah, I have definitely considered wrong meaning of 'TRI-AGE' here. 😓😓

5
  • 3
    You absolutely can and should choose unsalvageable if code is only provided as an image. The OP can then fix the question by replacing the image with text, the question can then be reopened and answered. If an answer is complete but only has spelling and grammar issues then by all means check requires editing, that is exactly what requires editing is for and yes it's pretty rare that you'd choose it. Commented Apr 29, 2020 at 6:11
  • @RobertLongson thanks for the information. I got my review ban (2 months long) for triage mistake. For the very same reason actually. Any opinion on my concept?
    – Dharmaraj
    Commented Apr 29, 2020 at 6:25
  • We already have "Requires Edit (from the user)". Those are questions that need to be closed. The label says "unsalvageable", which is a lie. Those can sometimes be salvaged by the post author. But that''s the button you need to choose for those cases.
    – yivi
    Commented Apr 29, 2020 at 6:44
  • 2
    Review suspension lifted. I strongly suggest you do not click "Looks OK" and "Requires Editing" in Triage.
    – Samuel Liew Mod
    Commented Apr 29, 2020 at 6:47
  • @SamuelLiew Thank you. I will go for triage when I'll be free so that I can spend more time on a question. :-)
    – Dharmaraj
    Commented Apr 29, 2020 at 6:49
2

I would assert that the problem here is that the options do not reflect all possibilities. Describing a post that requires additional information from the asker as 'Unsalvageable' is plainly incorrect. It can be salvaged - by being improved by the person asking the question. Required editing, to anyone who hasn't read the instructions doesn't describe what is meant, which is what is called a minor edit on other sites. Instead the options should be something along the lines of:

  • Looks Okay
  • Requires minor edit
  • Needs to be rewritten
  • Unsalvageable

So that questions that do belong on the site once they have the relevant information can be improved and then answered, and the spam etc. separately put in a bin and disposed of.

8
  • 2
    If we allow major edits, everything becomes salvageable. What we mean by "unsalvageable" is basically "unsuitable for this site in its current form". Maybe we should just label the button that...
    – Cody Gray Mod
    Commented Mar 12, 2020 at 16:05
  • 3
    Not everything becomes salvageable. Spam does not become salvageable. "Please do my homework for me" questions do not become salvageable. Questions that need to be rewritten are bounced back to the author and are no longer the reviewers problem.
    – RET
    Commented Mar 12, 2020 at 16:08
  • 3
    ”Questions that need to be rewritten are bounced back to the author and are no longer the reviewers problem.” Yes. And the button to do that is labeled “Unsalvageable”. Click it, and then indicate that the question needs to be closed.
    – Cody Gray Mod
    Commented Mar 12, 2020 at 19:55
  • 3
    In which case, at the very least, rename the unsalvageable button to "needs to be rewritten". It still wouldn't make perfect sense but it might achieve the desired response from reviewers without blocking them left right and centre.
    – RET
    Commented Mar 13, 2020 at 10:20
  • 3
    There's no functional difference between "needs to be rewritten" and "unsalvageable" from the reviewer's perspective, because the result is exactly the same - a flag or a close vote, with any difference determined by the specific reason chosen. Maybe the button needs to be relabeled, but there's no benefit to having two buttons (plus relabeling is less work, so more likely to actually happen). Commented Mar 17, 2020 at 0:03
  • Why is it more work to add an extra button rather than having additional reasons in the dropdown? Surely these are about as complex, but two buttons communicates the question to the reviewer more clearly.
    – RET
    Commented Mar 17, 2020 at 9:12
  • Because the dropdown already exists. It's tied directly into the general flagging/closing workflow, so to do what you suggest would require either overhauling that or building a separate system just for Triage. Commented Mar 17, 2020 at 21:39
  • 7
    Since "Unsalvageable" actually means "Return to Sender" why don't we put that on the button?
    – Ben Voigt
    Commented Apr 9, 2020 at 18:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .