55

Would it be feasible and desirable to have practice queues for the various review queues? Such queues could help those who are new to the review process learn how to do reviews; they could also help established reviewers improve the quality of their reviews by showing them worked examples of best practice.

Such queues would consist of old or fake posts, similar to audit reviews, and upon completion of the review action on each post the user would be shown an action that's considered appropriate for that post, along with the rationale for that action. Eg, a practice suggested edit review task could mention getting rid of "fluff" like greetings and "thanks in advance"; fixing capitalization, spelling and grammar errors; putting code samples into code blocks and quotes into quote blocks; etc.

For some review tasks only one action is the most appropriate response, but in other cases there are multiple possible responses that would be considered acceptable. I guess that makes implementing such review queues a little more complicated, but hopefully the technical issues are not insurmountable.

When I first thought of this idea I envisaged it as a purely voluntary thing to help new reviewers (including myself) become both confident and competent at executing review tasks. So it would be somewhat like reading the Help pages, only more active. However, I now think it might be a good idea to make it mandatory: new reviewers would need to reach an adequate score on the practice tasks for a given queue before they get the badge that gives them access to the live posts. I guess to make the system fair current reviewers would also need to get such badges, but that should be a swift process, assuming they actually do have good review skills. :)

FWIW, I came up with the idea of practice queues when the Triage system was introduced. At first I was excited by the Triage process, but I quickly realized that most of the time I didn't really know what to do with most of the posts presented, and wished that there were worked examples that I could refer to so that I could learn how to do it properly, partly so that I could avoid failing review audits (not that I've ever actually failed an audit :) ), but mostly so that I could contribute to the best of my ability.

6
  • 12
    Or even just to be able to see past reviews would be nice. For example going into the revision history for a post, and then clicking on "edit approved" link, will give you a page with the review. I'd like an easy way to that page for the most recently approved edits. Commented Apr 10, 2015 at 11:05
  • 4
    @JonasCz: Yes! Even if dynamic practice queues aren't feasible, it'd be great to be able to see a bunch of good worked examples of review tasks.
    – PM 2Ring
    Commented Apr 10, 2015 at 12:01
  • 2
    This sounds a lot like letting new reviewers start with a few audits (that explain them which actions are appropriate by example) and as soon as they have guessed enough "correct" actions they are unleashed onto the actual reviews…
    – Bergi
    Commented Apr 11, 2015 at 19:10
  • 1
    @Bergi: Sure, but hopefully by the end of the process they are choosing correct actions through understanding why the action is appropriate, rather than through mere guess-work. Ideally, they'd start seeing training posts where the correct action is fairly obvious, then they'd progress on to more ambiguous cases. Once they'd reached an adequate hit-rate they'd be permitted to do actual reviews, but they could also do more practice posts if they wanted to improve their confidence for any reason (eg if returning to reviews after a long absence).
    – PM 2Ring
    Commented Apr 12, 2015 at 6:11
  • FYI: I’ve highlighted this suggestion on the recent [product-discovery] post, “Improving Review Queues - Design overview I: Onboarding and updating workflows over on Meta Stack Exchange. Commented Jun 11, 2020 at 0:43
  • @JeremyCaney Thanks. As you might guess, I like your suggestion. ;)
    – PM 2Ring
    Commented Jun 11, 2020 at 1:26

4 Answers 4

19

What I would like is to simply roll this into the actual audits and thus strengthen them immensely from merely filtering out robo-reviewers and the occasional serial misclicker or unlucky victim of bad audit selection, into a full-on teaching tool, able to tell you "no, you should have downvoted that instead of flagging NAA because it was wrong", "no, you should have flagged that as unclear or too broad instead of belonging on SF", and so forth.

7
  • 2
    A guy can dream. Commented Apr 11, 2015 at 5:27
  • 2
    That would mean a whole different approach in selecting audits. It is done now on the quality of an question or answer and that one is a little obfuscated. Your proposal means someone has to pick and write audits. Commented Apr 11, 2015 at 5:55
  • That sounds good to me, Nathan.
    – PM 2Ring
    Commented Apr 11, 2015 at 8:32
  • 1
    @PatrickHofman: We could have a queue for that...
    – PM 2Ring
    Commented Apr 11, 2015 at 8:33
  • 2
    @PatrickHofman: I know. Commented Apr 11, 2015 at 15:20
  • @NathanTuggy: Nice. I wish I'd read that manual-audit-validation page before writing my question. I guess one major difference between normal audit posts and annotated audits for training purposes is that normal audit posts have a limited lifespan, whereas good training posts can last indefinitely, and if people add to the annotations they can even get better with age.
    – PM 2Ring
    Commented Apr 12, 2015 at 5:57
  • @PM2Ring: Yes, although I assumed they were disposable for the sake of being conservative. (Even so, they are still eminently practical; if they last indefinitely, still more so.) Commented Apr 12, 2015 at 22:56
18

I think it would be a waste of time to do this really.

What we need is much clearer explanations of how your actions are interpreted. I just learnt, for example, that during a 'triage' review you say 'should be improved' it just bumps it over the the 'help and improve' queue; it's easy to think that the initial 'should be improved' would be a flag to the OP.

I've had to learn by trial and error what the review queues consider appropriate actions. As far as I know, there isn't a proper 'guide' explaining what is considered best practice. I don't see how a mock queue will help me learn this any better, and instead I am spending time not reviewing.

7
  • 3
  • 2
    Well yes, it would consume reviewers' time, but I wouldn't consider it as wasting time if it improved our skills and our confidence that we're reviewing effectively. I certainly agree that clearer explanations would be great, but I suggest that explanations in the context of actual worked examples could be even more effective.
    – PM 2Ring
    Commented Apr 11, 2015 at 8:27
  • 1
    @NathanTuggy: Thanks for those links! I wish I'd read them before I started reviewing. Perhaps there should be a dedicated section on Reviewing in the Help centre... if there is one already, I missed it. :)
    – PM 2Ring
    Commented Apr 11, 2015 at 8:31
  • 11
    Each review queue should have a link to a faq post that explains in detail what is expected of the reviewers. Commented Apr 11, 2015 at 8:46
  • 3
    @NathanTuggy what use are those guides though? People will not go hunting for such information, it's needs to be shoved right under their nose, a nice clear Read these guides on how to review.
    – thecoshman
    Commented Apr 11, 2015 at 15:59
  • @thecoshman: I mostly agree, per Nisse's comment earlier. Commented Apr 11, 2015 at 16:02
  • @thecoshman, I agree that the explanations should be clearer. But I strongly disagree that implementing practice queues would be a waste of time. I think almost all of us learn by actively applying and not merely reading the theory. I do not think SO's reviews queues are an exception to this.
    – cel
    Commented Nov 8, 2015 at 7:35
3

I wouldn't mind having a training feature that consisted of an endless stream of unmodified, bare-bones audit tasks.

I just got review banned for five days because of a triage audit (overzealous use of "needs author edit"), and I'd like to get a better grasp on the lines between the four options so I can be better at reviewing. But it's really hard to do that when I can't access the review queues anymore.

Getting to practice on posts that are known to be good or terrible would be a decent starting point. It wouldn't be perfect, but it's better than trying to draw conclusions from three failed audits and a litany of borderline questions in my review history, and it's easier than some of the more full-featured suggestions out there. (Not easy, but certainly easier.)

0

This seems unnecessary. The review queues are self-correcting, already. Aside from the instructions at the top of the page and the random audits, there's the fact that nearly everything requires multiple votes. When a user first reaches the necessary rep to do XYZ reviews, there's just not that much risk that he or she will break anything by not fully understanding the rules.

When I first started reviewing, I had some misconceptions about how the reviews work. Failing a few audits set me straight pretty quickly, and I didn't do any real damage, because more experienced users could either counteract my votes or simply fix the problems. If a user isn't able to learn from his or her mistakes on audits, then they will pretty quickly fail so many audits they get put into a review ban.

I'm not arguing the queues are perfect. I'm just saying the SE devs have enough going on, and we have too few people willing to do the review queue work already. Imposing another barrier to participation seems unlikely, in my mind, to address any problems not already addressed by the audit system.

Edit: There seems to be some misunderstanding in the comments about my position. I am just saying that the audit system does effectively what this post suggests. True, it's just not a full-time training system; it's mixed in with real reviews. But it does offer the feedback that OP is suggesting we need. I'm not sure why we couldn't get basically the same results by making, say, the first 25 reviews a new reviewer does all be audits.

10
  • 1) Yes, the multiple votes do make the system somewhat robust, but IMO the random audits & fear of being banned are not sufficient - there needs to be more positive feedback to make the trial & error learning more effective. 2) I concede that creating the software for practice queues would add workload for the SE devs.
    – PM 2Ring
    Commented Apr 11, 2015 at 8:42
  • 6
    3) Practice queues may act as a barrier for some, but they may actually encourage others (like me) who currently feel uncomfortable with reviewing because they're not sure what they should be doing. The ability to write good answers (or questions) gives you rep, but it doesn't automatically mean that you also have the skills necessary to do good reviews.
    – PM 2Ring
    Commented Apr 11, 2015 at 8:42
  • @PM2Ring interesting points. In my experience, the instructions and audits were sufficient and worked pretty well. But, everybody's different, and everyone's experience is different, too. Nice comments.
    – elixenide
    Commented Apr 11, 2015 at 13:23
  • 1
    Note that two of the first three available queues on SO, which are the only queues available at that rep level everywhere else, have no consensus required to get anything done. These queues are also the most diffuse in required actions. So I'm not convinced these safeguards are really doing their job all the way. (Let's leave aside suggested edit auto-approval for now.) Commented Apr 11, 2015 at 15:23
  • @NathanTuggy You're right, Late Answers and First Posts don't require a consensus. But they also don't really risk breaking anything. They just encourage new reviewers to look at -- and interact normally with -- certain types of posts. Other than paying attention to flag obvious spam or other problematic posts, there's not a whole lot for users to learn how to do in those two queues. Honestly, I think we'd be better off just adding more audits than creating a special set of practice reviews for each queue.
    – elixenide
    Commented Apr 11, 2015 at 18:48
  • @EdCottrell: I disagree that there's not a whole lot for users to learn how to do. Flagging "other problematic posts" is non-trivial. Commented Apr 11, 2015 at 18:52
  • @NathanTuggy I agree that learning to flag posts takes practice and feedback. I'm just saying that (1) it isn't new to the review queues (users doing reviews already have flag powers) and (2) it's already covered by the audit system. In my experience, audits are much more common when one is starting out, and my understanding is that their frequency is largely driven by how often a user fails them. In other words, mixing real reviews with audits serves a purpose that is much like, but more directly useful to the site than, a training program. Obviously, not everyone agrees, but that's my take.
    – elixenide
    Commented Apr 11, 2015 at 18:59
  • 2
    @EdCottrell: I agree that audits are probably the right, or at least a right, course for improvement here (see comments on other answer); I disagree that they're sufficient in their present state to do the job. Commented Apr 11, 2015 at 19:37
  • @EdCottrell: Another thing is that, while flagging is not new to the review queues, FP and LA in particular add a new dimension to them: there is exactly one reviewer that will see that post that way (barring Skips), and if they don't flag, they've made a mistake; normally, not flagging is a substantially lesser error and cannot be detected by the site at all. Commented Apr 11, 2015 at 20:30
  • Agreed on all accounts. If you're ever confused about a particular queue, you can always come to Meta; there are plenty of guides around to help you figure out what to do.
    – AstroCB
    Commented Apr 12, 2015 at 0:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .