333

I had a review audit yesterday, and apparently the answer was a "good answer." I don't disagree that the answer was good, I wasn't going to downvote or flag, simply leave a comment. But as soon as I clicked "add comment" I failed the audit without even leaving my comment!

This seems like bad behaviour for the review system to fail you just because of a comment. I realize that one review audit isn't going to ban me from reviewing, I'm not concerned about that. I just think this feature of the audit system should be changed. Fail me for downvoting or flagging a good answer, not for leaving a comment on it. The review system cannot know what my comment was going to say... I could have been adding a helpful note to the answer.

Should this be changed?

29
  • 3
    Why were you going to leave a comment if you "don't disagree that the answer was good"? What would have that comment been?
    – ChrisF Mod
    Commented Jun 17, 2014 at 15:28
  • 78
    I was actually going to comment to ask for clarification, which I will agree has nothing to do with reviewing a post and was simply my own personal interest. However, it raised the question of what if someone was adding a comment to clarify an answer themselves or add information about a possible edge case or something like that? They will be failed simply for clicking "add comment." The review system assumes that I was going to comment negatively on the post, regardless of the fact that it can't possibly know what I would have said.
    – eddie_cat
    Commented Jun 17, 2014 at 15:33
  • 91
    I thought the purpose of reviewing was to improve the site... I think certain comments could do that regardless of whether the post was "bad" to begin with, so why should we have to go to a different page and leave the review process to do that? It's a bit counterintuitive.
    – eddie_cat
    Commented Jun 17, 2014 at 15:38
  • 38
    @ChrisF In that case, should the "add comment" button be replaced with an "add criticism" button? Just to indicate what sorts of being helpful are and are not permitted.
    – Sneftel
    Commented Jun 17, 2014 at 15:39
  • 16
    Or disable "add comment" until after the review decision point, and only enable it if the decision was unfavorable. Commented Jun 17, 2014 at 15:41
  • 5
    The review system assumes that I was going to comment negatively that's where you are wrong. It doesn't assume anything, it just tells you that commenting on a good question/answer isn't the action to take concerning said item when reviewing it. Commented Jun 17, 2014 at 15:57
  • 73
    See here: meta.stackexchange.com/questions/146988/…. But it is just as important to provide some sort of feedback to the new authors whose posts aren't terrible - whether that's an up-vote, or a comment, or even a small edit that puts a bit more of a shine on their already-useful contribution
    – eddie_cat
    Commented Jun 17, 2014 at 15:58
  • 24
    Clearly the original intent of the review queue was not to make commenting on good posts a Bad Action. The only reason it was bad is because this was an audit and everyone had already decided the post was good.
    – eddie_cat
    Commented Jun 17, 2014 at 15:59
  • 7
    @JonathanDrapeau So if you were to see someone commenting in a non-critical way on someone's First Post, would you flag that comment for deletion?
    – Sneftel
    Commented Jun 17, 2014 at 16:01
  • 21
    @ChrisF "Not all actions are available on all review queues so that you concentrate on the task at hand." But the comment action is available in this one, and there's no clarification that the "comment action" in the review queue is supposed to be any different form the "comment action" outside the review queue. We're not supposed to vote or flag differently in the review queues than outside of them, are we? Then why comment differently? Commented Jun 17, 2014 at 19:30
  • 11
    I was about to comment that it was a good question but perhaps belonged on Computer Science and then vote it up. "Failed" ??? Excuse me, but I was going to vote on the question, and not only that, but vote on the question in a way that you seemed to think I should vote anyway. What's wrong with leaving a comment first then voting?
    – ouflak
    Commented Jul 3, 2014 at 16:22
  • 26
    It is very annoying, and extremely illogical. And even more annoying that you get banned, after succesfully passing dozens of tests in a row, just because you wanted to add a constructive, helpful comment. After experiencing such punches in the face several times, without logic or explanation, my motivation for participating in "Help improve the site" is at a very, very low level. Commented Jul 18, 2014 at 7:34
  • 8
    @ChrisF, Why were you going to leave a comment if you "don't disagree that the answer was good"?, Thanks for demonstrating the problem with illogical test algorithms and the problem with doctrinal moderators itself. Commented Jul 18, 2014 at 7:46
  • 8
    I wasn't aware of this "feature" but if I ran into it, that would definitely be the last time I reviewed anything on the site. There are few things worse than a piece of software trying to second-guess my intentions.
    – biziclop
    Commented Jul 31, 2014 at 12:38
  • 11
    @ChrisF Just because a question or answer is already "good" doesn't mean it can't be made better with additional clarification by its author. "Good" is not a synonym for "perfect". (Except perhaps in the book of Genesis, but we'll leave that for Hermeneutics, not MSO.) Commented Jan 27, 2015 at 16:02

4 Answers 4

282

This most definitely should be changed. It is fundamentally illogical (which as a programmer, is something I tend to despise) and presumptive of the reviewer's actions in so many ways that it is impossible to believe an algorithm could handle, or even begin to handle, without a team of psychologists, all of the predictive text experts from Google, and a psychic.

18
  • 27
    6th of June, 2016: This has still not been changed and another soul has fallen victim to this madness.
    – RaminS
    Commented Jun 2, 2016 at 20:56
  • 9
    16th of June, 2016: Here's another one
    – Tim Malone
    Commented Jun 15, 2016 at 22:34
  • 16
    @Gendarme How did you possibly post on the 2nd June about a review on the 6th?
    – Tim Malone
    Commented Jun 15, 2016 at 22:35
  • 11
    @TimMalone The gods of time have to answer that; I have no clue.
    – RaminS
    Commented Jun 15, 2016 at 22:39
  • 2
    Caught by this today too. Couldn't understand why when all i did was attempt to ask a question about the question
    – Takarii
    Commented Dec 1, 2016 at 13:13
  • 4
    I've now been bitten by it again, and failed a review: nimbus.everhelper.me/client/notes/share/853796/… where I wanted to comment / aks what parts of the question now remain open after the bug report replies. I definitely considered it a good question! Previously I failed review often enough with comments that I was even temporarily banned for that reason... not nice... I shall indeed reconsider to skip reviews if I can fail by simply commenting on a posting. It's IMHO totally inappropriate to equate leaving a comment = bad posting. Commented Apr 7, 2017 at 10:23
  • 3
    and another one Commented Jul 26, 2017 at 14:44
  • 4
    Another one get bit, 2/5/18 Commented Feb 5, 2018 at 18:52
  • 7
    2/27/2018 and this is still happening. Don't fail people for making comments unless the system can judge each comment based on its merits. That is impractical since it would require advanced AI or a separate queue for reviewing comments (which is ridiculous).
    – peacetype
    Commented Feb 27, 2018 at 18:41
  • 3
    And another one bites the dust... RIP 8/07/2018
    – Davy M
    Commented Aug 7, 2018 at 9:30
  • 2
    This just happened to me so it hasn't been changed in the past 4+ years.
    – SamTebbs33
    Commented Nov 16, 2018 at 11:37
  • 3
    Carried to next decade : meta.stackoverflow.com/questions/393045/…
    – aksh1618
    Commented Jan 17, 2020 at 20:57
  • 5
    2020 July fresh stock stackoverflow.com/review/first-posts/26606602 Commented Jul 7, 2020 at 3:20
  • 4
    Another meta.stackoverflow.com/questions/401277/…
    – Ian Kemp
    Commented Sep 18, 2020 at 17:15
  • 3
    Another one: meta.stackoverflow.com/questions/404883/…
    – 10 Rep
    Commented Feb 1, 2021 at 1:06
49

If Commenting is a review failure there is no reason to leave the "add comment" option open?

That would eliminate all these cases of dubious failures and annoying bans immediately.

As a temporary workaround for those who review: don't comment in the review screen anymore but open the question in another tab and insert your comments there directly.

The best would be of course to count comments not as failure but unfortunately this has not been implemented so far.

8
  • 64
    As a temporary workaround for those who review: don't comment in the review screen anymore, yeah - "help to impove the site"? Another workaround is not to review at all. Commented Jul 18, 2014 at 9:07
  • 7
    @davidkonrad But if you want to review and you want to comment then for the time being just separate the two actions. Commented Jul 18, 2014 at 9:13
  • 6
    @davidkonrad Your suggested workaround, if widely adopted, would cause review to fail its essential purpose. Then you'd see "Why aren't users reviewing anymore?" questions on MSO. Commented Jan 27, 2015 at 16:03
  • @tepples, so long ago :) Was at the time really frustrated over many weird "failed audits" so didnt review in a period. Now the review system really seems to have been improved, so have started over again in a small scale. Commented Jan 27, 2015 at 17:29
  • 4
    Having been bitten by this just now (I wanted to comment on stackoverflow.com/questions/32017070 but it turned out that made me fail the review audit stackoverflow.com/review/first-posts/9459523) and having the review eat my comment text so I had to rewrite it, I will be more likely to distrust the review page in future.
    – Neil
    Commented Sep 10, 2015 at 11:22
  • 2
    @davidkonrad I widely adopt David's suggested workaround. No more audit fails for me. No more serial downvotes either. Commented Mar 5, 2018 at 9:22
  • for CVs we have a chatroom and suggested edit audits are actually fair. Commented Mar 5, 2018 at 9:23
  • 1
    Seems we still have this issue.
    – ouflak
    Commented Apr 14, 2022 at 11:52
27

Several years later!

We spent a while figuring out how to resolve this, and came to the decision that in this very specific audit case, comments can count as either a positive or negative review. Philosophically, I'd rather see a reviewer pass an audit they may have gotten wrong than penalize a reviewer for an audit that they did not get wrong, especially in an uncommon case like this.

The fix will be in the next build, which is coming either today or on Monday.

2
  • 1
    A Monday build? It's going to be fun :D
    – Braiam
    Commented Mar 5, 2021 at 17:58
  • I just saw this - awesome! I am curious why even count commenting as a 'review action' at all, though?
    – eddie_cat
    Commented Jul 27, 2021 at 14:32
-79

To be perfectly honest, I don't really think Comment should count as a review action on these posts at all. But folks felt strongly that commenting was an important part of reviewing posts from new users, and so we left that option available.

If you're gonna comment, you should have something worth saying. Constructive criticism, helpful advice, etc. - an explanation for why you're downvoting or flagging. Don't just kibitz on posts that are perfectly fine the way they are - that's noise, and someone else will eventually have to clean that up.

The review system doesn't know what you were going to write, but it can make some assumptions as to what you should be writing - and if you were planning to write something else, this is a good opportunity to remind you to think twice the next time around.

Frankly, I'm about fed up with this entire review queue - the goals are underspecified, and so the audits are going to be problematic since if you squint just right any action is "the right action". Time to kill the whole sorry mess and come up with something better.

24
  • 91
    This does not address the issue - it's utterly improper for an automated review system which can not know what the poster will write to fail them before they have even written anything. Unless the actual text of the comment is going to be judged, there is no proper alternative but for a comment to count as a "skip" for audit purposes. (Or possibly, to mark that question as too potentially controversial to ever be used in an audit again) Commented Jun 17, 2014 at 14:48
  • 10
    You'd prefer it failed them after they wrote something? Or even passed them after they wrote something that would then never actually appear on the site? ...The latter was actually the subject of a fairly long internal discussion; we eventually decided that letting someone spend time typing out a full comment only to discard it would be a pretty severe waste of their time, punishing folks doing it right for the sake of the few who weren't. If you have a better solution, let's hear it...
    – Shog9 Mod
    Commented Jun 17, 2014 at 14:51
  • 39
    In general, the audit system does a very bad job when the reviewer does anything other than the action it expects, no matter how justified and appropriate the alternative action might be. Commented Jun 17, 2014 at 14:52
  • 19
    Hitting the comment on a review should act as a skip for the review purposes, and as a normal ("real") comment on the post, as whatever the reviewer is going to say in their comment is presumably just as valid or invalid as any question-page-visit comment. Commented Jun 17, 2014 at 14:54
  • 17
    I don't think it should fail anyone for a comment at all because of what Shog9 mentioned about wasting people's time. The review system should really just ignore commenting behavior since it can't possibly understand what people will say/have said.
    – eddie_cat
    Commented Jun 17, 2014 at 14:54
  • 41
    You can't know that it won't improve it. Good answer != perfect answer. And having to open the question in another tab is a bit silly and cumbersome. It's a workaround, not a real solution...
    – eddie_cat
    Commented Jun 17, 2014 at 15:16
  • 26
    So remove the option for commenting during a review entirely. If I have the option to comment as usual, I should be allowed to do so without penalty.
    – eddie_cat
    Commented Jun 17, 2014 at 15:34
  • 3
    Before you kill the "first posts" queue, you might want to do a bit of analysis and find out how much good (and how much abuse) currently comes from it. The goal seems simple enough, if somewhat nebulous: bring folks to the users participating for the first time, so that they can provide some helpful guidance and steer them through the rough waters.
    – Robert Harvey Mod
    Commented Jun 17, 2014 at 15:41
  • 5
    +1 for "fed up with the entire review queue". But like Robert Harvey, I think we should be careful not to throw the baby out with the bathwater. The proliferation of low quality posts is a serious problem; they need to be reviewed and cleared out. Perhaps some attention should be devoted to making the review queue less of a landmine. The biggest complaint I see are people who failed review audits, many in unreasonable circumstances. I realize that one failure is not considered a serious problem by "the system", but it is by "the human" reviewer.
    – Cody Gray Mod
    Commented Jun 17, 2014 at 17:01
  • 37
    Since review audits are meant to catch robo-reviewers, the user should just pass the review audit when he or she tries to comment. If the user is making the effort to write a comment, then that user is not a robo-reviewer.
    – Pablo
    Commented Jun 17, 2014 at 18:15
  • 9
    @Pablo: Until the robo-reviewers start doing that too.
    – tmyklebu
    Commented Jun 17, 2014 at 18:32
  • 8
    I think commenting on a bad post is perfectly valid criteria for passing a review audit. Downvoting or flagging shouldn't be the only pass criteria. However, I think commenting on a good post should be a neutral action and skipped as suggested above. (Without making them type out the comment.) Commented Jun 17, 2014 at 18:43
  • 9
    @JonathanDrapeau "You are reviewing it, if it is good, there's no action needed, regardless if it could be improved" This isn't the precendent set by the Suggested Edit review queue, where Improving is actually one of the specified things that you're encouraged to do is something is good, but not as good as it could be. The "expected" actions might be different for different review queues, but if we get people in the habit of trying to help in one queue, it seems counter-productive to penalize them for it in another. Commented Jun 17, 2014 at 19:33
  • 11
    I was going to make a positive constructive comment and then vote up (which it turned out would have been the correct vote anyway). Why should that be counted as a failure? That doesn't make any sense!
    – ouflak
    Commented Jul 3, 2014 at 16:23
  • 2
    @Shog9 - I'm not sure why you got such a venomous reaction from the community. My apologies to you.
    – jww
    Commented Jul 8, 2014 at 3:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .