-9

Is there any consistency in what is expected when reviewing answers?

  • What exactly is the task when reviewing answers? I got this as a test, and I should have rejected it because it is a duplicate of previous answers. But I don't see the previous answers when reviewing, so how am I supposed to find out?

Additionally, the answer has been deleted, so someone must have flagged it as a bad answer for it to enter the low quality queue, unless a moderator saw it and directly deleted it.

On the other hand, I had flagged posts as low quality, it was declined because "flags should only be used to make moderators aware of content that requires their intervention".

  • Is this even true? Do low quality flags go to the moderators, or do the go to the Low Quality Posts queue? I assume these are not the same, as the Low Quality Posts just needs 2k reputation.

In other cases I got "flags should not be used to indicate technical inaccuracies, or an altogether wrong answer". But as mentioned, in the above example a question was flagged and deleted that was not even wrong, just a duplicate of another answer.

I understand that different people have different opinions on things. Otherwise there would be no point in getting multiple people to review the same posts or flags. But then it should be acknowledged that different opinions are possible and not present the result as black and white: "Pay attention! You have failed! You are stupid!".

At least the wording should be more neutral ("others have disagreed", not "you are wrong") and the threshold should be consider the balance of helpful and declined flags. At the moment, I have 47 helpful flags, but am blocked because of 3 declined flags.

Edit

For all those who said that the comments make the audits easy,

Here is the display during the audit. As you can see, there are no comments, the score is zero and answered today. Display during audit Here is the display after the audit. As you can see, now the comment is visible, the score is negative and answered 8 days ago. Display after audit So comments are not visible during an audit, they are shown after the audit was revealed.

8
  • 3
    Especially for the First Posts review queue it is essential that you look at the entire Q+A. Which is quite easy to do, just click the question title to open it in another browser tab. The audit failure message is poke-in-the-eye intentionally, its was meant to wake up reviewers from their usual catatonic robo-clicking state. 7 reviews in less than a minute is a good hint that you got into that state. Best to not review if you find it difficult to deal with or failed a lot of audits without knowing how to improve. Commented Jun 9, 2019 at 10:26
  • @Mat The comment on the answer is only displayed after the review is complete. While reviewing, the comment is not displayed. This is probably intentionally to NOT give that hint.
    – RalfFriedl
    Commented Jun 9, 2019 at 10:44
  • 2
    That answer was deleted two weeks ago. They made passing the audit intentionally easy. As long as you look. Commented Jun 9, 2019 at 10:52
  • @HansPassant As long as I look for what?
    – RalfFriedl
    Commented Jun 9, 2019 at 10:55
  • 7
    it is essential that you look at the entire Q+A. Commented Jun 9, 2019 at 10:58
  • @HansPassant Can you point me to the place in the Q+A where they state that I have to look for other posts to determine whether the answer presented is a duplicate of another answer?
    – RalfFriedl
    Commented Jun 9, 2019 at 11:07
  • 2
    Possible duplicate of How is this a bad answer? (there is also a cross-site dupe at MSE: While reviewing “The Late answers” should we compare it to other answers?)
    – gnat
    Commented Jun 9, 2019 at 11:19
  • 5

1 Answer 1

2

What exactly is the task when reviewing answers?

To determine if the answers meet our quality standards and are worth keeping around. Pretty simple, really. It's best if you don't overthink it.

If you need a more detailed guide, we have one here, but I personally believe that applying a common sense understanding about the larger purpose of reviewing (content curation) works far better than trying to create and follow a list of "rules".

But I don't see the previous answers when reviewing, so how am I supposed to find out?

Well, in this case, it was pretty easy. There was a comment on that answer stating unambiguously that it was a duplicate of previous answers. That should have been your clue to, you know, actually look at the other answers to assess the veracity of that comment.

In general, like Hans Passant has been saying in the comments, I recommend looking at the entire Q&A when reviewing. Is it a rule? No, not really. Does everyone do it? No. Are there people who disagree with this, and think that, since there's no rule requiring it, you don't have to do it? Absolutely. But will it help you to be a better reviewer if you take the time to do it? Yes! Because, remember, the goal when reviewing is to improve the site.

Additionally, the answer has been deleted, so someone must have flagged it as a bad answer for it to enter the low quality queue, unless a moderator saw it and directly deleted it.

Yup, somebody flagged it. You can pretty much assume that they did, given the comment that was left underneath the answer and the fact that a moderator deleted it.

On the other hand, I had flagged posts as low quality, it was declined because "flags should only be used to make moderators aware of content that requires their intervention".

Yeah, that also happens. Two reasons:

  1. There are multiple moderators. Moderators are humans, and each has slightly different standards or points of view when processing flags. In particular, some moderators will delete low-effort duplicates that add no value to the Q&A in response to VLQ/NAA flags on those answers, while others will not necessarily do so because they evaluate each answer in isolation.

  2. More likely, though, the problem is that you never called the issue of duplication to the moderator's attention. Simply flagging an answer as VLQ or NAA implies that the problems with the answer are obvious and can be sussed out by simply reading the body of the answer itself. When you're flagging an answer because it is a no-value-added late duplicate, it is highly recommended that you use a custom flag to explain the problem clearly. These types of flags are much less likely to be declined, unless a moderator disagrees with you on the "no value added" part. See also this recent discussion on flagging duplicate answers.

    You might be able to get away with leaving a comment explaining the problem underneath the answer and then raising a VLQ or NAA flag, but I don't recommend it. For one thing, a moderator might still miss the comment, as you apparently did when reviewing. Second, answers that are deleted in response to VLQ and NAA flags end up as audits, tricking people who don't read the comments. If the answer had been flagged with a custom flag, and deleted on that basis, it wouldn't have ended up as an audit. Typing the explanation into a custom flag textbox is no more difficult than typing it into the comment textbox. It also preserves the anonymity of the flagger, unlike a comment.

Do low quality flags go to the moderators, or do the go to the Low Quality Posts queue? I assume these are not the same, as the Low Quality Posts just needs 2k reputation.

They go to both.

Generally, what happens is that they go to the Low Quality Posts review queue first, where they can be reviewed by the community. If the community doesn't handle them from there within a relatively short period of time, and/or if the community cannot reach a consensus, then they get presented to moderators in the standard moderator flag dashboard.

It is also possible for moderators to view all pending VLQ and NAA flags, bypassing the usual delay (the "relatively short period of time" mentioned above).

So, when you raise one of these flags, assume that they can be processed by either the community or a moderator. The difference really shouldn't matter.

How, then, do you make sense of "flags should only be used to make moderators aware of content that requires their intervention"? I use this pretty often when declining VLQ flags because it's one of the canned reasons and thus saves me from a bunch of extra typing. You can replace "moderators" there with "reviewers"; that is not the point of the message. What I mean by that decline message is essentially that you should only be flagging posts as VLQ because they need to be deleted. The VLQ flag is not a "requires editing" flag. Note that there is no "requires editing" flag because editing is something that anyone can do themselves—even anonymous users. Therefore, I use this flag decline reason to essentially mean that you should not have flagged the answer for moderator or community reviewer attention; instead, you should have edited the answer yourself, using privileges that are already available to you.

Flags aren't meant as a way of getting someone else to do work for you that you could do yourself. They're a way of escalating problems to a group of people with a higher level of privileges who can do things that you cannot. Like, for example, deleting posts.

For example, you recently flagged this answer as VLQ, yet had the flag declined for this very reason. By my judgment, that post did not need to be deleted, as it did contain a potentially-useful answer to the question. Rather, it needed to be edited. The edit needed was a pretty severe edit—probably one that chopped off the entire top half of the answer, since it isn't terribly useful to see what didn't work—but still, an edit nevertheless. Deleting the answer (which is what a VLQ flag requests) would not, in my opinion, have been of service to the community, because it would have removed the potential solution that was provided. I did look and see if that solution was recommended elsewhere in the other answers before declining the flag, but I didn't see it. Note that that's a level of extreme diligence you really shouldn't expect from other moderators. As mentioned above, when you're flagging an answer that should be deleted because it's a rehash of an existing answer, please use a custom flag, rather than a VLQ flag.

As another example: you recently flagged this post as VLQ. That flag was declined by another moderator (i.e., not me). They used a different decline reason ("flags should not be used to indicate technical inaccuracies, or an altogether wrong answer"); if it were me, I would have used "flags should only be used to make moderators aware of content that requires their intervention". Why? Because the problem with the post was that it required editing, not that it required deletion. To be clear, at the time your flag was raised, the answer looked like this. Yes, that's garbage. But, we prefer not to delete things like this that are obviously attempts to answer the question. Instead, we prefer to edit them so that they are readable. Someone has since done that, solving the problem nicely, without requiring moderator intervention.

In other cases I got "flags should not be used to indicate technical inaccuracies, or an altogether wrong answer". But as mentioned, in the above example a question was flagged and deleted that was not even wrong, just a duplicate of another answer.

Sort of covered in the discussion immediately above. The moderator who processed that flag chose that canned decline reason instead of the one that I typically choose. He was being charitable: he assumed that you were flagging the answer based on some technical knowledge that you had. My assessment would have been the one I already presented above: you were flagging the answer because you thought someone should edit it. The question is, why was that someone not you?

But there is something else that needs to be said here: that decline reason is entirely truthful. Moderators do not delete flagged answers because of technical inaccuracies or because they are wrong. They do delete answers because they are not attempts to answer, and they do delete answers because they are low-quality rehashes of existing answers. Neither of those require moderators to judge the technical accuracy of answers. We don't do that, because we cannot be experts on all subjects covered by this website. If you think an answer is technically incorrect or wrong, then you should downvote it; not flag it. On the other hand, if there is an objective reason why you believe an answer needs to be deleted, then you can flag it for moderator attention.

I understand that different people have different opinions on things. Otherwise there would be no point in getting multiple people to review the same posts or flags. But then it should be acknowledged that different opinions are possible and not present the result as black and white: "Pay attention! You have failed! You are stupid!".

Oh, did they finally add in the "You are stupid!" part? That addresses one of my long-standing feature requests.

No, just kidding. It doesn't say that. But I do understand how it can feel like that. Honestly, it's supposed to feel like that, because the point of audits is to catch people who are making unarguably wrong decisions when reviewing. It is a common problem.

In this case, the audit wasn't great. You were still in the wrong because of that super-obvious comment there suggesting it was a duplicate, but this problem admittedly wasn't obvious enough to swat you in the face and call you an idiot. It would have been a terrible audit if that comment wasn't there.

This is kind of on me. When a garbage post is flagged, and I think it deserves to be deleted, I validate the flag and delete it. I don't overthink moderation. Robert Harvey summed up my moderation strategy elsewhere. So far, so good. The problem is, the audit system sees a flagged post, and a moderator who validated that flag by deleting the post, and concludes that the post would make an excellent review audit candidate. Well, not necessarily, as I said, especially in the case that the comment would not have been there. Then, there'd be nothing to suggest to you that you needed to look at the rest of the Q&A, and while I think that's always a good thing to do, it's not strong enough grounds on which to cause someone to fail an audit.

The audit system is broken. If it were up to me, it would be using posts with obvious problems as audit candidates, hand-selected by moderators. It would also be using posts that obviously had no problems as audit candidates, again, hand-selected by moderators. This would do a better job of serving the true purpose of audits: (1) catching people who are robo-reviewing, and (2) training people how to be better reviewers.

At the moment, I have 47 helpful flags, but am blocked because of 3 declined flags.

You are mixing two things together here. The review system is what presented you with that slap-in-the-face "Pay attention!" message when you failed an audit. But that is unrelated to flags, of which you have 47 helpful, 3 declined, and are currently banned from flagging. Often, reviewing will lead you to raise flags, but they are two separate mechanisms, each with their own bans.

Flags go to moderators and are—generally speaking—handled by moderators. There are some exceptions where flags are or can be handled by the community, but the community can never decline a flag (flags that the community disagrees with will be marked "disputed"). Only a moderator can decline a flag. And you can only be banned from flagging by having a certain portion of your recent flags declined.

I don't claim to have committed to memory all the details of how flag bans work; you can surely find this information on Meta. It doesn't seem to me like you should be banned from flagging, either with this ratio of helpful to declined flags, or even after reviewing your flag history. Unfortunately, moderators cannot manually reverse flag bans. They are entirely automatic. You'll be unbanned in 6 days. That's really all I can tell you, unfortunately.

As far as reviewing, if you fail too many audits, you can be blocked from reviewing. You are not, however, currently blocked from reviewing. In fact, you've never been blocked from reviewing. If you were, I could undo that. C'est la vie.

8
  • Thank you for your elaborate answer. Just a few points: When an answer is presented as an audit, it doesn't show any comments. Therefor, comments to the post are not a clue while deciding what to do with the post. My question was not so much about the duplicate, if you click on that link you will see "Review audit passed", because I did check the original question and found out about the duplicate. I just think from a user interface point of view, the information needed to process the review should be presented on the screen without the need to open the whole question in a new tab.
    – RalfFriedl
    Commented Jun 10, 2019 at 7:56
  • The duplicate answer was just an example of content that was correct and yet it was presented in the audit as clearly wrong, or bad quality, The others were at least as bad, in my opinion. Maybe you should change the text of VLQ: "This answer has severe formatting or content problems. This answer is unlikely to be salvageable through editing, and might need to be removed." Formatting problem can always be resolved by editing, and content problems can be fixed by removing everything and writing something else.
    – RalfFriedl
    Commented Jun 10, 2019 at 8:03
  • 2
    @RalfFriedl When answers are presented as audtis, the comments are visible. You can verify this by clicking on the audit you link in your question, and you will see the comment about that answer being a duplicate. Commented Jun 10, 2019 at 11:55
  • 2
    Here, I made a handy chart i.sstatic.net/FhDz5.png Commented Jun 10, 2019 at 12:12
  • @FélixGagnon-Grenier Yes, it is visible. But is visible only AFTER it is revealed that it has been an audit, not before. When it is presented, the comments are NOT visible. I can try to capture an image before I answer the next suspected audit.
    – RalfFriedl
    Commented Jun 10, 2019 at 14:26
  • @Ralf Thank you for reading my book. Since you mention renaming or otherwise modifying the VLQ flag, you might be interested in other novels in the series, including meta.stackoverflow.com/a/357344 and meta.stackoverflow.com/a/382793. In short, I agree with you; VLQ is a confusing, useless flag. It should be renamed to “unsalvageable” (with “plz delete” as the description), or removed altogether. I can’t make that UI change, though, so until that time, just treat it as a “needs to be deleted” flag. Regarding comment visibility on audits, I don’t know for sure.
    – Cody Gray Mod
    Commented Jun 10, 2019 at 19:53
  • @FélixGagnon-Grenier I added images of a question during and after an audit to the question. As you can see, the comments are NOT visible during the audit.
    – RalfFriedl
    Commented Jun 13, 2019 at 21:28
  • I see. I stand corrected it seems @RalfFriedl. Well, all the more reasons to always open the relevant posts. Commented Jun 13, 2019 at 21:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .