27

Up until very recently, the review queue for particular tags has given completely unrelated and, in my opinion, wildly inappropriate audit questions.

Now, it seems that the process has been "improved" by merely injecting the reviewed tag into the existing question.

For example, this review (which I am certain was an audit) has the ruby tag, despite clearly not having anything to do with ruby. The question has never had the ruby tag, and is clearly identified as being a delphi question.

I opted to skip it, but strongly suspected a crappy review audit.

The very next review was a review audit, as well. It no longer shows the ruby tag, but in the review, it showed ruby and scala. I appreciate that the previous method of sticking random unrelated audits into specific reviews needed to be improved, but it seems to me that the new solution is even more obvious and confusing than ever.

I am not asking about these specific reviews. I am trying to determine whether or not sticking the specific tag into completely unrelated items really the best way to solve this. There are 52,551 questions tagged ruby, do we really need to mine the scala questions in order to find good review questions?

6
  • 8
    What's confusing about it? To you, who is paying attention, it's dead obvious what's going on, making passing the audit easy. To someone not paying attention they won't even notice. Where's the problem here? And why are you skipping a post after recognizing it as an audit? Why not simply take the appropriate action and pass the audit?
    – Servy
    Commented Sep 26, 2014 at 16:52
  • 28
    Because, as a ruby question, it's almost completely senseless and should be closed. As a scala (for example) question, it probably makes a lot of sense, and should be left open. I am now left to visit the actual question to see what the eff is actually going on. There are 52,551 questions tagged ruby, do we really need to mine the scala questions in order to find good review questions? At least the old way was obvious on it's face, without being adulterated by inaccuracies by the review process itself...
    – Brad Werth
    Commented Sep 26, 2014 at 16:56
  • 2
    Right in your post you've said that it's clear to you that these posts are audits, so apparently they are obvious on their face, which is their goal. If the audit is obvious to someone paying attention, then it has succeeded at its goal.
    – Servy
    Commented Sep 26, 2014 at 17:05
  • 6
    Well, they just look fishy. But I often come across crappy questions with inappropriate tags, as well, so now I have to suss out which are which. I actually went to those questions to remove the incorrect tag, before realizing what was going on.
    – Brad Werth
    Commented Sep 26, 2014 at 17:10
  • It's all about paying attention - that's it.
    – user2140173
    Commented Sep 29, 2014 at 14:00
  • 1
    @vba4all I'm not sure it's as simple as that. The current process encourages situations like this. Paying attention would not be impacted by improving the relevancy questions. In my answer, I offer a solution that would encourage paying attention, and help find bad reviews, beyond simple audits.
    – Brad Werth
    Commented Sep 29, 2014 at 16:02

5 Answers 5

7

If the only problem with the question you're reviewing is an inappropriate tag, then just edit it to remove the tag.

That goes for audits as well as questions where someone has simply picked a woefully bad tag for an otherwise-reasonable question. Presumably if you're filtering by a tag, you know enough about the topic to recognize questions that don't belong there - make the most of it by helping to clean up the categorization.

3
  • 8
    Thanks for your feedback, as always. Unfortunately, it seems to address one specific review, rather than the question of "Is sticking the specific tag into completely unrelated items really the best way to solve this?". As indicated in the comments, there are 52,551 questions tagged ruby. Do we really need to mine the scala questions in order to find good review questions? It seems a proper review would be a real bad/good question from the pool being reviewed...
    – Brad Werth
    Commented Sep 26, 2014 at 17:09
  • 2
    This would work for some tags, @Brad - but not all; we would still have to fall back on the current behavior if too few questions were available. Also, slow. Worth considering in the future, as we move away from trying to choose audits on the fly though.
    – Shog9
    Commented Sep 26, 2014 at 17:59
  • 1
    I had wondered if some of this was just technological limitations compromising the ideal business logic. I've never scaled a site out to anywhere near as large as SO, I'm sure there are some unique challenges there. It is somewhat hard to imagine that there are very many actively reviewed tags that don't have at least a small handful of extremely good/bad questions to select from. It seems like, for the few that might not, it may not even be an issue. If the volume is low, and review audits happen every 50 (or whatever) reviews, it probably wouldn't even really come up.
    – Brad Werth
    Commented Sep 26, 2014 at 18:47
7

There are many better ways to handle this.

I think one better way to handle this might be to choose disputed audits on the existing tag, serve it up to many users, and use the results to determine which users need more audits in the future.

A simpler solution might just be to pick heavily up/down voted or closed questions from the tag in question, and use them. It already seems to be doing much the same thing, I imagine that it wouldn't be much harder to filter by tag than to inject a incoherent tag.

In any event, it seems to me that review audits would be much more useful if they "blended in" better.

2
  • 2
    Many heavy upvoted questions are the poor ones, that should be closed in the first line. Commented Sep 29, 2014 at 7:47
  • @Donaudampfschifffreizeitfahrt, yeah that's true... I wonder if there is a better metric to programmatically quantify question quality... I just threw this answer in because I dislike when people complain, but do not offer a solution, and the "official" solution did not sit right with me. I personally prefer the "disputed audit" method, but suspect it may not be entirely technically feasible at this time.
    – Brad Werth
    Commented Sep 29, 2014 at 15:58
6

Just to document another corner case, when I clicked a link to review close votes for a currently active burnination https://stackoverflow.com/review/close/?filter-tags=shop,shopping I ended up with a failed review because the system threw in a tag on the review question.

It's been a long time since I manually entered the review queue, but apparently last time I did, I actually specified a filter.

It's weird for this to bleed through when I am actually viewing a different set of tags via a link, though.

1

I was given this https://stackoverflow.com/review/close/18217093 as a C++ question. When Shog9 says:

If the only problem with the question you're reviewing is an inappropriate tag, then just edit it to remove the tag.

How can I remove the tag when it is not present on the question, it was just artificially added by the review system? Or is it done to review my ability to judge if tags are appropriate and how will I handle mis-tagged questions?

1

A question can be good and clear when asked for the correct tag. But "Unclear" or "Too Broad" when interpreted as a question for a different programming language. I just failed an audit because it gave me an R question as a Java question.

Audit questions should match by tag filter. I put the tag filter there for a reason. I don't appreciate the false claim that I have erred by voting-to-close a question that is "Unclear" because it has been rendered unclear by the automated audit system.

4
  • I understand the frustration of finding a completely unrelated question with the given tag. However, as currently implemented, the system expects us to check whether the question was properly tagged, which forces us into a different mindset than the one we usually have in the review queue. All else one could say here is YMMV, no shame in skip, review strike, etc.
    – E_net4
    Commented Jun 12, 2019 at 13:15
  • 2
    @SirE_net4theDownvoter "the system expects us to check whether the question was properly tagged": and if the question was improperly tagged, it would be right to close it as "Unclear"
    – Raedwald
    Commented Jun 12, 2019 at 13:34
  • 1
    @yivi You can not correct the tags if you don't know what they ought to be. Because the question is unclear.
    – Raedwald
    Commented Jun 12, 2019 at 13:38
  • 1
    "f the question was improperly tagged, it would be right to close it as "Unclear"" Hmm yeah, hence the YMMV. It's an interesting perspective nonetheless: the system tells us that we should not be closing questions just because they are mistagged. Not everyone might agree, sure.
    – E_net4
    Commented Jun 12, 2019 at 13:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .