Why is this a low quality answer? It gives a lot of information on how to solve the problem.
-
35Because the audit system is awful. It was deleted for policy reasons, not quality reasons, but the audit system can't tell the difference. (it's gpt generated)– Kevin BCommented Oct 1 at 18:36
-
@Kevin B Ahh I see. I didn't know that.– RichCommented Oct 1 at 18:38
-
14The combination of perfect grammar with utterly hopeless markup is a sign that it might have come from a bot. The phraseology is also very GPT-like.– Dan MašekCommented Oct 1 at 18:59
-
35This is why suspected AI posts should not be flagged as Very Low Quality or Not An Answer.– blackgreen ModCommented Oct 1 at 19:36
-
3IDK if one should be able to tell that was AI slop but but the answer itself has obvious quality issues. formatting is broken. that should have required edits at the very least. @BDL did you mean the answer? the question wasn't in question.– Christoph RackwitzCommented Oct 1 at 21:26
-
5I think voting "looks OK" was the only invalid move on that review. for how long did you look at the answer? how long do you usually spend on a review?– Christoph RackwitzCommented Oct 1 at 21:33
-
2@blackgreen maybe post an answer and expand on that?! :/ I do remember seeing comments (not necessarily from Mods) which were directing to use VLQ or NAA flags on AIGC.– M--Commented Oct 1 at 22:05
-
@M-- If you happen to know where those comments are, it would be good to know so that they can be corrected.– Ryan M ModCommented Oct 1 at 22:28
-
@RyanM needle in a haystack...– M--Commented Oct 1 at 23:55
-
3I admit to being a cheat and recommend doing the same. If you have any suspicions about the post you are reviewing, pop it open in another window. If it seems to have been deleted, problem solved; vote to nuke. If not, read through the comments, question and other answers for more context. There may be something subtle or outside your wheelhouse going on in the post.– user4581301Commented Oct 2 at 0:21
-
4related: I worry that ChatGPT answers could currently be too tricky for use in review audits– starballCommented Oct 2 at 3:50
-
3In general, if a diamond mod was involved in the deletion there might be more to the story than meets the eye. AI generated, plagiarism etc.– LundinCommented Oct 3 at 7:53
-
@RyanM blackgreen might know more; see this: meta.stackoverflow.com/questions/431746/…– M--Commented Oct 4 at 5:43
3 Answers
According to a comment under that answer (which only >10k users can see), the answer was deleted because the author posted AI generated answers. Afaik, it is impossible to notice that in a review queue unless you open the answer directly.
The audit system unfortunately doesn't take into account why a post was deleted. If it got deleted, the audit system assumes that it was a bad post.
-
15"unless you open the answer directly" -- IMHO doing so is not a bad idea in general. Commented Oct 1 at 18:56
-
1It seems that usually it's safer to only vote on bad answers to be deleted. I rarely if ever fail an audit on those.– RichCommented Oct 1 at 19:18
-
12This isn't quite right. The issue is that it was deleted with a helpful NAA/VLQ flag. If the flag had been declined or disputed, it wouldn't have been used as an audit. In this case, it was marked helpful by deletion, but VLQ flags also get automatically marked helpful when the post is edited, so mods wouldn't always have the option to decline.– Ryan M ModCommented Oct 1 at 22:25
-
1“Afaik, it is impossible to notice that in a review queue unless you open the answer directly.” - It’s fairly trivial to identify GenAI content if you are looking for GenAI content. If something was deleted, then I always go back to, “it can’t possibly be “ok”, now can it?” Commented Oct 2 at 11:16
You voted "looks OK" on that answer.
I think one should have noticed the broken formatting at least. That answer does not "look OK" in the sense that it is not fine to leave as is. It should have required editing or any other verdict.
Whether it was AI-generated or not is secondary to its broken formatting, for the purposes of this review audit.
-
15This is correct. Although I don't think users should flag AI-generated content as VLQ, and I don't generally think mods should mark such flags as "helpful" precisely because of the potential for bad audits, in this particular case, there was nothing wrong with the audit. This answer obviously did not look OK.– Cody Gray ModCommented Oct 2 at 0:58
-
4@CodyGray so then I wonder, do these things get discussed somewhere so the knowledge is spread among moderators? It's unrealistic to think the people pushing the flags in will change their ways so it really does fall on you guys to make the right choices.– GimbyCommented Oct 2 at 8:44
-
1@CodyGray If I'm not mistaken, "Edit" would also have failed the audit, so in that case there was something wrong with the audit. Commented Oct 3 at 6:40
-
1Yes, @Gimby, mods can and do discuss these things in private channels amongst ourselves. But it is still a bit of an issue that not all mods read everything ever posted. Something the site has lacked literally since day one has been comprehensive documentation. Fortunately, for mods in particular, there's been enough continuity of the team that there is almost always someone around who knows and can be asked. The same, unfortunately, cannot be said of staff. Another point worth mentioning: there are virtually no uncorrectable mistakes, and things like this are how mods and the community learn.– Cody Gray ModCommented Oct 3 at 8:48
-
2@DonaldDuck That's a fair point and may well be correct. In all honesty, I don't know enough about how the current iteration of review queues work to know whether that's a true statement or not. Which I guess goes back to Gimby's point. I could probably go find out, either via Meta or by asking around, but it's not something that I have the time or inclination to do, because it really doesn't matter all that much. (By coincidence, I happened to just come across a Meta Q&A that suggests "Edit" does not fail audits. So you might actually be wrong. But I still don't know for certain.)– Cody Gray ModCommented Oct 3 at 8:49
The answer ended up in the audits due to my mistake.
Answer had several pending flags which I handled in bulk while I was focusing on the AI part, and I missed that one of the flags was not a custom mod flag, but VLQ flag which should have been declined.
I have edited the answer which should (hopefully) remove it from being used in future audits.
-
10For the record, there was a time a couple years ago at the start of the AI wave where moderators would mark VLQ and NAA flags on AI posts as "helpful", with the rationale that bringing that content to our attention was always welcome regardless of the method. With time, and probably after the 2023 strike, this position has shifted: using the wrong flag ends up doing more harm than good, in the form of bad audits and risk of leaving users unaware of the AI ban. Also, the "Not An Answer" flag is plain incorrect, since AI posts do attempt to answer the question.– blackgreen ModCommented Oct 2 at 13:14
-
9The hoops y'all have to jump through to make the flag system work well are absurd. The biggest kudos in the world for the job you all do <3; I really wish the system got the overhaul it needed to make it less of a burden, too.– zcoop98Commented Oct 2 at 15:47
-
3@zcoop98 Thanks! Handling part is not that problematic, although it can cause issues like this one if you don't keep greater picture in head at all times ;) It would be better if audits would not be selected from posts that have multiple different flags cast, regardless of how they are handled. And it would be even better if we could more easily fix mistakes and change (dispute) how the particular flag was originally handled.– Dalija Prasnikar ModCommented Oct 2 at 17:52