I just got Late Answers Queue review item #33495800, which used a now-deleted ChatGPT-generated answer, a/11107541 as a review audit.
Since it was an audit, comments were hidden. There was a comment saying that the answer where someone had sniffed it out as being written by ChatGPT, but I couldn't see it until after I passed the audit. And I'm guessing that not all deleted-for-ChatGPT answers will have such comments on them.
I worry that this kind of thing can trip up reviewers (who aren't highly aware of the ChatGPT situation and haven't built up a sense for telltale ChatGPT signs) for some of the the very reasons that ChatGPT is banned (at the time of this writing): It writes in an authoritative style with proper spelling and grammar such that the many people/reviewers without enough subject-matter-knowledge/expertise to easily know whether it's wrong right off the bat will think it's probably right, and even people with subject-matter-expertise could take a while to verify the correctness.
If I wasn't so aware of the ChatGPT situation, having followed the meta posts and official announcements, and having gained some experience spotting such answers and learning the telltale signs from searching for ChatGPT posts when the flood started, I don't think I would have been able to know that I should have custom-flagged for ChatGPT or commented for suspicion of ChatGPT, linking to the ChatGPT policy page.
I don't think many people have even seen this MSO post on what to do if one suspects an answer is ChatGPT-generated. At the time of this writing, it has under 1k views.
Possible routes forward I thought of:
Give clear guidance to reviewers on how to spot possible ChatGPT answers (but I'm not sure if we really want to do this right now, since we seem to want to avoid doing that to avoid helping ChatGTP-answerers learn how to avoid detection), and tell them what to do when they are confident that they have found such an answer.
I'd suggest this guidance to be given somewhere visible in the review UI. I'm not sure if the review guidance FAQ post on MSE is an appropriate place, since ChatGPT policies are currently set per-site.
Somehow prevent ChatGPT-flagged answers from being used in review audits.
Thoughts?