22

Tl;dr

When a user raises a flag containing certain keywords/regexes (like "GPT"), remind them to include rationale in the flag.

Rationale

One of the reasons that LLM flags can be slow to handle is that flags are often vague (like "I think this is ChatGPT generated" with no further info). Unfortunately, flags like that mean that (diamond) moderators have to spend time reading the posts to determine if anything in the text would indicate AI usage. However, if a flagger includes rationale, such as "I think this is ChatGPT-generated because [reason #1] and [reason #2].", this means those are immediately obvious for moderators.

While it wouldn't be perfect (some users would ignore it, some users might have trouble pinpointing why something seems LLM-generated), it seems like it's unlikely to do harm, but might help.

Proposal

If you raise a custom moderator flag on Stack Overflow containing this text:

I want this migrated to Ask Ubuntu!

You will get the following warning, and you'll need to click the Flag question button again actually to flag (the first time triggers the warning):

Think this question should be migrated to another site? Please first check that it is clearly-written and on-topic for the site you're recommending. If you aren't an experienced member of that site, please take some time to read their help center first; rejected migrations are frustrating to both askers and the good people on the site who must reject it. Thanks!

I propose a similar warning for flags containing the following phrases. It would also exclusively work for answers (not questions). It's debatable if this is worthwhile for Collectives Article flags. My opinion there is probably "Yes, if it isn't a ton of work".

  • "GPT" - match this anywhere in the flag

  • "LLM" - match this anywhere in the flag

  • "AI" - Match when surrounded by word-boundaries. E.g. /\bAI\b/

    • Matches "AI-suspected", "Likely-AI", "I think this is AI, but I'm not sure", etc.
    • Does not match, "This flag is courtesy of BRAIN, the Bizarre Research Association for Investigating Nonsense"

Here is some proposed phrasing. I release the following proposed phrasing under the CC0 1.0 Universal license*. If someone edits this phrasing, it might be a good idea to mention that you also agree for your change to be under that license too (of course, only if you agree to that).

Think this post was created using generative AI tools (such as ChatGPT)? Please read through the guidance on writing a good flag for suspected generative AI content. In brief, include why you think this post is AI-generated. Even if you're not sure, including more details makes it significantly easier for moderators to handle these flags, so it's strongly encouraged.


*This is hopefully sufficient for SE to use that without attributing. If it isn't, ping me in chat.

3
  • Would it be feasible to have a new type of flag dedicated to suspected LLM content? Similar to the plagiarism flag, an extra text box can be provided for a support link. I would use that to put the LLM content detector I have used to check the content against. In addition, separating a new flag type could potentially enable us a new review queue and allow the community to offload the moderators' burden.
    – ray
    Commented Nov 22, 2023 at 12:29
  • 44 variations of ChatGPT observed in the wild (and by extension, GPT). Commented Nov 22, 2023 at 14:05
  • @ray see also New flag for ChatGPT answers and Add flagging AI generated answers
    – starball
    Commented Nov 22, 2023 at 19:57

0

You must log in to answer this question.