80

I recently raised a flag on a Question that's generated by an LLM (quoted at the end). The signs are clear, including the fact that the "Copy" sections remained, there are sections for the user to fill in (that is not code, so therefore not a duplicate of Should we flag human-written questions that use code generated by ChatGPT?), and the section of python code that "tried" was entirely missing their actual attempt. It was clear that it was generated by an LLM.

The flag, however, was declined as it being "about" an LLM. There is, however, no indication in the text it is about content from an LLM (such as "CrapGPT suggested this answer").

I flagged this again, because users aren't allowed to respond to flags any other way, further advising my conviction, and that a decline for such a flag raised in good faith isn't ok. It was declined again stating there is "no evidence" it is written by an LLM; the entire post is evidence enough.

This is a clear stance change from the moderator team, as I flagged a similar style post in the SG and that was deleted on the 12th. I can find no evidence of a change of stance posted here on Meta of the mods changing their stance.

Therefore I ask:

  1. When did Questions entirely generated by AI become permissable, and why was this not communicated?
  2. Why are Questions, like the below, flagged as AI generated declined, rather than disputed, when raised in good faith? This deters users from raising said flags, which only further encourages LLM content on the site, no?

Content of flagged post:

This is the exact content of the post, no adjustments have been made other than that made by putting the content in a quote block.

"I'm working on a project where I need to fetch data from several related tables in a PostgreSQL database. The tables include millions of rows, and my current query is slow. Here's a simplified version of the query:

sql Copy code SELECT a.name, b.detail, c.status FROM table_a a JOIN table_b b ON a.id = b.a_id JOIN table_c c ON b.c_id = c.id WHERE a.created_at >= '2024-01-01' AND c.status = 'active'; I've already added indexes on a.created_at, b.a_id, and c.id. However, the query still takes over 30 seconds to execute.

What I've tried:

Analyzing the query plan using EXPLAIN ANALYZE (but I'm not sure how to interpret some parts of it). Adding more specific WHERE conditions (minimal improvement). Experimenting with different types of JOINs.

I recently encountered the phrase 'What were you expecting?' in the context of [describe situation, language, or framework, e.g., Python, JavaScript, etc.]. Here's the scenario:

python Copy code

Example code causing the issue

result = some_function(input_value) print(result)

Error or unexpected output

The output I got was [insert output or error message]. Based on the documentation or my understanding, I was expecting [describe what you thought should happen].

Contents of flags

My text in italics, moderator in quote blocks.

This is clearly generated by an LLM, the "sql Copy code" and "python Copy code" and the sections for the OP to put their error cement it's pre-generated.

Declined - Questions *about* AI-generated content should be moderated or edited as usual instead of flagged. Please see: https://meta.stackoverflow.com/a/422440

I disagree this is about an LLM. There is place holder text here for the author to fill in and hasn't and no where do they mention this is from an LLM. At best my flag should be disputed, not declined, but I still disagree with this. If this is declined, I'll make a meta post about it, because a decline is wrong here.

Declined - There isn’t enough here to conclude that the whole question is AI-gen. The only markers are around code blocks, it’s reasonable to assume it’s a question “about” AI code

48
  • 7
    Possibly related/duplicate: Should we flag human-written questions that use code generated by ChatGPT?
    – Andrew T.
    Commented Nov 23 at 12:09
  • 6
    So you are telling me that this text "What I've tried: Analyzing the query plan using EXPLAIN ANALYZE (but I'm not sure how to interpret some parts of it). Adding more specific WHERE conditions (minimal improvement). Experimenting with different types of JOINs." looks AI generated? I beg to differ. I mean, sure, I see a lot of things that scream AI but overall it does not look 100% AI generated to me as you claim. Are you saying that the user asked chatgpt or whatever to write a SO question for them and they copy pasted it verbatim? Commented Nov 23 at 12:39
  • 7
    That is exactly what I am saying, @MarcoBonelli . That the OP effectively asked for a template from CrapGPT, and used it for their post without even filling it in.
    – Thom A
    Commented Nov 23 at 12:56
  • 54
    the question in question literally contains the text quoted in this question. anything that looks like a placeholder was there in the original question already. it's literally a fill-in-the-blank provided to the asker by some LLM. it literally contains telltale signs such as sql Copy code that only happen when someone copies text out of ChatGPT's page. that is lazy beyond belief. this is an AI-generated question. nobody can deny that in good faith. Commented Nov 23 at 13:42
  • 8
    now, if the debate was about "do we tolerate questions that were run through ChatGPT to pretty them up?", I'd still say no. that's a terrible policy. ChatGPT obfuscates by repetition and fluff. we can handle ESL syntax/spelling artefacts, we have since the inception of this site. we don't need something as heavy-handed as ChatGPT to "improve" questions. Commented Nov 23 at 13:45
  • 6
    personally, I rarely bother flagging ChatGPT questions because I get that people are looking for solutions, didn't mean anything by it, probably just used it as a fancy language translation tool from their native language. I point out the issues with the ChatGPT obfuscation and maybe beat the post into shape. I do flag ChatGPT answers because that's often not a measure to pretty the answer up, but an attempt to gain unearned rep points (cheating the system). such behavior ruins communities. Commented Nov 23 at 13:56
  • 7
    I'd hope and assume that there is no change in policy, but rather the moderator in question is just not as adept at spotting LLM-generated questions as you are. Commented Nov 23 at 14:14
  • 3
    @ChristophRackwitz If that's the case, people should be using DeepL or other translation specialized tools, not ChatGPT. Wikipedia has such a policy. Commented Nov 23 at 15:43
  • 48
    I gave up flagging anything as AI generated after I flagged a question where the poster said in the first sentence that it was made with ChatGPT and it was declined for not enough proof. Why even bother?
    – Shawn
    Commented Nov 23 at 15:52
  • 3
    Would you share the exact wordings of your flag and decline message?
    – M--
    Commented Nov 23 at 23:08
  • 4
    "I think the Asker asked 'ChatGPT' to generate a Template" yes I already mentioned this, @chivracq , and well implied (at worst) in the OP. "Fishy" isn't likely to be a good indication something is LLM generated though, and I would expect a decline if I said that; there are much better indicators throughout the post, which I denote.
    – Thom A
    Commented Nov 24 at 11:42
  • 5
    It's in the OP multiple times as well, @chivracq . "there are sections for the user to fill in", "There is place holder text here for the author to fill in and hasn't". The comments also call it out as well. It's been covered.
    – Thom A
    Commented Nov 24 at 12:03
  • 3
    And this most certainly appears to be a 1:1 copy, @nbk. If it weren't, it wouldn't contain text such as "[describe situation, language, or framework, e.g., Python, JavaScript, etc.]" and " [describe what you thought should happen]." It would, instead, have actual meaningful information, not place holders.
    – Thom A
    Commented Nov 24 at 16:51
  • 6
    No, that is a literal copy and paste from the original post, @dan1st , as I stated in the post, and why I put it all in a quote block. If I altered/omitted some of that content, I would have been explicit in stating so. Chris linked to the post earlier (now deleted, so 10k+ only) in the comments, which I'd personally omitted to "protect the 'innocent'".
    – Thom A
    Commented Nov 24 at 16:55
  • 3
    My read is that the policy hasn't changed, but that the moderator simply disagreed with your assessment it was AI generated. I also think that your comments are unclear (especially the second one). Commented Nov 25 at 13:02

3 Answers 3

2

When did questions generated by AI become allowed?

Short answer: Policy is unchanged and AI generated questions are still not allowed on the site.

Long answer: Moderating questions that used AI in some way is complicated for many reasons. The fact that moderators cannot transparently disclose everything about moderating AI posts to avoid exposing heuristics does not make it easier either.

Primary concern and reason we don't allow AI posts is accuracy as it is explained in Policy: Generative AI (e.g., ChatGPT) is banned

We (moderators and experts) cannot successfully moderate AI generated answers based on accuracy alone. We must remove AI generated answers on basis that they are either completely or partially AI generated. We cannot verify and know which parts are correct and which may be incorrect.

Even if user has vetted the answer and verified the AI output, we cannot verify their claims, nor such vetting means that answer is actually correct. Code in such answers may seemingly work, but it can still be faulty or dangerous code. Explanations may sound plausible, but they can be completely wrong.

Trying to moderate AI answers based on accuracy would be arbitrary and error prone. To ensure fairness and avoid any potential harm all answers where AI was used in some way will be deleted. AI generated answers, articles, and other posts where accuracy matters can cause significant harm to the site and we have zero tolerance for those.

However, with questions accuracy is not a concern.

Questions don't have to be accurate in any way. They can be utterly wrong. Person asking question does not have to know what they are doing, they can have complete misconception about how something works, can use dangerous or poorly written code. The whole reason why they are asking question in the first place (unless it is meant to be self-answered one) is that they need help with some problem.

Because accuracy is not important for questions and AI generated questions don't cause any more harm than regular unsuitable questions, moderating those AI question is not very high on moderators' priority list.

In addition to having lower priority, there are other complications and concerns and while I could mention some, some of those I cannot freely disclose without directly or indirectly disclosing used heuristics and how we moderate AI posts in general, so I will not write about any of those. All those concerns have impact on how each individual flag on AI assisted questions are handled, and there is also more variability in how each particular moderator will handle those.


I know that this is not ideal situation for curators who want to flag AI assisted questions and not have their flags declined.

The best guidance without going into other details I can give is following:

  • handle questions where AI is used the same way you would any other question - close vote and downvote where appropriate
  • leave a comment (optional) to users with link to AI policy to notify those who may not be aware of the policy
  • if you see some other (potential) abuse of the system use custom mod flag, you can also mention suspicion of AI

That does not mean you should not flag AI questions, but merely that you can expect that some of those flags have higher chances of being either declined or marked helpful but post will not be deleted.

Note: When asking on Meta about why some AI flag is handled in particular way, unless moderator can say "Oops, this was a mistake", we generally cannot fully elaborate why we handled something in certain way. But such Meta posts and moderator actions are being discussed internally and there are multiple moderators involved in the process.

12
  • 5
    Thanks Dalija. I must admit, this doesn't fulfil me with confidence of flagging questions in the future that appears to be entirely generated by an LLM. Despite the fact that you state the policy hasn't changed, the end statements do appear to imply that there is a high likelihood your flag will be declined because. (Yes, that is the end of the sentence.) Even if it is a question, not an answer, I would still have hoped that such content would be fully applicable to the site rules on LLM content; that it isn't is disappointing at best. I'll cast down and close votes in future only.
    – Thom A
    Commented Nov 26 at 21:27
  • 10
    Although I have accepted the answer, I do want to make it transparent that I've also downvoted the answer; it is the "solution", hence the accept, but I can't say I agree with it. I'm stating how I have voted, because I feel it important when I am the only one who can accept an answer here, and this the acceptance could be seen as agreement with the answer.
    – Thom A
    Commented Nov 26 at 21:29
  • 4
    "When asking on Meta about why some AI flag is handled in particular way, unless moderator can say "Oops, this was a mistake", we generally cannot fully elaborate why we handled something in certain way." - I take it the motivating example was not determined to be a mistake, then? Commented Nov 26 at 22:14
  • 2
    Given that information, it might be worth reconsidering Should moderators delete AI-generated questions in Staging Ground? or add information (somewhere on MSO, possibly even Reviewing in the Staging Ground: A practical guide) on how Staging Ground reviewers should treat AI generated questions.
    – dan1st
    Commented Nov 26 at 23:15
  • 4
    This answer seems to be mudding the lines to me, it boils down to "blatant AI generated questions can be allowed and flagging them is rolling a dice" but starting with "AI generated posts are banned, period" as short answer. It doesn't make sense, the post in the question is blatantly an AI template (on top of being bad). So either there's a change of policy and this kind of posts are OK, or it's banned and blatant cases like that should never get a declined flag. Real point: mods should not decline flags on blatant AI usage, even when not deleting the post, you're deterring users to flag.
    – Tensibai
    Commented Nov 27 at 13:31
  • 2
    You know, I can understand people get fed up with Stack Overflow. This just sucks. They setup a policy and then we get the ambiguity dance again through vague flag declines and a meta answer which goes "well yes but actually no".
    – Gimby
    Commented Nov 27 at 14:42
  • 2
    @ThomA I fully understand from where you are coming from. I never expected this answer would be a popular one.
    – Dalija Prasnikar Mod
    Commented Nov 27 at 19:36
  • @Tensibai Like I said, moderating AI suspected questions is extremely complex and there are many contributing factors that impact the final result, including whether flag will be declined or not. There is always some difference in how different moderators handle particular flag, but with AI questions maintaining consistency can be hard even for single moderator. Sometimes there are not right or wrong solution and yes, it may look like we are rolling the dice at times. It is that kind of difficult, even in "blatantly obvious" cases.
    – Dalija Prasnikar Mod
    Commented Nov 27 at 19:44
  • 2
    @Gimby I am really sorry that moderators cannot be more transparent. This is why it took me three days to write this answer. If I could be less ambiguous, trust me I would.
    – Dalija Prasnikar Mod
    Commented Nov 27 at 19:47
  • @DalijaPrasnikar no ill will towards you, it's more that I have developed more appreciation for people that have thrown in the towel over the years. What this also makes clear is that moderators are not "community members with elevated privileges" anymore. You're instructed to keep secrets, essentially. Not a good feel either.
    – Gimby
    Commented Nov 28 at 7:39
  • 1
    @Gimby In this case it is not that we are instructed to keep secrets. We are the ones that need some things to be secret. We need to keep information about how we moderate AI generated content private as being fully transparent would make our job much harder, if not impossible. See (Interim) Policy on AI-content detection reports Also this information is not moderator only as Heuristics Working group involves other SO users. I was member of that group before I became mod.
    – Dalija Prasnikar Mod
    Commented Nov 28 at 8:17
  • I appreciate this response. Even with its inevitable ambiguity, it gives me some direction on how to handle AI questions. I think moving forward I will not flag an AI question if it's a single occurrence but will flag if I find out a user have posted multiple AI questions.
    – M--
    Commented Nov 29 at 17:48
27

This question is a clear cut. To avoid giving away how we identify AIGC, I would refrain from pointing out the specifics, but there are multiple indicators literally from the start to the very end to support your flag. So, I think the flag was declined due to an oversight. However, if we know the exact wording of your flag and decline message (if you have shared specifics on why this is AIGC, please retract those parts) we can better judge the decision made here.

When did Questions entirely generated by AI become permissable, and why was this not communicated?

I suppose this is a rhetorical question. Questions which are partly or entirely generated by LLMs are not allowed on Stack Overflow (see this exception).

Why are Questions, like the below, flagged as AI generated declined, rather than disputed, when raised in good faith? This deters users from raising said flags, which only further encourages LLM content on the site, no?

I don't agree with this. I don't think flags are declined only when they are raised in "bad faith". Handling AIGC flags is a heavy lift. So, if a user raises those flags frequently in error, deterring them from doing so is actually a desired outcome (I am not suggesting that's you, especially considering that your flag should've been marked helpful; moreover, a single declined flag would not result in being deterred/flag-banned).

0
-11

I flagged this again, because users aren't allowed to respond to flags any other way,

That's not true. The correct way to respond to a declined flag is to post a question as you did here. That is the other way. A second flag is risky and may result in the original moderator processing the new flag.

Perhaps there should be a mid-level alternative, an "I disagree with this decline". That's a sensible feature request that I personally would support.

Alternately, perhaps if someone flags, is declined, and flags again, a different moderator should be assigned to process the flag. That's also a sensible feature request that I personally would support. Others might not, particularly on other stacks, where there are fewer moderators.

Some might reasonably argue that asking a meta question is the best response. It's not trivial to do, so people won't do it on every declined flag. It's not terribly difficult, so people can always do it. It brings greater community visibility instead of just a second moderator. It supports coming to a consensus decision rather than relying on the judgment of a single person.

This answer suggested:

a single declined flag would not result in being deterred

I think it should be obvious to most of us that it depends on the person. Some people would ignore a single declined flag. Others would view it as a clear message to stop that behavior. Others might view it as a message that StackExchange does not want their contributions, so they should leave the site.

In general, I would agree that flags should be declined rarely. Yes, if someone is flagging incorrectly, decline. But the correct response to a marginal flag that is marginally refused should be to mark helpful. This is especially so if multiple people raise similar flags. That tends to be a sign that the question needs to be edited. The flag, while not supporting deletion, was still helpful in noting that.

Another possibility is that the guidance for when to flag a question as AI generated should be clearer.

Moderators also can post questions like this when they feel that the community is flagging things more than is necessary.

5
  • I vaguely recall reading somewhere that you could reflag if you disagreed and wanted another mod to look... but don't remember where. maybe I'm just making this up.
    – starball
    Commented Nov 25 at 2:57
  • 6
    @starball Perhaps you're thinking of "Flag for moderator attention and request a second moderator to have a look." from What recourse do I have if I believe a moderator has abused their privileges?
    – cocomac
    Commented Nov 25 at 3:51
  • 5
    This isn't a case, even alleged, of a moderator abusing their privileges, but a case of a declined flag. The appropriate response for the flagger is to, apparently, "Do nothing" meta.stackoverflow.com/q/252063
    – cigien
    Commented Nov 25 at 6:02
  • 2
    What point are you trying to make with this answer?
    – Cerbrus
    Commented Nov 25 at 8:32
  • 1
    @cigien While I agree, I was just trying to provide a reference to what starball may have been referring to (even if the question doesn't really apply). This is just my opinion, but considering the bullet point examples, I don't think the suggestions in that answer are terrible options if someone strongly disagrees with a flag outcome.
    – cocomac
    Commented Nov 25 at 8:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .