146

January 10, 2024 Update

The experiment has now been graduated, and the banner has been enabled on Stack Overflow.


January 5, 2024 Update

We will be graduating this experiment and adding functionality to enable the "AI-generated content" policy banner on Stack Overflow. While the variant group had no significant impact on answer rates, there was a reduction in posts flagged for AI-generated content.

Results

We ran an A/B test on question visits between September 21st and November 28th, 2023. Our control group did not see the new AI content policy banner, and our test group did. We split traffic 50/50 on 1% of total traffic and measured the conversion rate (the percentage of users who viewed a question and went on to post an answer to that question), on a session level, of visits to a question to posting an answer on that question. Sampling on the top of the funnel was due to the volume of pageviews Stack Overflow gets in a given month.

Here’s what we found, with the null hypothesis being there is no difference in conversion between the two groups:

Group Conversion Rate Confidence Interval
Control (No AI Banner) 0.97% [0.88% - 1.08%]
Test (AI Banner) 0.98% [0.88% - 1.09%]

The test group only outperformed by 0.72%. With a p-value of 0.92, we fail to reject the null hypothesis that there is no difference in conversion rate between the two groups. This being the case, we feel confident that we are doing no harm to our answer rates by having the banner in place.

We also wanted to confirm that this banner reduced AI-related flags and flagged posts. We saw a reduction in the number of posts flagged for AI content and the total flags on those posts. Deletion rate on these answers was effectively the same for the test and control groups.

“AI Flags” in this case are flags that reference ChatGPT, LLMs, or the like, as we don’t have a distinct flag for this category at this time.

Group Questions Answered AI-Flagged Posts Flag Rate Deletion Rate
Test 58,040 625 1.08% 16.42%
Control 57,953 707 1.22% 16.49%

Decision

Next week we’ll be graduating this functionality in order to enable you to address concerns with the rise of AI-generated content. As we graduate the experiment, the following changes will take place:

  • All sites in the Stack Exchange network will be able to opt in (the feature is off by default, network-wide)

  • We will initially offer two banner text options that all sites in the Stack Exchange network can opt-in to. Those options are the following:

    • **Reminder**: Answers generated by Artificial Intelligence tools are not allowed on [Site Name]. Learn more

    • **Reminder**: Answers generated by Artificial Intelligence tools must be cited on [Site Name]. Learn more”

  • The banner will display once users select the answer field

  • The “new contributor” banner will no longer be shown in the answer field

  • All users will see this banner when posting an answer with the option to dismiss. Once dismissed, logged-in users will not see this banner again

Next Steps

We've created a separate post providing additional context on Meta Stack Exchange, and providing next steps on how sites will be able to opt in.


September 21, 2023 Update

The A/B test went live the morning of 21 Sept 2023 as did the new AI policy Help Center page. If you have any feedback or questions during this test, feel free to use an answer to this question to do so.

Summary

We are planning to run an A/B test for two weeks on Stack Overflow in late September to draw attention to the site-specific policy about the use of AI-Generated content when creating answers on the site.


As part of the agreements made during negotiations, we took the concerns raised during that discussion and worked on a design that we plan to A/B test here on Stack Overflow to see the effect it has on answering. We shared a preliminary design and plan with moderators a couple of weeks ago and are moving forward with sharing the plans for this project here on MSO. While the design is in development to be built, we still have time before we start the test to make adjustments to the text in the notice.

The Test Design

We went through a few variations for placement and functionality to get to the design we're proposing below. For the duration of the test, we'll remove the "new contributor" banner (example) on all answer fields for both test groups with the control group seeing nothing and the test group seeing the design below as they use the same space. Based on the outcomes of this test, we'll determine next steps for the "new contributor" banner.

We are planning to show this to all user types in the test group in the "Your Answer" area of the question page. Here's what the design we're going to test looks like:

The editor showing the banner design as described in the following paragraph.

Directly under the formatting bar, there's a blue bar with text that reads "Reminder: AI-generated answers are not allowed on Stack Overflow." It includes a "Learn more" link to the Help Center page about AI-generated content.

For the duration of the experiment we don't plan to make the banner dismissable as we want to know that the test group was shown the banner each time they posted an answer.

The Testing Phase

To test this design, we plan to do a two-week A/B test so that we can compare the new design with the current one and see if people in the new group have a reduced likelihood of posting AI-generated answers. We are currently only planning to run this test on Stack Overflow as it is one of the sites driving this change and the higher volume should help us assess the impact more quickly but are open to testing on or shipping to other sites once the initial tests are completed.

The metrics we plan to review include:

  • Reduction in posts containing AI-generated content as determined by the number of validated flags and mod messages sent to users in the test group.
  • Answer rate from test group drops at a rate consistent with the above metric with some allowance for cases that may go unflagged.

We will wait a week after the test ends before reviewing the data to allow community members time to flag and mods to handle those flags.

What we need from you

While we're interested in your thoughts about this test, we're planning to start with the design you see here. Based on the results, we'll determine the next steps based on whether we're seeing an impact in the test group.

As noted, the current copy is a suggestion. We've considered a few alternatives but would be interested in if you have any particular suggestions you feel should be included.

Some notes for your consideration - since this will be shown to everyone in the test, the bulk of people seeing this banner likely have no plan to post AI-generated content. Going into detail about policy and reasoning should be saved for the Help Center - which we are planning to update before this test begins.

  • The text should be short.
  • The text should be informative.
  • The text should be neutral.
  • We only want to link to the Help Center page.

If you have any questions, please let us know - we really appreciate your help and look forward to working collaboratively to find a solution that meets our goals. Once this test is live, we'll update this post to keep you informed.

53
  • 24
    "We will wait a week after the test ends before reviewing the data to allow community members time to flag and mods to handle those flags.' I'm really not sure a week is going to be long enough. I have tens of flags for potential AI generated content with 1 pre-dating the strike. The team have a significant back log to go through, so it's going to take them time. A week to work through the flags isn't going to give meaningful results when, presumably, older flags are going to be being worked on more often. Unless for the entire duration, and a week afterward, focus will be on new flags?
    – Thom A
    Commented Sep 7, 2023 at 21:28
  • 6
    All great points and I definitely understand the timing of when we review the data can be quite impactful to the results we see - I don't necessarily think we need all of the flags to be handled to accurately gauge the outcomes, though. Depending on how many flags there are and how many get handled, we may be able to project the percentage of flags that would be validated to the remaining flags from the test period. It wouldn't be precise but I'm under the impression that's a pretty standard way of reviewing data.
    – Catija Staff
    Commented Sep 7, 2023 at 21:49
  • 14
    Some new users seem to have problems to express themselves (read/write) in English. I think some might not understand the term AI-generated standalone. I'd change it to Remeber: AI.generated answers (like from ChatGPT)..... ChatGPT everyone knows... Just an idea...
    – Vickel
    Commented Sep 7, 2023 at 21:56
  • 26
    I think the phrasing of the warning strikes a good balance between 1) being brief enough to not annoy legitimate users and 2) clearly articulating the policy.
    – Nick ODell
    Commented Sep 7, 2023 at 22:05
  • 8
    Thanks - @Vickel I see your point. I do think we want to avoid naming specific companies in the UI directly but we can certainly include a non-exhaustive list in the Help Center information. I think we're open to alternative terminology if there's better options. :)
    – Catija Staff
    Commented Sep 7, 2023 at 22:05
  • 6
    @Vickel Would you mind putting a bit more detail into your concerns and making them into an answer? I think it's worth keeping in mind and I'm worried they'll get lost in the comments.
    – Catija Staff
    Commented Sep 7, 2023 at 22:26
  • 23
    @philipxy Describing images in text is generally considered a good practice, both for users of accessibility services and those who may be behind a firewall that blocks Imgur. While alt text is one way to do it, it can be nicer if it's worked into the normal text as well.
    – Ryan M Mod
    Commented Sep 7, 2023 at 22:44
  • 21
    @philipxy I mean - I did intentionally not use a freehand red circle because I was trying to avoid that exact response - amusingly, I was doing the calculus of whether I'd get more points for the freehand circle or lose them for "having" to circle the banner to make it more obvious to people. Oh well! The description is not actually for people who can see the images, though - I opted to explain it in the post rather than hiding the image description in the alt text.
    – Catija Staff
    Commented Sep 7, 2023 at 22:45
  • 7
    @philipxy so, here's what I would have said if you had taken a more constructive tack and maybe asked about our design decisions rather than just calling them "bad" - as noted in the question, we did go through several designs and the feedback from mods did indicate some concern about it being easily overlooked. The goal of this test is to see whether something subtle but insistent can change behavior rather than immediately jumping to options that might be more overt but potentially more likely to cause a reduction in good answers. If it seems to go unnoticed, we can try something else.
    – Catija Staff
    Commented Sep 8, 2023 at 1:43
  • 6
    @user13267 this text is for SO only at this point and will only be enabled on sites that expressly ask for it because they have policy that prohibits this content. The banner/notification was requested by mods and SO users repeatedly because the policy prohibiting the AI-generated content was easily missed and causing a negative impact on curators and moderators. My understanding is that some of the people who are mod messaged about this reply that they were unaware of the policy and some even apologize and state they will stop. As such, user education is needed, at least in the short term.
    – Catija Staff
    Commented Sep 8, 2023 at 1:52
  • 8
    We generally require an indication from the community that such changes are deemed necessary - usually in the form of a meta post, @user13267. We can not perpetually have a banner on the entire site to notify users about this policy. Mods don't want to keep the policy featured on meta and that bulletin isn't visible on all viewports anyway - that's why we're doing this at all. It's a test. As has been stated, if there's no difference in behavior between the control and test groups, that's useful information. But having a banner up six months ago doesn't mean today's posters saw it.
    – Catija Staff
    Commented Sep 8, 2023 at 2:14
  • 6
    @PetəíŕdTheLinuxWizard that's how we generally run A/B tests, yes. I have no clue what algorithm we use or why it might matter.
    – Catija Staff
    Commented Sep 8, 2023 at 5:36
  • 6
    @user13267 "Now that there is a standard (heuristics) to fall back on when actioning a post for (possibly) being AI, why still this debate of allowed and not allowed?" Because we want to prevent users posting AI posts in the first place and reducing the workload. All AI is banned because it is not feasible determining the accuracy of every AI answer. This is not the only reason, but before we can even discuss others, we would need the solution for that one. SO community voted for banning AI posts with significant majority meta.stackoverflow.com/questions/421831
    – Dalija Prasnikar Mod
    Commented Sep 8, 2023 at 6:54
  • 6
    @user13267 You have every right to state your opinion, and just the same I have the right to state mine. I am not dragging you into pointless discussions, it is you who are dragging everyone else.
    – Dalija Prasnikar Mod
    Commented Sep 8, 2023 at 7:59
  • 10
    Progress! Really thankful for all the hardwork that the people behind the scenes put in here, both on the community end and the corporate end. Thank you.
    – Travis J
    Commented Sep 8, 2023 at 19:06

3 Answers 3

74

As noted, the current copy is a suggestion. We've considered a few alternatives but would be interested in if you have any particular suggestions you feel should be included.

as suggested by @Catija and to sum up my comments in the OP:

  • Some new users seem to have problems to express themselves (read/write) in English. I think some might not understand the term AI-generated standalone. I'd change it to:

    Remember: AI-generated answers (like from ChatGPT)....

    because ChatGPT everyone knows...

I've observed the lack of language knowledge many times during my many years here on SO. Communication with users who are not native/2nd language speakers is a major problem.

  • Remember AI in German is KI (Künstliche Intelligenz), in Spanish+Portuguese it is IA (Inteligência Artificial), I have no idea how it could sound in languages I don't speak. People are used to their "local" abbreviation of Artificial Intelligence and might simply miss (in the sense of overreading/ignoring it) or misunderstand/don't understand it in English.
10
  • 31
    I think this is an excellent point, particularly since we have evidence from the company that suggests questions that have historically been deleted as being AI generated are more likely to originate from countries where English is not the primary language. We should do everything we can to make the banner understandable for users whose first language is not English. Commented Sep 7, 2023 at 22:45
  • 16
    Maybe writing it out would help? "Reminder: Answers generated by artificial intelligence (AI) are not allowed on Stack Overflow."
    – wjandrea
    Commented Sep 8, 2023 at 4:09
  • 1
    FWIW, in French it's also IA and it seems to be the equivalent of II in Russian. To get foreign terms for things, I like to go to the Wikipedia page, look at the list of other languages, and hover over them to see the article title.
    – wjandrea
    Commented Sep 8, 2023 at 4:22
  • 4
    The style guide says "Write for a 7th grade reading level.". For example, "Don’t complicate your ideas with business or insider jargon. Only use jargon when you have reason to believe it will improve user understanding.". I don't know if it applies here or not. Commented Sep 8, 2023 at 15:02
  • 1
    Unless you’re Norwegian, and hear «AI» much more than «KI»… :( Damn, can my fellow people learn how to use their own language? Commented Sep 8, 2023 at 16:39
  • Yep indeed, 'AI'="IA" also in French (= "Intelligence Artificielle")... // Typo in the Answer, ... quickly corrected...
    – chivracq
    Commented Sep 9, 2023 at 2:26
  • It should be "Machine Learning" instead. AI is the general term - we had AI before ML - ChatGPT is specifically using ML. Commented Sep 9, 2023 at 7:15
  • 2
    @wjandrea "Reminder: Answers generated by machine learning models are not allowed on Stack Overflow." Commented Sep 9, 2023 at 7:21
  • Since SO doesn't want to mention any specific product names in the banner (understandably), what about saying "GPT"? It does "resemble" ChatGPT and it is simply a NN model.
    – M--
    Commented Sep 13, 2023 at 15:36
  • Is "Reminder: Answers generated by artificial intelligence (AI) language models such as ChatGPT are not allowed on Stack Overflow." too long?
    – Someone
    Commented Sep 13, 2023 at 20:23
11

The metrics we plan to review include:

  • Reduction in posts containing AI-generated content as determined by the number of validated flags and mod messages sent to users in the test group.

I'm wary of going by number of validated flags and mod messages to qualify what's AI-generated and what isn't. The number of pending AI flags is in the thousands (last I heard- about 5k), and they're not going to magically all be handled any time soon.

To really get accurate data to interpret, either you're going to have to go by something that doesn't take so long to process, such as just whether a post has been flagged with keywords about AI, or you're going to have to wait for those flags to be handled, which is significantly blocked by the slow progression of heuristics testing and approval, which is significantly slowed down by the limited number of staff allocated to testing and approval (just Slate right now).

If you really go by whether a mod message was sent and flag validated, you will miss out on all the AI that is flagged but not handled by mods yet (because heuristics are severely underdeveloped (many are waiting for testing and approval)), and there's a whole heck ton of that.

1
  • 4
    Just launch it regardless of the results, the banner it's replacing was useless/redundant anyway.
    – Kevin B
    Commented Nov 22, 2023 at 15:33
1

The term used should be "language model" - because it is more accurate.

In this vein I suggest the message be:

Reminder: Answers from ChatGPT or other language models are not allowed.

And maybe like a short wiki describing what hallucinations are and for that including the case where a lawyer made a fool of himself using ChatGPT.

10
  • 24
    I think that "machine learning" is potentially too technical a term, and people may not understand that LLMs are "machine learning". I would believe that they're more likely to understand that they are "artificial intelligence". (also, in the event that someone decides to generate answers by some sort of non-ML "AI", then...that also seems bad)
    – Ryan M Mod
    Commented Sep 9, 2023 at 7:26
  • @RyanM Yeah well I suggested an alternative variant - implying the actual term - and citing an compelling case. Commented Sep 9, 2023 at 7:31
  • 3
    I don't think we should use that alternative, either, as we would also not permit answers from non-ChatGPT AIs.
    – Ryan M Mod
    Commented Sep 9, 2023 at 7:48
  • 1
    @RyanM Updated my answer - and used "language models" instead. Commented Sep 9, 2023 at 7:51
  • 3
    "Language models" is perfect :D
    – Cerbrus
    Commented Sep 9, 2023 at 7:51
  • 4
    The term used in the community policy is AI, specifically "generative AI" but in this context I assume "generated" implies "generative", though this isn't my field. On the other hand, the official policy only covers GPT.
    – wjandrea
    Commented Sep 9, 2023 at 13:40
  • 4
    A rose by any other name. Why does the term matter since no AI generated content is allowed - the purpose of the banner isn't to provide some technically precise definition. If you can get some video game to generate SO questions then that's very likely off-topic and against policy too. There exists no situation where something AI-related can generate a valid SO question, so there's no need to split hairs over it.
    – Lundin
    Commented Sep 12, 2023 at 13:37
  • 2
    Instead of getting mired in the technical details of what term is both broadly-understood and fully accurate (goals that are often at odds), especially because answers generated by any future or past artificial agent are presumably not allowed, regardless of the technical details, why not focus on what we do want? "All answers must be written by a human" or something similar.
    – lmhamilt
    Commented Sep 12, 2023 at 20:56
  • @lmhamilt Well so we are technically accurate and not confusing. - besides I think the term "Language models" is a good fit. Commented Sep 12, 2023 at 23:27
  • I work with AI professionally and I wouldn't describe ChatGPT as a "language model", at least not colloquially (but as a technical description, sure). AI and ML are the much more recognised terms (although "language model" has been gaining popularity). AI is more recognised than ML, and while AI is broad, it's also specific enough: I don't think anyone will try to post a self-driving-car-generated answer any time soon, but the distinction between pasting text as is from ChatGPT versus writing an answer based on the output of some model should be clarified under the "Learn more" link.
    – NotThatGuy
    Commented Sep 14, 2023 at 13:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .