January 10, 2024 Update
The experiment has now been graduated, and the banner has been enabled on Stack Overflow.
January 5, 2024 Update
We will be graduating this experiment and adding functionality to enable the "AI-generated content" policy banner on Stack Overflow. While the variant group had no significant impact on answer rates, there was a reduction in posts flagged for AI-generated content.
Results
We ran an A/B test on question visits between September 21st and November 28th, 2023. Our control group did not see the new AI content policy banner, and our test group did. We split traffic 50/50 on 1% of total traffic and measured the conversion rate (the percentage of users who viewed a question and went on to post an answer to that question), on a session level, of visits to a question to posting an answer on that question. Sampling on the top of the funnel was due to the volume of pageviews Stack Overflow gets in a given month.
Here’s what we found, with the null hypothesis being there is no difference in conversion between the two groups:
Group | Conversion Rate | Confidence Interval |
---|---|---|
Control (No AI Banner) | 0.97% | [0.88% - 1.08%] |
Test (AI Banner) | 0.98% | [0.88% - 1.09%] |
The test group only outperformed by 0.72%. With a p-value of 0.92, we fail to reject the null hypothesis that there is no difference in conversion rate between the two groups. This being the case, we feel confident that we are doing no harm to our answer rates by having the banner in place.
We also wanted to confirm that this banner reduced AI-related flags and flagged posts. We saw a reduction in the number of posts flagged for AI content and the total flags on those posts. Deletion rate on these answers was effectively the same for the test and control groups.
“AI Flags” in this case are flags that reference ChatGPT, LLMs, or the like, as we don’t have a distinct flag for this category at this time.
Group | Questions Answered | AI-Flagged Posts | Flag Rate | Deletion Rate |
---|---|---|---|---|
Test | 58,040 | 625 | 1.08% | 16.42% |
Control | 57,953 | 707 | 1.22% | 16.49% |
Decision
Next week we’ll be graduating this functionality in order to enable you to address concerns with the rise of AI-generated content. As we graduate the experiment, the following changes will take place:
All sites in the Stack Exchange network will be able to opt in (the feature is off by default, network-wide)
We will initially offer two banner text options that all sites in the Stack Exchange network can opt-in to. Those options are the following:
**Reminder**: Answers generated by Artificial Intelligence tools are not allowed on [Site Name]. Learn more
**Reminder**: Answers generated by Artificial Intelligence tools must be cited on [Site Name]. Learn more”
The banner will display once users select the answer field
The “new contributor” banner will no longer be shown in the answer field
All users will see this banner when posting an answer with the option to dismiss. Once dismissed, logged-in users will not see this banner again
Next Steps
We've created a separate post providing additional context on Meta Stack Exchange, and providing next steps on how sites will be able to opt in.
September 21, 2023 Update
The A/B test went live the morning of 21 Sept 2023 as did the new AI policy Help Center page. If you have any feedback or questions during this test, feel free to use an answer to this question to do so.
Summary
We are planning to run an A/B test for two weeks on Stack Overflow in late September to draw attention to the site-specific policy about the use of AI-Generated content when creating answers on the site.
As part of the agreements made during negotiations, we took the concerns raised during that discussion and worked on a design that we plan to A/B test here on Stack Overflow to see the effect it has on answering. We shared a preliminary design and plan with moderators a couple of weeks ago and are moving forward with sharing the plans for this project here on MSO. While the design is in development to be built, we still have time before we start the test to make adjustments to the text in the notice.
The Test Design
We went through a few variations for placement and functionality to get to the design we're proposing below. For the duration of the test, we'll remove the "new contributor" banner (example) on all answer fields for both test groups with the control group seeing nothing and the test group seeing the design below as they use the same space. Based on the outcomes of this test, we'll determine next steps for the "new contributor" banner.
We are planning to show this to all user types in the test group in the "Your Answer" area of the question page. Here's what the design we're going to test looks like:
Directly under the formatting bar, there's a blue bar with text that reads "Reminder: AI-generated answers are not allowed on Stack Overflow." It includes a "Learn more" link to the Help Center page about AI-generated content.
For the duration of the experiment we don't plan to make the banner dismissable as we want to know that the test group was shown the banner each time they posted an answer.
The Testing Phase
To test this design, we plan to do a two-week A/B test so that we can compare the new design with the current one and see if people in the new group have a reduced likelihood of posting AI-generated answers. We are currently only planning to run this test on Stack Overflow as it is one of the sites driving this change and the higher volume should help us assess the impact more quickly but are open to testing on or shipping to other sites once the initial tests are completed.
The metrics we plan to review include:
- Reduction in posts containing AI-generated content as determined by the number of validated flags and mod messages sent to users in the test group.
- Answer rate from test group drops at a rate consistent with the above metric with some allowance for cases that may go unflagged.
We will wait a week after the test ends before reviewing the data to allow community members time to flag and mods to handle those flags.
What we need from you
While we're interested in your thoughts about this test, we're planning to start with the design you see here. Based on the results, we'll determine the next steps based on whether we're seeing an impact in the test group.
As noted, the current copy is a suggestion. We've considered a few alternatives but would be interested in if you have any particular suggestions you feel should be included.
Some notes for your consideration - since this will be shown to everyone in the test, the bulk of people seeing this banner likely have no plan to post AI-generated content. Going into detail about policy and reasoning should be saved for the Help Center - which we are planning to update before this test begins.
- The text should be short.
- The text should be informative.
- The text should be neutral.
- We only want to link to the Help Center page.
If you have any questions, please let us know - we really appreciate your help and look forward to working collaboratively to find a solution that meets our goals. Once this test is live, we'll update this post to keep you informed.