Update
The A/B test went live the morning of 21 Sept 2023 as did the new AI policy Help Center page. If you have any feedback or questions during this test, feel free to use an answer to this question to do so.
Summary
We are planning to run an A/B test for two weeks on Stack Overflow in late September to draw attention to the site-specific policy about the use of AI-Generated content when creating answers on the site.
As part of the agreements made during negotiations, we took the concerns raised during that discussion and worked on a design that we plan to A/B test here on Stack Overflow to see the effect it has on answering. We shared a preliminary design and plan with moderators a couple of weeks ago and are moving forward with sharing the plans for this project here on MSO. While the design is in development to be built, we still have time before we start the test to make adjustments to the text in the notice.
The Test Design
We went through a few variations for placement and functionality to get to the design we're proposing below. For the duration of the test, we'll remove the "new contributor" banner (example) on all answer fields for both test groups with the control group seeing nothing and the test group seeing the design below as they use the same space. Based on the outcomes of this test, we'll determine next steps for the "new contributor" banner.
We are planning to show this to all user types in the test group in the "Your Answer" area of the question page. Here's what the design we're going to test looks like:
Directly under the formatting bar, there's a blue bar with text that reads "Reminder: AI-generated answers are not allowed on Stack Overflow." It includes a "Learn more" link to the Help Center page about AI-generated content.
For the duration of the experiment we don't plan to make the banner dismissable as we want to know that the test group was shown the banner each time they posted an answer.
The Testing Phase
To test this design, we plan to do a two-week A/B test so that we can compare the new design with the current one and see if people in the new group have a reduced likelihood of posting AI-generated answers. We are currently only planning to run this test on Stack Overflow as it is one of the sites driving this change and the higher volume should help us assess the impact more quickly but are open to testing on or shipping to other sites once the initial tests are completed.
The metrics we plan to review include:
- Reduction in posts containing AI-generated content as determined by the number of validated flags and mod messages sent to users in the test group.
- Answer rate from test group drops at a rate consistent with the above metric with some allowance for cases that may go unflagged.
We will wait a week after the test ends before reviewing the data to allow community members time to flag and mods to handle those flags.
What we need from you
While we're interested in your thoughts about this test, we're planning to start with the design you see here. Based on the results, we'll determine the next steps based on whether we're seeing an impact in the test group.
As noted, the current copy is a suggestion. We've considered a few alternatives but would be interested in if you have any particular suggestions you feel should be included.
Some notes for your consideration - since this will be shown to everyone in the test, the bulk of people seeing this banner likely have no plan to post AI-generated content. Going into detail about policy and reasoning should be saved for the Help Center - which we are planning to update before this test begins.
- The text should be short.
- The text should be informative.
- The text should be neutral.
- We only want to link to the Help Center page.
If you have any questions, please let us know - we really appreciate your help and look forward to working collaboratively to find a solution that meets our goals. Once this test is live, we'll update this post to keep you informed.