In the coming weeks, we will be conducting a new experiment aimed at improving the quality of questions on the site. This experiment is designed to use AI to help users format their question body and code according to the language they are using, making it easier for reviewers and curators to focus on the content and meaning of the question over superficial mistakes such as typos or rough formatting. The formatter intends to format code and fix grammatical errors (including typos). The ultimate goal is to enhance the quality of questions on Stack Overflow by suggesting edits during the question-asking workflow, and as a result, improve the overall user experience on the platform. The edits suggested by the formatting assistant should be similar to curators’ edits: cleaning up the content of the post, without fundamentally altering its meaning.
These experiments are designed to give us insights into how we can help improve the question-asking experience. We aim to have these experiments finished in approximately two weeks. We also hope the improved content clarity benefits answerers and curators on the site.
How we’re expecting this test to look on the platform
In the spirit of full transparency, I would like to detail how we plan to conduct the experiment and also share how these experiments will look. We will be testing two layouts to see which leads to higher usage. We aim to determine the most suitable form this feature should adopt if it proves beneficial.
Test Variant #1: Ask Wizard
Test Variant #2: AskV2
Our experiment plan
By comparing different variants, including control groups without AI-suggested edits and groups with AI-suggested edits, the experiment will determine if the AI-generated edits lead to more completed and successful questions. Ultimately, the experiment seeks to validate the hypothesis that AI assistance in the question-asking workflow can lead to faster question completion, fewer abandoned questions, a smaller work burden for curators, and more successful questions, thereby contributing to the platform's growth and user satisfaction.
We’re launching 2 variants, including the control group in the Ask Wizard and one variant in the AskV2 workflow. Thus, we’ll split askers equally across all variants, including the control group. We will also be experimenting with minimizing the Ask Wizard, eliminating the second editor on the first page and the entire second page. We will be consolidating the workflow to one unified page.
Our initial target is to collect data from up to 10,000 question-asking sessions per variant over the course of the experiment. We will be monitoring results throughout the experiment and may adjust our target number of sessions based on what we see.
We hope this experiment allows us to gain insights into 4 key points:
- Does the formatting assistant improve the rate at which questions receive answers?
- Does the formatting assistant increase or reduce the number of edits performed on the question?
- Are questions that went through formatting assistance abandoned by their authors at a lower rate?
- Do people post questions they begin to draft more or less often when they receive formatting assistance?
Although we have a few initial variants planned, our intention is to adapt and refine the plan based on insights gained from previous outcomes and community feedback. If there is a specific question or test that you strongly believe should be included in this experiment, we encourage you to inform us in the responses below. We will consider incorporating it into our plans for the current and future iterations.
We are open to feedback
Members of the Product, Development, and Community teams will be taking time to monitor this post for any feedback you may have. We encourage you to share any questions, comments, or concerns that you may have in the answers below, and we’ll do our best to respond to them. We need community feedback in order to ensure that what we build improves the user experience for everyone. The input provided here can be made into action items we can present to the development team.
Update
We hear your feedback around the content formatting experiment released yesterday, and have decided that we have gathered sufficient input on a couple of things that were being tested, so we are going to be turning off the question formatting assistant experiment. Because we have some parts of the experiment that are still in-flight (efficacy when the tool is used by new users with poorly formatted questions) and wrapping up data collection, it won’t be turned off immediately, but before the end of the weekend we should see it totally disabled. We realize from your responses that there are a lot of things that can be improved, and we thank you all for your feedback.
Additionally, our Dev team is working to add additional guard rails based on the issues you have brought to our attention for future uses of this tool or similar features.
Thank you for working with us to test and interact with this experiment; your feedback is invaluable and we are listening carefully.