TLDR: We will be exploring a new experiment, within Staging Ground, to understand whether AI can be helpful in evaluating question drafts and providing real-time suggestions for improvement, before questions are submitted to human reviewers. AI WILL NOT be editing users’ content.
As we mentioned in our recent Community Products Roadmap Update, Staging Ground is an area we’ve invested in to improve question success for new askers. Earlier this year we rolled out the core experience—a space where new askers get help from experienced users to improve their questions before posting to the broader community. As a result, we’ve seen question quality improve (as noted here in July, questions that go through Staging Ground have higher success and survival rates than those that don’t, which remains true today).
One challenge askers face today is the time it takes to get feedback from reviewers, and that occasionally a question may not receive feedback at all because there are not enough reviewers available. We have recently implemented several updates to increase reviewer participation (including adding more badges, as well as displaying stats) to help Staging Ground keep up with demand. We are also looking at decreasing the time it takes askers to get reviewer feedback from another angle, utilizing machine learning.
What’s being tested?
We're exploring whether an ML/AI-powered question assistant can provide first round feedback earlier, before it gets to a human reviewer, making the whole process move more efficiently. We will be conducting an A/B test in Staging Ground sometime in November, in which machine learning models will be utilized to evaluate new askers’ question drafts within the Ask Wizard experience, and an LLM will provide real-time suggestions based on the machine learning models’ outputs before they’re routed to Staging Ground. To start, we will be utilizing Google Gemini to power this experiment, but in the future we could work with other models.
To be clear, the Question Assistant will not be editing users’ content. It will be pointing out possible areas of improvement in new askers’ questions and encouraging them to make changes before submitting their question. Our intention is to provide relevant and actionable tips for users as they draft questions.
Half of new askers, within Staging Ground, will get to try the Question Assistant experiment and the other half of new askers will still experience the standard functionality as it is today. The goal of this experiment is to test whether AI can help askers get past superficial problems, so that reviewers can focus their efforts on issues that are deeper or more nuanced. All suggestions made to askers are completely optional.
How will it work?
We are currently approaching this in a two step fashion. First we trained several binary classification models to detect whether or not a question needs certain elements (what we are calling feedback indicators here). The Question Assistant will use machine learning to predict the presence or absence of the four indicators listed below. That information will then be passed to an LLM (Google Gemini for this test), along with the title and body of the question, which will then return feedback for what to improve in the question by customizing a canned response with the context it has.
So we’re not currently using an LLM to do the detection itself, although this approach may evolve based on further testing and research we continue to do.
Question Assistant Feedback Indicators:
1. Problem Definition - The problem or goal is lacking information to
understand what the user is trying to accomplish.
2. Attempt Details - The question needs additional information on what
you have tried and the relevant code.
3. Error Details - The question needs additional information on error
messages and debugging logs (as relevant).
4. Missing Minimum Viable Reproducible Example - The question needs
some code demonstrating the problem in a way that others can
reproduce.
We did some offline testing, including a labeling exercise and research study, to develop the list of indicators which are being used to train our machine learning models. The indicators we are starting with in this experiment are a subset of a broader list coming out of that testing. Other indicators which we might include in future experiments are: formatting, expected outcome, image to code, non-English, and off-topic.
See the right sidebar in the mockup below to get a sense of what the Question Assistant experience will entail. Note that the exact designs are subject to change.
We have done some trials of this process with real Staging Ground questions. For example, with a question about sorting arrays in Python that failed to provide any attempt details. Our pre-canned feedback when this indicator is identified is, "The question needs additional information on what you have tried and the relevant code." This feedback would then be passed into the LLM along with the question title and body to return specific, contextualized feedback. In this case Gemini’s output was, "To improve your question, describe the methods you've tried to sort the array, including code snippets and the results you observed. Mention any specific challenges or unexpected behavior encountered during your attempts."
For the course of this experiment, the question will still go to Staging Ground reviewers before making it onto the main site, whether or not the user has made changes based on the Question Assistant’s suggestions. If this feature is adopted and built upon, we can imagine a future in which some askers are able to improve their questions enough, by fielding tips from the Question Assistant, that they can go directly to the main site – making more space in Staging Ground for users who really need one-on-one support from human reviewers.
What's next?
If this experiment is successful we will conduct a second experiment testing some additional feedback indicators, and then consider expanding the feature to all Staging Ground eligible users on Stack Overflow. The primary metrics we will use to measure in this first experiment success are:
- Question approval rates
- Average review times
As we have stated in other recent communications, the company remains committed to testing AI/ML thoughtfully and purposefully to support the core values of Stack Overflow: human connection, collaboration, and knowledge sharing. The goal is to build and support a healthy ecosystem of active users and community contributors. This experiment will be tested in a safe, controlled, and transparent approach where humans are always in the loop. We remain open to concluding the experiment early if we find the results unfavorable for any reason.
We will report back on our findings before we initiate the follow up experiment, or consider scaling this feature to all users. What do you think of the success metrics we have laid out for this experiment? Are there others that you think we should consider?