We’d like to share some results from our experimentation with two earlier releases: the first, a tool to generate titles for your post, and the second, a tool to help users format their questions’ bodies. We’re sharing both sets of results at the same time because we think the insights they offer are similar to each other, and taken together they can better communicate the way we’re thinking about proceeding from here.
Not to bury the lede: We won’t be proceeding to release these features on the platform in their current iteration. By the numbers, these features did seem to improve the experience of the people who used them; however, they didn’t present a sufficient benefit to on-site outcomes to justify releasing them in their current forms. The content formatting assistant also experienced some issues during its launch, such as hallucinations, and inadvertently fixing the bugs with users’ code while editing their questions, which were not desired behaviors. We may still look to test incorporating features like these into the site at a later point since the users who used these tools did appreciate them. However, before we do so, we’ll have to take more time to develop and test appropriate safeguards, and flesh out exactly how we hope users will use these tools on the platform.
With that said, let’s start with the title-drafting assistant’s results. We tested four variants over the course of this experiment:
- The ask page as it currently is, with no changes.
- The ask page as it currently is, except we move the ‘title’ field to the bottom of the question page, after a user writes their question body.
- The ask page with title-drafting assistance.
- The ask page with title-drafting assistance, except the ‘title’ field is moved to a step after the question body is written, similar to variant #2.
For more information about the variants, you can check out the original post on the topic.
We measured a few key values from this experiment, which I’ll summarize briefly.
- The ‘question completion rate’ represents the percent of the time a user begins to write a question, and then actually posts one. While it’s hard to tell what the maximum reasonable value here is, we hoped that the question completion rate would see a slight boost, or at least be neutrally affected, by AI assistance.
- The ‘time to question completion’ represents the amount of time a user spends drafting a question before posting it. It takes users a considerable amount of time to write a question on Stack Overflow – an average of around 11 minutes. (The distribution is long-tailed, with some users taking significant amounts of time to write questions, but the vast majority of questions come in close to 11 minutes.)
- The ‘question success rate’ represents the percent of the time a question receives one of the following: 2+ question score, 2+ total answer score, an accepted answer, or at least two non-deleted answers. While there are other ways to define question success, we generally accept this definition as representing the improvement or degradation in the question asking experience on the site.
- Edits to the question title – this one’s fairly straightforward.
- The ‘satisfaction rate’ is the percentage of people who gave positive feedback for the suggested title on the ask page. (Users had the option of giving ‘thumbs up’ or ‘thumbs down’ feedback.)
Without further ado…
Baseline (#1) | Variant #2 | Variant #3 | Variant #4 | |
---|---|---|---|---|
Question completion rate | 48% | 47% | 46% | 48% |
Time to complete question | 10 mins 36 secs | 10 mins 30 secs | 10 mins 12 secs | 10 mins 36 secs |
Question success rate | 22% | 23% | 23% | 23% |
Edits to the question title | 10% | 10% | 8% | 7% |
Satisfaction rate | n/a | n/a | 88% | 94% |
So, here’s our subjective evaluation. Moving the title to the bottom of the question page by itself doesn’t seem to have an impact on the questions’ success rates. However, generating AI-assisted titles one step after the user writes their question body is better than leaving the title in the position it’s currently in. Otherwise, we do see a small reduction in the amount of time it takes users to write their questions, but it’s not by an overwhelming amount. Question success has an analogous story: title-drafting assistance didn’t seem to harm the user experience surrounding question asking, but neither was the improvement significant enough for us to pursue further as-is in this iteration.
We ran essentially the same measures for the content formatting experiment. However, the test variants were naturally different. We tested both on the Ask Wizard, and on our normal ask page (“AskV2”).
- The Ask Wizard page as it currently is, with no changes.
- The Ask Wizard page with AI support, and with the question review step removed.
- The Ask Wizard page without AI support, but with the question review step removed.
- The AskV2 page as it currently is, with no changes.
- The AskV2 page with AI draft-editing assistance.
Before you read too much into this table, please keep in mind – due to issues with the experiment, it was only on for a short time. Normally we’d want to leave an experiment like this running for at least a week or two, in order to capture a better understanding of what normal user activity should look like. So I’d recommend tempering expectations a bit for how strongly these results can be interpreted.
Ask Wizard - Baseline | Ask Wizard Var. 2 | Ask Wizard Var. 3 | AskV2 - Baseline | AskV2 - AI | |
---|---|---|---|---|---|
Question completion rate | 46% | 51% | 49% | 70% | 69% |
Question success rate | 15% | 16% | 18% | 28% | 29% |
Edits to the question body | 46% | 36% | 46% | 30% | 36% |
AI suggestion acceptance | 83% | 83% | 69% |
There was a noticeable improvement in question success rate for newer users who were using the Ask Wizard, though not an overwhelming one. For users who are more experienced, we saw no meaningful change in question completion rate. Newer users also saw the most benefit from the AI content formatter, though again the margin here is very small. The most significant impact seems to be on the question body edit rate, which - for new users only - seems to have fallen around 10%. Though, the degree to which this is due to user engagement differences on these questions, or a genuine reduction in need for edits, was not directly tested.
Overall, newer users liked the content formatter’s suggestions significantly more than established ones. This isn’t too surprising, as new users are often the ones who need the most help massaging their questions to form.
So, where are we going from here? Well, both experiments have concluded, and we’re not planning to release these as features to the site as-is.
On the positive side, the results we see here show that these features are at least safe for us to play around with, which is a good baseline result – we don’t expect further experimentation with these features to have immediate, catastrophic effects. This means that there’s room for us to play around with design, implementation, and workflows without risking severe effects in production. (Yes, it is clear to us that if we want to publish final features, we need better safeguards to proceed.)
On the downside, these results aren’t as good as we were hoping for. While there does seem to be a reduction in curator workload as a result of these features, we were hoping that they would have more of an impact on the questions’ overall success rates, and therefore give a greater proportion of the users who come to our sites a good experience. While folks did overwhelmingly appreciate having AI suggestions on hand (even experienced askers liked it on the balance), this alone doesn’t necessarily make the overall user experience better if it doesn’t lead to better on-site outcomes.
The net result is that we’re going to shelve the public release of these features for a bit while we work on better proposals for how to integrate it into user workflows. And, if we believe our ideas are promising, then as usual, we’ll come back here to share more details and ask for your feedback.