June 7th 2023 -- The experiment has now concluded, and the title drafting assistant is no longer enabled on Stack Overflow. We will update you soon with the results of the experiment. Thanks all for your feedback along the way.
This experiment is now live, and testing is under way. Also, Philippe (the VP-Community at Stack Exchange) has created a post on Meta Stack Exchange about the launch.
Beginning the week of May 15th, our developer team plans to experiment with an assistive feature to help users generate more descriptive, accurate titles for their questions. This experiment builds on the proposal first introduced on Meta Stack Exchange, where we shared some early draft screenshots of what such a feature could look like, and how it might perform. This experiment will run at most three weeks, but we will likely have all the data we need by the end of the second week. (We may do more rounds of experimentation later, if needed.)
This experiment is designed to provide critical insights into whether such a feature could improve the day-to-day experience of askers, answerers, and searchers using the site, and if so, what forms it should take. (We will also be taking the opportunity to test a couple small layout changes for the Ask Wizard page, including relocating the “Title” input box to a step after a user writes the question body text.) Once this initial experiment concludes, we’ll be opening up a data-informed Meta discussion about what such a feature should look like, and how such a feature can function on the platform in its final implementation.
I know the introduction of these features is a sensitive topic, so I want to give you a bit of a peek behind the curtain at what we’re actually going to test, what we’re hoping to learn from it, and how many users we’ll be running this test with. I also want you to know that we are not rushing this forward into implementation: the design is certainly not set in stone. Community comments will be carefully considered, and your feedback is more than welcome.
How we’re expecting this test to look on the platform
We’re going to be testing four different layouts for the Ask Wizard. All experiments will run on the Ask Wizard page only at this time.
- Ask page and review page: No change from the current state.
- Ask page and review page: We’ll move the ‘title’ field to the question review step, and ask users to write one after they have written their question body and tags. We won’t offer the title assistant in this variant.
- Ask page and review page: We’ll offer suggestions for titles in this variant, during the review step.
- Ask page and review page: We’ll move the ‘title’ field to the question review step, and we’ll also offer title suggestions at this step.
Our plans for this experiment
For each layout we are testing, we will collect data for around 9000 questions. We’re principally interested in assessing five core questions:
- Do users’ questions perform better when they’ve received title-drafting assistance?
- Are titles written with assistance edited more or less often by community members?
- Do readers and answerers interact differently with questions that received title-drafting assistance? For example, via reviews or comments.
- Do better titles lead to a reduction in users – particularly new users – abandoning their questions? Does it increase the rate at which new users come back to the site in the future?
- How many users will accept the titles that are recommended to them?
While we have a couple of experiments we plan to start with, we’re going to be revising this plan as we go, based on information we learn from prior results. This means that we don’t have a pre-prepared list of the exact statistical tests we are going to run, nor the full list of A/B tests ready at the start of this experiment. On the flipside, this means that if you think there is something specific you feel strongly we should test as part of this experiment, please let us know in the answers below, and we may be able to work it into our plans.
Feedback - what we’re looking for, and where to post it
We’re interested in a wide range of feedback on this experiment. However, we’d ask that you hold on to feedback about the appearance and question workflow integration for the time being – we’ll come back to this when we’re ready to begin final design and implementation, once the results of this experiment are in. Lastly, we’ve also already gotten some feedback on Philippe’s Meta Stack Exchange post about this proposal, which I’d encourage you to look through and read, as other users’ thoughts may inform the feedback you’d like to provide here. (Don’t worry – we’ve read the answers there, too.)
For this post...:
- Conceptually, are we asking all the questions we need to measure success during the experiment? Are there more facets of site participation that we need to pay attention to as a part of this test?
- Given how the feature is going to look and how we believe that people will use it, are there reasons you would be skeptical of the results of the experiment?
- Supposing the feature works as designed, do you see any subtle or hidden downsides to implementing it on the platform?
- Suppose the feature is successful in its stated aims. What concerns would you have with its implementation?
And if you’ve got other questions, the answers are, of course, open to you as well. Finally, I hope it’s clear by now that this is a genuine request for community input, coming to you early enough to change plans based upon the feedback, and we have a history of modifying experiments or plans based on the outcome of consultations like this one. All that to say this: your voice is important, and we would love to hear what you think. Members of the product team and the community team will be actively monitoring this question for your ideas for the next two weeks. (After that I’ll still be reading, so no worries if you’ve gotta slip some critical feedback in late.)
AI
anywhere in this description, right?