This experiment was live from 2022-03-21 14:20 UTC until 2022-04-06 12:00 UTC.
Initial data from the test looks to be positive. We plan on doing some bigger data analysis on the results and will post it on Meta Stack Overflow when this is completed.
We will soon be turning on an experiment on Stack Overflow which will test out a new Ask Wizard. The experiment will only target users asking their first questions on the site (to be more specific, questions that would qualify for inclusion in the First questions queue).
The new wizard draws inspiration from the earlier iteration of the Ask Wizard that ran on the site in 2019, as well as the New Ask Page that succeeded the old Ask Wizard network-wide.
This new wizard is intended to serve as the first step in a new user onboarding system that we right now have in discovery (for which many more details will be shared in the coming weeks). We are releasing this new wizard as a separate test in order to be able to measure how it will affect the quality of questions from new users, as a standalone feature. Metrics that we will be looking at in judging the success of this experiment include percent close rate, average question score, and potentially the number of positive answers received and return engagement from the new user. If you have thoughts on what might be good metrics to track here, we are happy to hear them below.
How will it work?
The wizard will be split into the following steps, each of which will be accompanied by additional instructive text that will appear in the right sidebar (on a wide screen) or above the section (on a narrow screen):
Guidance and Title. We are trying to introduce the user to the process that will be taking place on the page, and give initial guidance about on-topic rules.
The text will link to the how to ask and on topic help center articles, as well as the list of Technology sites on the Stack Exchange site.
Duplicate question detection, based on the title. This will use the same duplicate detection that is used today, and will require user confirmation that there are no matches in order to proceed.
In the future, we are considering adding a second duplicate detection step at the end of the wizard that will take into account the body text and tags, but that won't make it into this stage of testing.
Describe the details of the problem.
Describe what was tried and what results were expected.
Tag Entry
The two sections of the question are merged, and presented for review
We are deliberately including a merge step here in order to replicate the review step that exists in the normal Ask experience, to instruct the users on how the different parts of the question form a whole, and to give them a chance to review the question as a whole before it is submitted.
FAQs
When will the experiment begin and how long will it run for?
It will begin sometime in the next two weeks, and should run for a period up to one month. We will update this post when the status of the experiment changes and plan on sharing results after the experiment has concluded.
On what sites will the experiment be run?
Only on Stack Overflow.
I notice that the wizard is using the new Stacks Editor. Why is this being used here, since it is not being used anywhere else on Stack Overflow yet? Does this mean that the Stacks Editor is going to be released soon on Stack Overflow?
We are deliberately using the new editor here, even though it is not yet used on Stack Overflow, and is not yet used for question-asking anywhere.
The reasons why we are using it:
- We are splitting the question up over multiple fields. The old editor with the preview beneath it will not work in this context (two active editors on the screen), and we feel that the inclusion of a preview is very important for new users.
- It is being shown only to brand new users. So we are not as concerned with confusion between the editor and the old one (if anything, the new editor is much closer to what many users are used to from elsewhere on the Internet).
- We are planning on moving forward very soon with fixing outstanding issues with the Stacks Editor, with the goal of getting it ready for wider use. Using it here will be a new area for testing, and additional impetus to make sure we fix things. That said, this does not mean that we are as of yet committed to opening up the Stacks Editor for wider use on Stack Overflow or other sites on the network.
- This initial release is a Minimum Viable Product (MVP)/test. It is not replacing the existing ask-question function. So we are more open to testing things in this context as well.
I am an experienced user. How will I be able to try it out (and give feedback)?
Once the experiment is live, instructions will be posted that will allow any user to access and use the live wizard (without having your interactions included in test analysis). We are especially interested in hearing from experienced users about their experiences with the tool (while trying to experience the new tool from the viewpoint of a new user), since the main target audience (brand new users on the site) are not very likely to give feedback through traditional channels.
I am a new user who was included in the experiment for my first question on Stack Overflow. Is there a way that I can opt out?
There will be no way for users sampled into the experiment to opt out. However, they will only see the new wizard while they are on the first question (or questions if the first one or two don’t do well).
Will this be offered on other sites in the future?
If the experiment is successful and the feature graduates, we hope to be able to adapt this to be an option for first-time askers on other sites. At that time we will try to determine what adaptations and customizations will be needed to allow the instructions to be relevant across different sites on the network.
We will be happy to address below any questions, comments, or concerns that you may have.