Three months ago we ran an experiment testing a new Ask Wizard on Stack Overflow. The Ask Wizard is planned as the first phase of the Staging Ground project (see here and here for more info), and we wanted to test it first to be able to identify the impact on our goal metrics of the Ask Wizard alone, before having the Staging Ground involved.
I am happy to report that the test was a success, meeting or exceeding many of its goals. Details on the different metrics and results are below, along with next steps.
To review, our original plan for metrics to track was:
Metrics that we will be looking at in judging the success of this experiment include percent close rate, average question score, and potentially the number of positive answers received and return engagement from the new user.
It also shares some of the primary goals of the Staging Ground:
Address quality issues that exist with first questions by lowering the close/deletion rates and improve their overall quality
Take the pressure off of the First questions queue
On to the results…
Completion of asking process
An initial worry of ours with the Ask Wizard was that introducing additional steps to what is currently a one-step process, with more validation and more instructions, would result in an increase in the number of users who load the question asking page but do not ultimately submit a question. In theory, we would have some tolerance for an increase in users not completing the process of asking a question if it resulted in achieving our other goals. And due to our improving the accuracy of duplicate detection (by including tags in the duplicate search), it could also be reasonable to see a legitimate dropoff in question completion due to a higher percentage of users finding a duplicate question and abandoning the process for that reason.
Here is what happened (in this and other tables below, users in the Baseline Test group used the current ask page, and users in the Ask Wizard Test group used the new Ask Questions Wizard tool):
Test group | Ask question clicks | Question asked | Completion % |
---|---|---|---|
Baseline | 35,967 | 18,243 | 50.72% |
Ask Wizard | 35,801 | 18,516 | 51.72% |
So we actually saw a statistically significant increase in the completion percentage from users who were in the Ask Wizard test. This makes it clear that our initial concern of “more steps = more confusing” is not something that we need to be worried about. It may even be that making the process of asking a question clearer, and adding more instructions, is more encouraging for users to complete the process.
(Note: The numbers listed above relate to all users who were shown the new workflow, including users who qualify for the First questions review queue for their second or third questions. In contrast, the data below is only for actual first questions. This was done both to help make the querying easier, and in order to have all questions being evaluated below starting from the same place (first question for the user). So there is a discrepancy between the numbers, but this is known and ok.)
Deletions
There was a tremendous reduction in the number of posts deleted within three days of creation for the test group using the new Ask Wizard versus the baseline group:
Test group | Questions asked | # Deleted | % Deleted |
---|---|---|---|
Baseline | 13,854 | 2,185 | 15.77% |
Ask Wizard | 13,448 | 1,880 | 13.98% |
First-time askers going through the Ask Wizard experienced a 11.36% reduction in deletions (a statistically significant result).
Close percentage and reasons
We saw a similar (though of lesser magnitude) reduction in close rates of questions within three days of creation:
Test group | # Non-deleted | # Remaining open | % Closed |
---|---|---|---|
Baseline | 11,669 | 10,662 | 8.63% |
Ask Wizard | 11,568 | 10,617 | 8.22% |
First-time askers going through the Ask Wizard experienced a 4.74% reduction in the closure of non-deleted questions. This has a p-value of 0.13 and is therefore not statistically significant, but it is still trending in the right direction.
Our data team performed a comparison of the different close reasons that were used for both test groups. This led to some very interesting results:
If we put the data into a table, it looks like this:
Close Reason | Baseline # (%) | Ask Wizard # (%) |
---|---|---|
Duplicate | 767 (22.4%) | 662 (22.4%) |
Needs details or clarity | 994 (29.0%) | 803 (27.1%) |
Needs more focus | 526 (15.4%) | 428 (14.4%) |
Off-topic | 1,066 (31.1%) | 990 (33.4%) |
Opinion-based | 72 (2.1%) | 78 (2.7%) |
The stats that stand out the most here are:
Questions that were asked through the Ask Wizard and subsequently closed had a significant drop in the percent of questions closed for two close reasons associated with poor quality (needs details or clarity/needs more focus), going from 44.4% of closures in the Baseline group to 41.5% (a drop of around 6.5%).
Many of these closures went over to the off-topic/opinion-based grouping of closed questions, which went from 33.2% up to 36.1% (an increase of 8.7%). However, even though these went up as a percentage of closed questions, they still dropped as a percentage of total questions, from 8.2% to 7.9%.
These results indicate that the Ask Wizard has helped to improve the quality of on-topic questions, and opens up an opportunity to iterate more on trying to prevent off-topic and subjective questions from being asked (more on this below).
We are also seeing no drop in the rate of duplicate closures. This is not alarming, but it is a little disappointing, considering that we did improve the accuracy of duplicate post detection in the Ask Wizard.
Impact on Review Queue actions
One benefit that we noticed is the impact on review queues, specifically the number of actions each unique reviewer is handling in each queue. The actions per reviewer metric is a good indication of workload per reviewer, per queue.
There were four queues that stood out to us: Close Votes, Reopen Vote, Suggested Edit, and Triage. In all four queues, Ask Wizard posts saw a decrease in the number of actions per reviewer.
In the table below:
- UR = # of Unique Reviewers
- A/R = # of Actions per Reviewer
Baseline UR | Ask Wizard UR | Baseline A/R | Ask Wizard A/R | |
---|---|---|---|---|
Close Votes | 273 | 169 | 11.20 | 5.02 |
Reopen | 693 | 312 | 3.35 | 2.18 |
Suggested Edits | 1,682 | 990 | 6.40 | 2.75 |
Triage | 1,019 | 732 | 8.68 | 5.15 |
The first two queues associated with votes could be capturing some noise. Vote reviews are not always associated with a post’s quality. The last two queues, however, indicate that Ask Wizard posts are coming out of the gate requiring less suggested edits or triage intervention.
This lines up with what the Ask Wizard is supposed to do: help facilitate better questions getting asked. A better question will, ideally, have less suggested edits to deal with because it is more likely to be complete the first time. And any triage intervention would also be less likely as a result. We saw evidence of both happening with the decrease in reviewer workload.
Question success & user follow-up
We looked into whether questions that went through the Ask Wizard were more likely to be successful on the site, using a number of different measures.
In the table below (measuring states reached within 30 days of question creation):
Open = % of questions not closed or deleted
Answer = % of questions that received at least one non-deleted answer
Accepted = % of questions that have a non-deleted accepted answer
Q-Scored = % of questions had a score of at least 2
A-Scored = % of questions where the sum of scores on all deleted answers is at least 2
Q/A Scored = % of questions that have a score of at least 2 or where sum of the scores on all non-deleted answers is at least 2 (the composite of the previous two)
Test group | Open | Open + Answered | Accepted | Q-Scored | A-Scored | Q/A Scored |
---|---|---|---|---|---|---|
Baseline | 72.27% | 36.66% | 13.46% | 2.71% | 4.56% | 6.15% |
Ask Wizard | 74.54% | 37.36% | 13.78% | 2.88% | 5.00% | 6.61% |
We see consistent — albeit small — improvements in each of these different categories that relate to question success for Ask Wizard versus the baseline. While the magnitude of these improvements is rather small in scale, and while some of these on their own may have questionable statistical significance, taken together, we can see from here that the Ask Wizard is definitely helping to nudge question quality in the right direction.
User Follow-Up
Another question we sought to answer: Were users who went through the Ask Wizard more likely to return to the site? The thinking here is that a more positive experience asking a first question on the site would translate to a higher likelihood of returning to the site in the future (including just visiting the site without doing anything else).
We examined the data in a number of ways, looking at different metrics for the four weeks following publishing a first question:
Mean/median number of days visiting the site: no difference between test groups
Number of times logging into the site: no difference between test groups
Number of users posting additional questions:
20.1% of Ask Wizard users and 21.3% of Baseline users went on to post an additional question.
This 5.9% outperformance by the baseline group is a statistically significant result.
Number of users posting an answer to any question:
10.8% of Ask Wizard Users and 11.4% of Baseline users go on to answer a question.
Baseline group outperforms by 5.5%, but this is not a statistically significant result, with a p-value of 0.11.
There are less users answering overall, making this difference less drastic on raw answer scale.
We can only speculate at this point as to the reasons why more users from the Baseline group went on to ask questions. One theory is that a higher question close/deletion rate would lead to more users asking the same question over in a different way. However, we did not have sufficient time allocated to this in order to perform a deep dive in this area.
While the Ask Wizard did not deliver on these metrics, this was more of an aspirational goal (and one where the metric has a much more tenuous link to the Ask Wizard experience than the metrics above).
Summary and Next Steps
We are very happy with the results of the Ask Wizard test that was run on Stack Overflow. It met or exceeded just about all of our expectations when it came to the goals of the component.
Additional areas to test
We have only concluded the initial test of the Ask Wizard, and realize that there are still additional improvements that can be made. So we are definitely planning on testing out different options on the wizard to see which areas should be further updated.
Some areas that we are thinking of for more testing:
Seeing if we can catch more questions up-front that would eventually be closed as duplicate, off-topic, or subjective. One approach that we might try is an upfront “honeypot question” designed to let users identify early on if their question is off-topic, subjective, or belongs on a different site.
There is obviously much more work that can be done in the area of better duplicate post detection and suggestion.
Additional tweaks to section names and instructions.
If you have any additional ideas for things to try out on further testing iterations, please let us know in an answer below.
Feature graduation & FAQs
While the Ask Wizard itself was planned as the first stage of the Staging Ground project, based on these results, we don't see any reason not to go ahead with planning graduation of the new Ask Wizard as the default question-asking tool on Stack Overflow. Some questions/answers related to this:
Who will be using the new Ask Wizard once it graduates?
Based on the success of the tool, we are planning on making it the default for all question askers on Stack Overflow, including users who have already asked in the past.
Will users still have the option to use the current (non-wizard) question-asking page?
No decision has been made on that. We could potentially let users toggle between the two, and set one or the other as their default. Or we could default all new askers to the wizard, and keep everyone who has already asked a successful question defaulted to the current asking page. Or some other variation along these lines. We’re interested in your thoughts on this; feel free to share them in an answer below.
When will this happen?
We are targeting the end of June. The team working on the new Stacks Editor (which features prominently in the new Ask Wizard) has been quietly plowing through many of the issues that were already reported, and are pushing right now for their Beta 1 release, in which the goal is to achieve feature parity with the legacy editor. We are going to wait for that release of the Stacks Editor before graduating the new Ask Wizard.
What about the Stacks Editor?
We know that there was a tremendously long time between the initial test of the editor, and resumption on work for many of the originally-reported issues. The goal still is to continue to resolve reported issues and to improve usability of the editor. That work has (without public fanfare) picked up in the past few months (the team just hit and released their alpha 2 milestone) and should continue moving forward. As such, for reasons discussed in the original post announcing the Ask Wizard test, and because we did not see any negative effect from having the Stacks Editor in the Ask Wizard test, there are no plans to remove the Stacks Editor from the Ask Wizard. Please continue to submit bugs and feature requests here on Meta using the stacks-editor tag, or use the GitHub project for the editor to check out the issues still scheduled for work and to submit new issues. Update: read about the Beta release (in early July) for the Stacks Editor, along with plans for further development and testing.
Will the Ask Wizard be available to other sites on the Stack Exchange network?
We are hoping to make the Ask Wizard available as an option to other sites, and plan on presenting details about this in a future post on Meta Stack Exchange. That post will present the new tool to users who haven't yet heard about it, and will also open a discussion about what (general) default instructions we should include in the wizard (that will be more friendly on all sites than the Stack Overflow-oriented language currently used), and what areas we might be able to make customizable on a per-site basis. We will also discuss whether the new wizard should be defaulted to on or off.
Will the new Ask Wizard be available for testing before the official feature graduation?
It is not available now, but we hope to make this an option soon (using a querystring parameter to indicate editor preference, like we did during the test). We will update this post if/when this is possible.
How should I make a bug report or feature request relating to the Ask Wizard?
These can be made on MSO or MSE (once it is made available across the network), tagged with the ask-question-wizard tag.
Please share your feedback and questions below as answers (not as comments). Some of the areas that we highlighted above for feedback include: ideas for future test iterations and feedback on feature graduation options.