32

Three months ago we ran an experiment testing a new Ask Wizard on Stack Overflow. The Ask Wizard is planned as the first phase of the Staging Ground project (see here and here for more info), and we wanted to test it first to be able to identify the impact on our goal metrics of the Ask Wizard alone, before having the Staging Ground involved.

I am happy to report that the test was a success, meeting or exceeding many of its goals. Details on the different metrics and results are below, along with next steps.

To review, our original plan for metrics to track was:

Metrics that we will be looking at in judging the success of this experiment include percent close rate, average question score, and potentially the number of positive answers received and return engagement from the new user.

It also shares some of the primary goals of the Staging Ground:

  • Address quality issues that exist with first questions by lowering the close/deletion rates and improve their overall quality

  • Take the pressure off of the First questions queue

On to the results…

Completion of asking process

An initial worry of ours with the Ask Wizard was that introducing additional steps to what is currently a one-step process, with more validation and more instructions, would result in an increase in the number of users who load the question asking page but do not ultimately submit a question. In theory, we would have some tolerance for an increase in users not completing the process of asking a question if it resulted in achieving our other goals. And due to our improving the accuracy of duplicate detection (by including tags in the duplicate search), it could also be reasonable to see a legitimate dropoff in question completion due to a higher percentage of users finding a duplicate question and abandoning the process for that reason.

Here is what happened (in this and other tables below, users in the Baseline Test group used the current ask page, and users in the Ask Wizard Test group used the new Ask Questions Wizard tool):

Test group Ask question clicks Question asked Completion %
Baseline 35,967 18,243 50.72%
Ask Wizard 35,801 18,516 51.72%

So we actually saw a statistically significant increase in the completion percentage from users who were in the Ask Wizard test. This makes it clear that our initial concern of “more steps = more confusing” is not something that we need to be worried about. It may even be that making the process of asking a question clearer, and adding more instructions, is more encouraging for users to complete the process.

(Note: The numbers listed above relate to all users who were shown the new workflow, including users who qualify for the First questions review queue for their second or third questions. In contrast, the data below is only for actual first questions. This was done both to help make the querying easier, and in order to have all questions being evaluated below starting from the same place (first question for the user). So there is a discrepancy between the numbers, but this is known and ok.)

Deletions

There was a tremendous reduction in the number of posts deleted within three days of creation for the test group using the new Ask Wizard versus the baseline group:

Test group Questions asked # Deleted % Deleted
Baseline 13,854 2,185 15.77%
Ask Wizard 13,448 1,880 13.98%

First-time askers going through the Ask Wizard experienced a 11.36% reduction in deletions (a statistically significant result).

Close percentage and reasons

We saw a similar (though of lesser magnitude) reduction in close rates of questions within three days of creation:

Test group # Non-deleted # Remaining open % Closed
Baseline 11,669 10,662 8.63%
Ask Wizard 11,568 10,617 8.22%

First-time askers going through the Ask Wizard experienced a 4.74% reduction in the closure of non-deleted questions. This has a p-value of 0.13 and is therefore not statistically significant, but it is still trending in the right direction.

Our data team performed a comparison of the different close reasons that were used for both test groups. This led to some very interesting results:

Chart showing the data in the the table that immediately follows, giving the total number of posts closed broken down by reason, for both Baseline and Ask Wizard test groups

If we put the data into a table, it looks like this:

Close Reason Baseline # (%) Ask Wizard # (%)
Duplicate 767 (22.4%) 662 (22.4%)
Needs details or clarity 994 (29.0%) 803 (27.1%)
Needs more focus 526 (15.4%) 428 (14.4%)
Off-topic 1,066 (31.1%) 990 (33.4%)
Opinion-based 72 (2.1%) 78 (2.7%)

The stats that stand out the most here are:

  • Questions that were asked through the Ask Wizard and subsequently closed had a significant drop in the percent of questions closed for two close reasons associated with poor quality (needs details or clarity/needs more focus), going from 44.4% of closures in the Baseline group to 41.5% (a drop of around 6.5%).

  • Many of these closures went over to the off-topic/opinion-based grouping of closed questions, which went from 33.2% up to 36.1% (an increase of 8.7%). However, even though these went up as a percentage of closed questions, they still dropped as a percentage of total questions, from 8.2% to 7.9%.

These results indicate that the Ask Wizard has helped to improve the quality of on-topic questions, and opens up an opportunity to iterate more on trying to prevent off-topic and subjective questions from being asked (more on this below).

We are also seeing no drop in the rate of duplicate closures. This is not alarming, but it is a little disappointing, considering that we did improve the accuracy of duplicate post detection in the Ask Wizard.

Impact on Review Queue actions

One benefit that we noticed is the impact on review queues, specifically the number of actions each unique reviewer is handling in each queue. The actions per reviewer metric is a good indication of workload per reviewer, per queue.

There were four queues that stood out to us: Close Votes, Reopen Vote, Suggested Edit, and Triage. In all four queues, Ask Wizard posts saw a decrease in the number of actions per reviewer.

In the table below:

  • UR = # of Unique Reviewers
  • A/R = # of Actions per Reviewer
Baseline UR Ask Wizard UR Baseline A/R Ask Wizard A/R
Close Votes 273 169 11.20 5.02
Reopen 693 312 3.35 2.18
Suggested Edits 1,682 990 6.40 2.75
Triage 1,019 732 8.68 5.15

The first two queues associated with votes could be capturing some noise. Vote reviews are not always associated with a post’s quality. The last two queues, however, indicate that Ask Wizard posts are coming out of the gate requiring less suggested edits or triage intervention.

This lines up with what the Ask Wizard is supposed to do: help facilitate better questions getting asked. A better question will, ideally, have less suggested edits to deal with because it is more likely to be complete the first time. And any triage intervention would also be less likely as a result. We saw evidence of both happening with the decrease in reviewer workload.

Question success & user follow-up

We looked into whether questions that went through the Ask Wizard were more likely to be successful on the site, using a number of different measures.

In the table below (measuring states reached within 30 days of question creation):

  • Open = % of questions not closed or deleted

  • Answer = % of questions that received at least one non-deleted answer

  • Accepted = % of questions that have a non-deleted accepted answer

  • Q-Scored = % of questions had a score of at least 2

  • A-Scored = % of questions where the sum of scores on all deleted answers is at least 2

  • Q/A Scored = % of questions that have a score of at least 2 or where sum of the scores on all non-deleted answers is at least 2 (the composite of the previous two)

Test group Open Open + Answered Accepted Q-Scored A-Scored Q/A Scored
Baseline 72.27% 36.66% 13.46% 2.71% 4.56% 6.15%
Ask Wizard 74.54% 37.36% 13.78% 2.88% 5.00% 6.61%

We see consistent — albeit small — improvements in each of these different categories that relate to question success for Ask Wizard versus the baseline. While the magnitude of these improvements is rather small in scale, and while some of these on their own may have questionable statistical significance, taken together, we can see from here that the Ask Wizard is definitely helping to nudge question quality in the right direction.

User Follow-Up

Another question we sought to answer: Were users who went through the Ask Wizard more likely to return to the site? The thinking here is that a more positive experience asking a first question on the site would translate to a higher likelihood of returning to the site in the future (including just visiting the site without doing anything else).

We examined the data in a number of ways, looking at different metrics for the four weeks following publishing a first question:

  • Mean/median number of days visiting the site: no difference between test groups

  • Number of times logging into the site: no difference between test groups

  • Number of users posting additional questions:

    • 20.1% of Ask Wizard users and 21.3% of Baseline users went on to post an additional question.

    • This 5.9% outperformance by the baseline group is a statistically significant result.

  • Number of users posting an answer to any question:

    • 10.8% of Ask Wizard Users and 11.4% of Baseline users go on to answer a question.

    • Baseline group outperforms by 5.5%, but this is not a statistically significant result, with a p-value of 0.11.

    • There are less users answering overall, making this difference less drastic on raw answer scale.

We can only speculate at this point as to the reasons why more users from the Baseline group went on to ask questions. One theory is that a higher question close/deletion rate would lead to more users asking the same question over in a different way. However, we did not have sufficient time allocated to this in order to perform a deep dive in this area.

While the Ask Wizard did not deliver on these metrics, this was more of an aspirational goal (and one where the metric has a much more tenuous link to the Ask Wizard experience than the metrics above).

Summary and Next Steps

We are very happy with the results of the Ask Wizard test that was run on Stack Overflow. It met or exceeded just about all of our expectations when it came to the goals of the component.

Additional areas to test

We have only concluded the initial test of the Ask Wizard, and realize that there are still additional improvements that can be made. So we are definitely planning on testing out different options on the wizard to see which areas should be further updated.

Some areas that we are thinking of for more testing:

  • Seeing if we can catch more questions up-front that would eventually be closed as duplicate, off-topic, or subjective. One approach that we might try is an upfront “honeypot question” designed to let users identify early on if their question is off-topic, subjective, or belongs on a different site.

  • There is obviously much more work that can be done in the area of better duplicate post detection and suggestion.

  • Additional tweaks to section names and instructions.

If you have any additional ideas for things to try out on further testing iterations, please let us know in an answer below.

Feature graduation & FAQs

While the Ask Wizard itself was planned as the first stage of the Staging Ground project, based on these results, we don't see any reason not to go ahead with planning graduation of the new Ask Wizard as the default question-asking tool on Stack Overflow. Some questions/answers related to this:

Who will be using the new Ask Wizard once it graduates?

Based on the success of the tool, we are planning on making it the default for all question askers on Stack Overflow, including users who have already asked in the past.

Will users still have the option to use the current (non-wizard) question-asking page?

No decision has been made on that. We could potentially let users toggle between the two, and set one or the other as their default. Or we could default all new askers to the wizard, and keep everyone who has already asked a successful question defaulted to the current asking page. Or some other variation along these lines. We’re interested in your thoughts on this; feel free to share them in an answer below.

When will this happen?

We are targeting the end of June. The team working on the new Stacks Editor (which features prominently in the new Ask Wizard) has been quietly plowing through many of the issues that were already reported, and are pushing right now for their Beta 1 release, in which the goal is to achieve feature parity with the legacy editor. We are going to wait for that release of the Stacks Editor before graduating the new Ask Wizard.

What about the Stacks Editor?

We know that there was a tremendously long time between the initial test of the editor, and resumption on work for many of the originally-reported issues. The goal still is to continue to resolve reported issues and to improve usability of the editor. That work has (without public fanfare) picked up in the past few months (the team just hit and released their alpha 2 milestone) and should continue moving forward. As such, for reasons discussed in the original post announcing the Ask Wizard test, and because we did not see any negative effect from having the Stacks Editor in the Ask Wizard test, there are no plans to remove the Stacks Editor from the Ask Wizard. Please continue to submit bugs and feature requests here on Meta using the tag, or use the GitHub project for the editor to check out the issues still scheduled for work and to submit new issues. Update: read about the Beta release (in early July) for the Stacks Editor, along with plans for further development and testing.

Will the Ask Wizard be available to other sites on the Stack Exchange network?

We are hoping to make the Ask Wizard available as an option to other sites, and plan on presenting details about this in a future post on Meta Stack Exchange. That post will present the new tool to users who haven't yet heard about it, and will also open a discussion about what (general) default instructions we should include in the wizard (that will be more friendly on all sites than the Stack Overflow-oriented language currently used), and what areas we might be able to make customizable on a per-site basis. We will also discuss whether the new wizard should be defaulted to on or off.

Will the new Ask Wizard be available for testing before the official feature graduation?

It is not available now, but we hope to make this an option soon (using a querystring parameter to indicate editor preference, like we did during the test). We will update this post if/when this is possible.

How should I make a bug report or feature request relating to the Ask Wizard?

These can be made on MSO or MSE (once it is made available across the network), tagged with the tag.


Please share your feedback and questions below as answers (not as comments). Some of the areas that we highlighted above for feedback include: ideas for future test iterations and feedback on feature graduation options.

17
  • 77
    "So we actually saw a statistically significant increase" - citation needed. Quoting the dictionary definition for statistical significance: "the extent to which a result deviates from that expected to arise simply from random variation or errors in sampling." - 1% @35k is by no means statistically significant. We have plenty of natural variation over time well exceeding 1% in both directions Commented Jun 2, 2022 at 15:19
  • 40
    "we did not see any negative effect from having the Stacks Editor in the Ask Wizard test" I disagree. It led to some absolutely horrific questions. Disable the Stacks editor in the new Ask Wizard
    – Dharman Mod
    Commented Jun 2, 2022 at 15:45
  • 18
    @ZoestandswithUkraine In this case it was close to a 2% improvement if you look at the final conversion rate versus the baseline. In this case, the p-value is 0.038, which is less than the 0.05 value we generally look for when judging statistical significance. So I must disagree with your conclusion.
    – Yaakov Ellis StaffMod
    Commented Jun 2, 2022 at 15:53
  • 23
    The fact that the barely functional stacks editor is out there being used and forced on newcomers when it is, in your own words, in an alpha phase is frankly awful. Commented Jun 2, 2022 at 17:16
  • 11
    I like the research, a very comprehensive breakdown that allows users to make an informed opinion. Provisional conclusion: Ask wizard is doing better in all stats.
    – bad_coder
    Commented Jun 2, 2022 at 17:17
  • 5
    @NickstandswithUkraine the editor is not being forced on anyone at the moment. As mentioned above, we are planning to graduate this only after the editor has had its beta 1 release, which will include full parity with existing editor features, plus many more bug fixes. And then there will continue to be releases after that addressing other reported issues. I contest your assessment that the editor is barely functional. If that were so, then presumably it would have had quite a negative effect on the test being reported above.
    – Yaakov Ellis StaffMod
    Commented Jun 2, 2022 at 17:37
  • 5
    @ayhan if you read the bottom of the post, I put forth a number of possibilities for how this will work. We are not planning on forcing it on experienced users - at a minimum, a toggle will be available, and it is quite likely that we would only default to the ask wizard being on for new users. If you would like to give your opinions and thoughts on the manner in a constructive way, please do so in an answer below.
    – Yaakov Ellis StaffMod
    Commented Jun 2, 2022 at 17:38
  • 8
    The summary seems to be that it's a small to medium but consistent improvement. There might still be somewhat of a novelty effect, i.e. the positive effects might shrink over time when people learn how to game the system (in a negative way). I'm a bit disappointed by no visible effects on duplicate closure. The wizard does not seem to make people do more research, just to have a few more looks over their work before publishing. Commented Jun 2, 2022 at 18:40
  • 3
    @Trilarion thanks. I'm not sure though if novelty effect is relevant when we are talking about first-time question askers. And I'll take the consistent improvements any day. At our scale, it can make a big difference.
    – Yaakov Ellis StaffMod
    Commented Jun 2, 2022 at 18:42
  • 34
    If you insist on talking about statistical significance, please always quote confidence intervals, and don't use so ridiculously many significant digits. E.g. "11.36% reduction in deletions (a statistically significant result)." is silly. I guess the confidence interval is on the order of [5%,15%] or something like that. The way you put the results looks like the person writing it doesn't understand how to communicate data/statistics very well. Statistical significance isn't that relevant. We care about effect size and uncertainty in effect size. This is not biology. Commented Jun 2, 2022 at 20:00
  • 2
    Typo "A-Scored = % of questions where the sum of scores on all deleted answers is at least 2"--"all non-deleted" is consistent with the next bullet (& common sense).
    – philipxy
    Commented Jun 2, 2022 at 22:38
  • 4
    "And I'll take the consistent improvements any day. At our scale, it can make a big difference." Fully agree. I take improvements on that scale also every day. It looks as if askers invested a bit more effort there and that increased quality of questions, but it does not seem to make such a big difference. The amplitude of the observed change is too small for that. There is no magic bullet available. Still, keep up the good work. These changes surely add up and all together can make the difference in the end. Commented Jun 3, 2022 at 8:24
  • 42
    I find the following to be inconsistent with a basic understanding of statistics (emphasis mine): "This has a p-value of 0.13 and is therefore not statistically significant, but it is still trending in the right direction." Despite its flaws, the entire point of a p-value is to help us avoid mistaking differences that can be attributed to random variation as "still trending in the right direction". And, had this gone the other way, you surely would have dismissed it as such. Better to just own that it's statistically insignificant rather than read the tea leaves of random variation. Commented Jun 3, 2022 at 21:39
  • 14
    "we did not see any negative effect from having the Stacks Editor in the Ask Wizard test" That statement is blatantly false. Negative effects were definitely reported during the trial. As to it showing or not showing in your data, there is no indication that you arranged to test in a way which would allow you to say anything about the Stacks Editor, one way or another, other than "it wasn't catastrophic". The primary way to get that data would be to try the Ask Wizard with and without the Stacks Editor, which, as far as I know, wasn't done.
    – Makyen Mod
    Commented Jun 8, 2022 at 0:04
  • 12
    Saying "we did not see any negative effect" & basing a decision on it when you created a situation where it's impossible for a non-catastrophic negative effect about that thing to show in your data is intentionally abusing the testing process and reporting of statistics to get the desired result (particularly when negative effects were, in fact, reported). I really expected better of you. I'm seriously disappointed (not that the Stacks Editor will be used, but that deception is used to justify it; well, OK, I am also disappointed the Stacks Editor will be used, but it was expected).
    – Makyen Mod
    Commented Jun 8, 2022 at 0:04

9 Answers 9

38

What about the Stacks Editor?

..because we did not see any negative effect from having the Stacks Editor in the Ask Wizard test, there are no plans to remove the Stacks Editor from the Ask Wizard..

Why would you see any negative effect?

Lots of new posts have absolutely no formatting whatsoever. In the rare occasion I see a first post with nice formatting, I always wonder if that's an old user with a new account. And I have to ask myself "How do you know the difference between lack of formatting between these two?"

  • No formatting using the Stacks Editor
  • No formatting not using the Stacks Editor
11
  • 10
    @bad_coder The Stacks Editor gives me an eye twitch ;)
    – Scratte
    Commented Jun 2, 2022 at 18:13
  • I see Stacks editor struck a nerve for you to post after all this time. I also don't like the new editor but since SO Teams is using it by default SE has no choice but to iron out the bugs fast...
    – bad_coder
    Commented Jun 2, 2022 at 18:15
  • 9
    Why would you see a negative effect - if the user experience was as awful as folks here are making it out to be, then I would expect to see a higher degree of user abandonment of the question-asking process. Or a much lower quality of question formatting, which would result in more closures, more review activities, lower question success. We see none of that.
    – Yaakov Ellis StaffMod
    Commented Jun 2, 2022 at 18:21
  • 11
    That people were able to make it work, isn't evidence it was better.
    – Kevin B
    Commented Jun 2, 2022 at 18:24
  • 34
    @YaakovEllis I do not think that is correct. Some users seem to monitor new posts and try their best to edit in proper formatting. We're not suppose to close posts due to bad formatting, we're suppose to edit them. A Question can be good despite formatting and vice versa. As to user abandonment in the asking phase.. they're already not formatting their post. They have an issue, and they want an Answer. Question quality isn't a priority.
    – Scratte
    Commented Jun 2, 2022 at 18:24
  • 8
    @YaakovEllis If we can have people make better posts by using (or forcing :-) a different or better wizard, then I'm all good with that. I just don't understand why or how the Stacks Editor has any bearing. It seems to me it was just thrown into the mix.
    – Scratte
    Commented Jun 2, 2022 at 18:35
  • 18
    Personally during the test, I recall seeing hundreds of questions which had extremely difficult and tedious to fix issues with formatting. Specifically Is the new editor's inline code tool supposed to break up code blocks? requiring basically line-by-line removal of sometimes hundreds of ` characters and Punctuation characters being escaped in code were extremely difficult to fix because sometimes it was not clear which things were escaped and which were part of the actual code.
    – Henry Ecker Mod
    Commented Jun 2, 2022 at 20:20
  • 8
    @YaakovEllis A negative and a positive might cancel each other out. If the new wizard is generally better but stacks editor makes it worse then the results should be similar to baseline.
    – Dharman Mod
    Commented Jun 2, 2022 at 22:49
  • 32
    I fully agree with this answer. The bad formatting resulting from the new editor would have no impact on the measured statistics. If you were able to measure the nerves of viewers or people who actually took the time to fix the formatting, you could see some difference. The numbers you collected simply do not show this and do not support the conclusion SE came to.
    – Dharman Mod
    Commented Jun 2, 2022 at 22:52
  • 7
    Thanks for your feedback. Please see my request to submit (in the comments to that answer) links to posts on SO with these quality issues, Meta bug reports (if any), and GH issues (if any). I know that this has been written about in other places - aggregating these specific issues here can help us to prioritize these issues more, and get as many as possible done before the Ask Wizard graduates.
    – Yaakov Ellis StaffMod
    Commented Jun 3, 2022 at 11:36
  • 2
    The issues brought up here have been fixed in the Beta 1 release. Read more about the recent release of the Stacks Editor, and plans for continued development.
    – Yaakov Ellis StaffMod
    Commented Jul 12, 2022 at 18:23
16

So we actually saw a statistically significant increase in the completion percentage from users who were in the Ask Wizard test.

The ideal case is that the user finds a duplicate and doesn't have to bother other humans with their question. Or that the process of reducing their problem to a MCVE helps them spot the bug, again without having to take up anyone else's time.

Since you found weak evidence for a slight decrease in questions being closed, that might mean that even fewer people gave up, vs. more stopping for the right reasons (solving their problem without actually asking).

But if that's still not happening, then there's room for improvement. @TravisJ's answer reports that the list of duplicates shown from inside the ask wizard itself is still often not useful canonicals from the right tags.

Best case for many beginner questions is no question at all cluttering up future search results and taking up people's time reading it; second best is an easy-to-read question where experts can quickly identify the problem and close it as a duplicate of a canonical Q&A.

I think overall we want that number to go down, though. Of course I understand it would be a concern if the ask rate went down without significantly reducing the number of duplicate closures, but as an overall long-term goal more questions asked isn't purely good.

(Sometimes it's not easy for beginners to realize what their real problem is, and it's very helpful for an expert to point them at the right canonical. That's understandable when it's not something that would be covered in most tutorials for the language in question.)


Would it be useful to give the user a button to indicate that they're abandoning because they found an answer?

e.g. next to "discard", a button like "solved while making a [MCVE] and/or looking for duplicates". Only with some more UI-friendly name, obviously that's too long a phrase for a button. Having another button at all might be too much clutter, I'm not a UI designer.

And you'd probably only get a weak signal, with many users closing the page instead of using "discard".

It seems to me, if you're tracking follow-through percentage on users who start to ask a question, it's incredibly valuable to somehow figure out whether they gave up for the right reasons (solved on their own) or they gave up without really finding an answer. Perhaps there isn't a good way to do that.

4
  • "Best case for many beginner questions is no question at all cluttering up future search results and taking up people's time reading it; second best is an easy-to-read question where experts can quickly identify the problem and close it as a duplicate of a canonical Q&A." What should be an appropriate metric to measure success then: "Successfully, non-closed questions asked minus quickly closed duplicate questions minus question that get closed later multiplied by 5 and then try to maximize this score? Commented Jun 10, 2022 at 9:28
  • 3
    @Trilarion: I'm not a data analyst or user analytics expert at all, I don't know how to measure this! Obviously that's very challenging, but it means that a drop in asker follow-through is probably a good thing if it comes as a result of improvements in duplicate finding, or maybe other things that encourage a better MCVE. That's the point of my answer, just to remind everyone that the best case is no question. Commented Jun 10, 2022 at 9:48
  • 1
    @Trilarion: The part about "easy" duplicates was mostly just to point out that I realize there are cases where beginners will have a hard time realizing they have a duplicate and thus finding an existing Q&A, because they don't know the terminology or what the actual problem is. IDK if there's any sense in trying to measure "easy" duplicates vs. other kinds of closure, and that wasn't what I intended to suggest. Merely that better quality questions are a good thing even when it leads to duplicate closure. Commented Jun 10, 2022 at 9:50
  • 1
    You could do some of this data collection by looking at whether the user who abandoned their question clicked on any of the suggested duplicates. If someone clicked on a duplicate, spent a while on that page, and scrolled down far enough to read an answer or two, it's likely that they solved their problem. It's not a perfect indicator, but it's at least a strong hint. If you never clicked on any of the suggestions, then we know that's not what happened.
    – bta
    Commented Jun 17, 2022 at 20:47
14

A/B testing involves exactly one A and one B. If you add more than one thing to either side, it's not A/B testing—it's A/B+C testing. And that isn't a thing because it means that the data on each side cannot be meaningfully compared in any way shape or form.

All those beautiful statistics you've trotted out about the Ask Wizard? Useless. Because they aren't about the Ask Wizard, they're about the Ask Wizard + Stacks Editor. So comparing them to anything at all is impossible.

Even if we set aside the flawed data-collection process and accept the data on face, a one percent variation is not statistically significant in samples of this size.

In other words, this entire exercise is completely pointless because your data is invalid. In fact, this exercise is worse than pointless because, in a frustrating pattern we've seen time and again, you're going to use this data (which is intrinsically flawed) to justify imposing the Ask Wizard and Stacks Editor on users.

This is not the first time this sort of data capture and analysis failure has happened here, and it quite honestly baffles me. Concepts like A/B testing are, in my opinion, quite straightforward. It is surprising that Stack Exchange staff members would fall victim to such fundamental mistakes. It almost seems that this particular A/B test was botched due to malice. In other words, you don't actually care about doing a proper A/B test because you'd already decided that the Ask Wizard and Stacks Editor will be imposed on users because, as staff, you know better than this community, many of whom have been users of this site longer than most Stack Exchange Inc. staff members have been employed.

If you're going to impose things on us, fine, do it; it's your site. But it's off-putting to pretend that the decisions are driven by data and community feedback when the conclusions seem to have been foregone.

4
  • "a one percent variation is not statistically significant in samples of this size" Is it? Or isn't it? I don't know. One typically used way to get error estimates is bootstraping, which is kind of robust because it's not so model dependent. Maybe the analysis here could have applied bootstraping in order to evaluate if the observed differences are statistically significant (larger than the error margins). Commented Jun 10, 2022 at 9:30
  • 3
    While I agree with the sentiment, I think this goes too far in stating the data is invalid and "cannot be meaningfully compared in any way shape or form". The comparison which was done was A vs (C+D), which is a valid A/B comparison, but where "B" is "(C+D)". The point is that the data from such a test doesn't allow breaking out conclusions about C or D separately, but only about the combination. Saying that the data can't be used at all is, at best, an exaggeration. The data just isn't valid to use for anything regards to the Ask Wizard or Stacks Editor separate from each other.
    – Makyen Mod
    Commented Jun 10, 2022 at 15:06
  • OTOH, this doesn't mention the negative feedback about the effects of using the Stacks Editor on the state of questions, and the need for substantially increased volunteer time to correct those issues. The statistics don't cover that data and it's completely ignored in the analysis. Volunteer time is actually a limited, valuable resource, even if there's not a clear monetary value attached to it for the company. Increasing the necessary volunteer work is likely to result in larger negative consequences for the site, particularly when the volunteers feel the increased work was avoidable.
    – Makyen Mod
    Commented Jun 10, 2022 at 15:14
  • "I think this goes too far in stating the data is invalid ... The data just isn't valid to use for anything regards to the Ask Wizard or Stacks Editor separate from each other." That's the problem. We cannot be sure. Not only may the observed differences not be statistically significant, they also may be biased by an unknown amount. The relevant question is, what we have learned from the test in the end. Should the conclusion really be "We are very happy with the results [...]. It met or exceeded just about all of our expectations when it came to the goals of the component." or sth. else? Commented Jun 10, 2022 at 15:59
12

I can't get excited over these statistics like you can since we were never informed of what your key metrics were (i.e. x% improvement in y) or what you would consider statistically significant ahead of time.

To me, this looks pretty much the same with just an inch of improvement being made (in a wave of a lot of the same). Y'all took a positive increase and just ran with it.

Nothing here speaks to a significant or ground-breaking improvement in question quality or participation, except for the fact that - if we put a wizard in front of people it doesn't seem to make that much of a difference.

So at the end of the day this gets a hearty shrug from me. Y'all seem to be happy with it, so it's kinda whatever at this point. But if you want someone like me to also share in your happiness and success, you should do a job of including or identifying what it is that someone like me would consider success from functionality like this.

19
  • 2
    A lot of the stats seem to demonstrate marginal improvements. That said, unless I've misread the post, the review load seems substantially better under the wizard. That moved my needle a bit.
    – canon
    Commented Jun 6, 2022 at 19:20
  • 2
    This is an improvement though, as they were able to introduce a feature with a lot of future value without taking on any negative effects. If anything, using an ask a question wizard represents a barrier to participation, and as such measuring output which is inline with the status quo is a success. When you are learning to pitch, you don't go straight to the fastball, first you have to consistently hit the strike zone.
    – Travis J
    Commented Jun 6, 2022 at 21:46
  • 5
    I don't think the company had specific performance goals. They're just looking for any improvement that they can get. Which... I don't think is completely unreasonable, given the situation on the ground.
    – Cody Gray Mod
    Commented Jun 7, 2022 at 3:16
  • 11
    We did not have any specific percentage goals. The success criteria here were that it did something positive in line with the goals of the staging ground, and would give us a platform to continue to iterate on for the furtherance of these goals. We achieved this. I am having trouble understanding why there is so much negativity around celebrating this.
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 7:05
  • 5
    @YaakovEllis: It's not negativity, and I hope that you're not categorizing my interactions here on the site as "negative". From my perspective, as someone who's about an order removed from the experiment and its numbers, and as someone who's been left hanging on how successful this effort actually was in the past, I remain skeptical at anyone shoving p-values or percentages in my face telling me that this is "good" and that we're moving in the "right" direction.
    – Makoto
    Commented Jun 7, 2022 at 19:12
  • 5
    @YaakovEllis: At worst, you could characterize my response here as unenthused. Until literally now, I didn't know what you were going for, or by how much of a margin you were trying to go for it. The numbers here just don't really speak to me or how I want to see Stack Overflow improve or grow, and I have this deep and abiding concern that the company is just going to run with these numbers as positive and treat that as a baseline justification to do...stuff.
    – Makoto
    Commented Jun 7, 2022 at 19:13
  • 6
    @YaakovEllis: Given your tacit response earlier, though, I do have a concern that while you may want feedback, you don't particularly appreciate the kind of feedback I'm providing to you. It feels like from there, there's a pretty substantial breakdown in how we approach one another on Meta. How would you like me to address that, or am I misreading this?
    – Makoto
    Commented Jun 7, 2022 at 19:14
  • 3
    @Makoto Maybe I am unfairly taking my frustrations at the overall trend and tone in responses above out on you. If I am doing so, my apologies. In my eyes we had positive results here. Just about every single stat moved in the right direction, and to get a 10% drop in deletions on our first iteration is pretty great. As I wrote, this gives us a platform to iterate on, and the Ask Wizard is envisioned as the entry point to the Staging Ground (and not a standalone), where we will hopefully be able to get more results in improving question quality (and not just having things be less bad).
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 19:41
  • 4
    @Makoto I realize that you feel you have been burned in the past. I can't make up for the past, but for this project, I feel like we have been doing all that we can to be very open with communication, open to making changes and adjustments based on feedback, open with data and results. And the responses on this post have — with some notable exceptions — been nitpicky, overly critical, pedantic, and seeming to assume bad intentions from folks like me (I would have thought that I have earned the benefit of the doubt in this area) and others who have been working on this.
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 19:53
  • 4
    @Makoto I have been asking for feedback. And the word that I continually use to qualify this is constructive. Even if your message was that you were looking for clarity on our goals in the project, the way that you wrote it seemed to me to take a tone that was not constructive at all. Just overemphasizing in many different ways how underwhelmed you are, and even your comments above belie that you don't trust me and others working on this to be coming into this with intentions of honestly trying to build better tools for different segments of the community. And that hurts.
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 19:57
  • 3
    @YaakovEllis: I don't know how you (or the company at large) defines "constructive" feedback. There's this tenuous approach to how we're meant to give feedback to y'all at all if we have hopes of it being listened to which...hasn't improved in recent years. More to the point though, the goal is for you to get feedback to a feature you're building, and the community wants to give that feedback. But a lot of this speaks to some of the things I had mentioned to Phillipe when he started.
    – Makoto
    Commented Jun 7, 2022 at 20:19
  • 4
    @YaakovEllis: We're not your enemy, nor are we necessarily against you, but if the posture is to regard what we're saying as immediately negative - and frame that as not being constructive to boot, which feeds back into the cycle of what we say to you then actually becoming more negative - then we're always going to be stuck in this mentality. Something needs to change.
    – Makoto
    Commented Jun 7, 2022 at 20:21
  • 4
    what... benefit does knowing someone is "unenthused" with a feature have for those building it? What useful thing can someone draw from that information? I don't particularly trust the statistics either, given how often I feel like stats from all over SO and the surveys are often abused to write articles or come to conclusions for seemingly no reason other than for clicks, but, saying that doesn't... help develop this tool.
    – Kevin B
    Commented Jun 7, 2022 at 20:33
  • 12
    @Yaakov "the responses on this post have—with some notable exceptions—been nitpicky, overly critical, pedantic, and seeming to assume bad intentions from folks like me" I've read the responses, and I think that is a very unfair characterization/assessment. I suppose there has been criticism in the comments regarding the inaccurate use and misleading presentation of statistics, but that tends to happen any time a non-statistician tries to use statistics. I can only recommend not taking it personally. I know I don't respond differently because it's you. That's not a good precedent.
    – Cody Gray Mod
    Commented Jun 7, 2022 at 21:51
  • 15
    Most of my criticism, and that of others, has been due to the apparently incidental use of the editor in this experiment, rather than what we see as the actual thrust of the experiment, which is why we've been frustrated about it from the beginning, as we thought and still believe that that will/did skew the results. We've been expressing this for a long time, and our concerns about the editor; you, yourself, have been extremely dismissive of this, so pretending like you're somehow the victim of hurtful criticism with no part in it... well, TBH that strikes me as more than a bit tone-deaf.
    – Cody Gray Mod
    Commented Jun 7, 2022 at 21:53
11

Would just like to reiterate the need to offer a duplicate suggestion to the asker that takes both the title and tags into consideration. A good test run is to look at the title "how do I sort an array" with the tag "javascript". This should produce a suggestion of a highly upvoted, multi linked, duplicate target.

4
  • 4
    The Ask Wizard is already doing this. Duplicate suggestion happens after tags are selected, and includes tags. This change was made in response to user feedback from the initial announcement of the test, and was made public in the test is launched post.
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 6:26
  • 7
    @YaakovEllis - Please don't just dismiss this without considering the description of the problem. The crux of the issue is highlighted in the last sentence of this answer. The duplicate suggestion may be considering tags, but it is not producing highly valued targets which would qualify as existing posts which answer the askers question.
    – Travis J
    Commented Jun 7, 2022 at 18:54
  • 2
    In our testing of this, the duplicate answers given when taking the tags into account are definitely more relevant. That said, we saw absolutely no change in the number of closed-as-dupes, indicating the better accuracy in duplicates offered didn't really accomplish anything. We definitely want to find ways to attack the duplicate issue, but it will most likely require a separate research, design, and dev initiative that is beyond the scope of this current project.
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 19:35
  • 4
    @YaakovEllis - It didn't seem more relevant, although it did at least seem to include them. There is likely a direct correlation between the increase in relevance and the rate at which questions are prevented, which would be reflected in a drop in the closed-as-dupes rate from the wizard. What exactly did you try, when including the tags? Because the list generated, to my observation, does not look relevant at all, except for the fact that it at least may have jumbled more towards a tag than not. To be honest, if you cannot get the suggestions to work this wizard wont have much impact.
    – Travis J
    Commented Jun 7, 2022 at 22:56
9

It seems to me that you're using results of a test on a certain population to draw inferences about the treatment effect on a different population. You say

First-time askers going through the Ask Wizard experienced a 11.36% reduction in deletions (a statistically significant result). ... First-time askers going through the Ask Wizard experienced a 4.74% reduction in the closure of non-deleted questions.

But then you say:

Who will be using the new Ask Wizard once it graduates?

Based on the success of the tool, we are planning on making it the default for all question askers on Stack Overflow, including users who have already asked in the past.

That conclusion isn't supported by "the success of the tool," which was for first-time askers only. As you probably know, treatments can have different effects on different populations.

Perhaps you have some compelling reason to believe that the Ask Wizard for New Users will be equally beneficial for experienced users. If so, please share, and please also explain why experienced users weren't included in the test.

1
  • 1
    New users are the users most likely to need assistance asking a question that falls within the site's guidelines. Yes, experienced users can also ask low quality questions, however that bucket is far smaller. Additionally, targeting only new askers eliminates any bias due to the new editor vs the old.
    – Kevin B
    Commented Jun 15, 2022 at 21:56
7

Thanks to all who have given feedback so far. I have heard consistently from a number of users about quality issues that they have seen with the content that came out of the Stacks Editor during the Ask Wizard test (even though this might not be reflected in the stats reported above).

As I wrote above, we want to wait (at the very least) for a pending milestone release for Stacks Editor to go out before graduating the Ask Wizard, which will include more bug fixes, and full feature parity with the current editor.

Please continue to submit bugs and feature requests here on Meta using the stacks-editor tag, or use the GitHub project for the editor to check out the issues still scheduled for work and to submit new issues.

That said, if there are specific types of quality issues that you saw come through from questions written in the initial test, please reply to this answer with as much info as you can: link to a question or questions showing the issue, link to meta issues reporting it (if any), link to GitHub issues reporting it (if any).

There have been many issues reported for the Stacks Editor since it went to alpha testing. Many have been fixed. Many still remain on the roadmap. And we want to get to all of them. But having a more concrete list here of items that you saw coming through can make it much easier for me to help the team working on the Stacks Editor to prioritize these issues higher, so that we can hopefully get through them (or as many as possible) before the Ask Wizard graduates.

We want this release to be helpful to all users, and it is definitely not anyone's intention to subject the gracious community of editors/curators/reviewers to avoidable quality issues with post formatting.

(Links to posts that complain in general about the Stacks Editor or that are collections of complaints are less helpful here than links to questions on SO with quality issues, reports on MSO and MSE, and GitHub Issues).

Thanks again with your help on this.

3
  • 26
    I find this answer, and others like it, to be reasonable on its face, but ultimately unsatisfying, as it does not attempt to explain why something that is admittedly only an "alpha" release, and something which was repeatedly brought to staff's attention as being exceptionally buggy, was actually released as part of an unrelated A/B test, and, more importantly, why it continues to be something that is considered for future release. That's just not how code having an "alpha" status is supposed to be treated, or at least it's not in my world, and I expect(ed) better of Stack Overflow, too.
    – Cody Gray Mod
    Commented Jun 3, 2022 at 11:38
  • 9
    @CodyGray Read the intro post for the ask wizard why we used it. The current editor with preview just did not work well with the wizard experience that had multiple editors on the same page (along other reasons). As for the bugginess, work on the stacks editor has already restarted at the time (and there had also been issues already fixed in the interim), so we deemed it worth trying out in a limited test (especially for new users for whom the WYSIWYG mode would be more helpful).
    – Yaakov Ellis StaffMod
    Commented Jun 3, 2022 at 12:34
4

Why is the ask wizard considered part of the Staging Ground at all? Isn't how people ask questions not related to the SG's purpose of ensuring questions asked by new and problematic users meet our standards before they are released into the wild?

Surely a tool like an ask wizard, meant to improve the ability of people to ask questions, should be considered on its own and presented as a tool for everyone who wishes to use it... not just as part of the SG.

How has the performance of this iteration of the ask wizard compared to previous attempts at creating an ask wizard?

7
  • 1
    Almost feels like this was shoehorned in as an excuse to use the new stacks editor. The old wizard used our current editor, why can't this one? Is people's ability to use markdown really that big of a barrier to entry?
    – Kevin B
    Commented Jun 7, 2022 at 21:04
  • 3
    It is considered the first stage of the Staging Ground. The SG is a self-contained unit, and in theory could work with the current asking experience. However, the overall goal of ensuring questions meet our standards, and helping the general onboarding of new askers, is also furthered by improvements to the question-asking process. That said, we are also treating it as its own tool (and sites will eventually have the option to only use the Ask Wizard without the Staging Ground if they so choose).
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 21:06
  • 5
    No, this wasn't built to give us a reason to use the new editor. No conspiracy over there. I am saying that as someone who was in all the planning meetings. The original plans and designs were not committed to one editor or the other. We made the decision during the dev process primarily because of the difficulties in constructing and usability when having multiple (old) editors on the screen one on top of the next. The old wizard was designed in a way that this was less of an issue (each step took up its own screen).
    – Yaakov Ellis StaffMod
    Commented Jun 7, 2022 at 21:09
  • 2
    I guess what I was trying to get at was, the last time a wizard like this one was implemented, it was generally considered a great idea... until it was implemented and generally resulted in useless boilerplate being added to questions. Improving the asking process is certainly a great goal, but I'd hate for this to just repeat the mistakes of the past.
    – Kevin B
    Commented Jun 7, 2022 at 21:56
  • 1
    Regarding the editor, I certainly have a bias against WYSYWIG editors, as I've implemented them quite a few times myself and found them to generally produce very inconsistent results. I've also implemented SO's current editor a few times for our own projects and found it to, obviously, be more consistent given that it doesn't rely on the browser generating html from user input. however, I certainly understand using something more friendly to newcomers can be attractive, and necessary for your paid products. I just... see using it in this project counter-intuitive, ...>
    – Kevin B
    Commented Jun 7, 2022 at 22:00
  • 1
    given how long it's been in development while still having many of the same issues it's had all along.
    – Kevin B
    Commented Jun 7, 2022 at 22:01
  • 2
    Effectively we’ve repeated a past experiment and potentially solved its shortcomings, but… didn’t really improve upon the results in any meaningful way, aside from getting a potentially more new user friendly editor in front of some users.
    – Kevin B
    Commented Jun 7, 2022 at 22:33
2

There's something odd about the reasoning here. This test was essentially designed to see how well the ask wizard worked for Stack Overflow users. The test was successful, so the feature gets approved. It then later gets rolled out to other sites in the network. The step that's missing - which I feel is the one that will give the most useful information - is the step where you test how well the ask wizard works on other sites in the network. There's an underlying assumption that something that works (or doesn't) on Stack Overflow will behave similarly elsewhere, but the sheer variety of sites on the network more or less guarantee that isn't true. If you roll it out globally and only then learn that it has significantly negative results on every site that isn't Stack Overflow, then you have to do a major redesign and most of the hard work you've done so far is wasted. Individual sites can disable it, but if it works for SO and doesn't work for the other 170+ sites in the network, it still seems like an overall waste of time. I really would have expected this sort of trial to be run in parallel on a diverse handful of the more heavily-trafficked network sites, and then the results analyzed in aggregate and on a per-site basis.

If you do re-run this test on other network sites (which I really hope you do), here are a few small things that you could tweak.

Your numbers for "completion percentage" compared the number of people that clicked on the "Ask Question" button with the number that completed the question submission process. This percentage will be artificially low because it includes accidental clicks, people that (for reasons beyond my comprehension) think you need to double-click things on web pages, people who have a tab hibernate and reload itself, etc. It would be better to only track users who actually started filling out the "New Question" form.

Also, one of the more interesting data points to me was the data for "suggested edits", which was hardly mentioned:

Baseline UR Ask Wizard UR Baseline A/R Ask Wizard A/R
Suggested Edits 1,682 990 6.40 2.75

Combine the "# reviewers" and "actions per reviewer" stats to get "total number of suggested edits required". The baseline group required 10,765 edits, and the ask wizard group required 2,723 edits. That's a very large decrease. This is particularly important because making edits can be significantly more time-consuming than merely voting to close, etc. What I think would be particularly interesting is if you added code to measure how long each reviewer spent reviewing a question. That would give the missing data to convert these numbers into "total time spent editing questions", which would arguably be the most important number in this whole experiment. Reviewer time is perhaps the most scarce commodity we have, and anything that significantly reduces time required would be welcome. If the total number of closed/deleted questions increased by 10% but the average time spent reviewing each question dropped by 30%, that's still a very noticeable improvement in our ability to do timely and effective reviews. That would also let you segregate "suggested edits" into sub-categories - a large reduction in very short duration edits (fixing spelling errors, etc) is not as helpful as a smaller reduction in long duration edits (like having to decipher and format a massive block of copy/pasted code).

3
  • 3
    While I agree other sites on the SE network have substantial value, solutions which work only on Stack Overflow, with 65% of all SE Network answers, are not of no value, nor are they a waste of time. Dialing back the hyperbole in your statements about that would be beneficial. Doing so might result in people taking your arguments a bit more seriously, rather than dismissing them due to being substantially wrong as a result of the hyperbole. Basically, having obviously inaccurate statements at the start will result in at least some people ignoring anything past those statements.
    – Makyen Mod
    Commented Jun 17, 2022 at 22:52
  • @Mayken I'm not saying the solution is a waste of time. I'm saying that a widespread roll-out before doing this sort of testing globally is a waste of time. It's much easier, faster, and less disruptive to collect the data upfront than to try to do significant work on a feature after it goes live.
    – bta
    Commented Jun 20, 2022 at 21:31
  • 2
    You literally say "most of the hard work you've done so far is wasted" and "it still seems like an overall waste of time". How you've used those statements reads to me as you are saying the entire thing is a waste of time. That may not be what you're intending, but it's how it reads.
    – Makyen Mod
    Commented Jun 20, 2022 at 22:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .