As a preface, since this question was asked, we've implemented the Ask Question Wizard, which tackles the problem of duplicate questions by automatically showing a list of questions immediately after the tags and title are entered. Therefore the interstitial isn't shown for those users. So I'm going to divide the results into two eras: pre- and post-wizard.
Era |
Visited |
Thanks |
% |
Asked |
% |
|
Search |
% |
Asked |
% |
pre |
3,692,047 |
3,047,882 |
82.5 |
2,109,823 |
69.2 |
|
1,249,694 |
33.8 |
789,989 |
63.2 |
post |
51,473 |
37,357 |
72.6 |
23,562 |
63.7 |
|
11,985 |
23.3 |
6,697 |
55.9 |
I'm looking at account-level statistics, which should be close to measuring the number of people who have taken the various actions. The design encourages people to search and then proceed to check the "thanks, I will keep these tips in mind when asking" checkbox if they don't get useful results. The "Asked" columns count people who asked after clicking the "Thanks" checkbox and after doing a search. It's two separate yet overlapping funnels. Finally, people who have visited the page since the wizard launched would have first opted out of the wizard to use the traditional asking UI. (For the rest of the post, I'm going to lump both eras together since the results are not significantly different.)
As you can see, people who search first are less likely to go ahead and ask. Presumably, some people find the answer to their question and don't need to post it after all. (We've been experimenting with the way potential duplicate questions are surfaced to achieve that result without an explicit search.) Also notice that quite a few people don't end up asking a question even if they don't do a search. The nature of adding a step in any process is such that some people will stop right there. Of course, it's impossible to know if the questions that aren't asked would have been more or less interesting than the ones there are.
We can tell whether questions asked after using search were better or worse according to question grade:
Search first? |
Close % |
Dupe % |
Bad (%) |
Neutral (%) |
Good (%) |
Avg. Score |
true |
19.5 |
5.7 |
29.67 |
22.84 |
49.55 |
-0.18 |
false |
13.9 |
4.5 |
29.87 |
21.31 |
50.86 |
-0.16 |
Counter-intuitively, askers seem to do worse if they searched first. It's hard to imagine searching is causing people to ask duplicate questions. Instead there's likely some set of confounding factors. One might imagine, for instance, some people prepare to ask their question and have already searched for duplicates before hitting this page. That group of people wouldn't benefit from an extra search and seem prone to ask better questions. Alternatively, it could be that people who have not acquired skill at searching try to compensate by searching more often. Or, of course, I could have an error in my queries. (Joining chronological events in SQL isn't always straightforward.)
We can infer that people who search are looking at the results:
Search first? |
Median read time (s) |
true |
935 |
false |
131 |
Obviously, some people read a lot more quickly and some people leave the tab and don't come back to ask for quite some time. That said, many people seem to be taking the time to read the advice thoroughly. My biggest concern with the interstitial page has always been that it can be hard to internalize a wall of text. One thing that seems to be working well with the wizard is that it parcels out guidance in a more focused way. I'm not convinced it's a good thing that people are spending 15+ minutes on the interstitial. Learning a skill (such as asking a programming question) is often better in the context of working on a task and getting timely feedback.