36

Way back in 2009, an interstitial page was added to Stack Overflow for users with less than 10 reputation when asking a question.

It looks like this:

Stack Overflow interstitial


I have a couple questions about effectiveness.

  • How often is that search box used?
  • After it is used and a new question is still posted, what is the outcome of those questions? (I'm really looking for if the question stays open or is closed and for what reason)
  • How quickly, on average, is the page clicked through? (This gives a general idea of if much of the page is being read)

gnat raises a good point that this information about Server Fault would also be useful. Could the same information, but for Server Fault, also be provided?


I am attempting to answer a question over on Hardware Recommendations about adding such a page to improve our quality and guidance to new users.

I am also operating on the assumption that the search bar is not preventing a lot of new questions from being closed as duplicates.

4
  • 2
    consider also asking for stats at SF meta, they too have interstitial since 2013 (on a general note, Server Fault may be a better example to follow at HR than SO, their scale is much closer to HR and they overall make an excellent example of long term successful smaller site)
    – gnat
    Commented May 27, 2016 at 15:18
  • 1
    @gnat Added a request for SF's data too. Thanks for pointing out they have the interstitial too.
    – Andy Mod
    Commented May 27, 2016 at 15:21
  • 6
    I remember when I got that box, I thought it was pointless because I had already searched Stack Overflow. I also remember thinking that the results when you type up the question were better.
    – Laurel
    Commented May 27, 2016 at 16:23
  • 4
    @Laurel I never use SO search anywhere, I use google to search SO. I remember many times trying to find a question in the "close as duplicate" dialog and always going for google in the end.
    – Sulthan
    Commented May 29, 2016 at 8:24

1 Answer 1

10

As a preface, since this question was asked, we've implemented the Ask Question Wizard, which tackles the problem of duplicate questions by automatically showing a list of questions immediately after the tags and title are entered. Therefore the interstitial isn't shown for those users. So I'm going to divide the results into two eras: pre- and post-wizard.

Era Visited Thanks % Asked % Search % Asked %
pre 3,692,047 3,047,882 82.5 2,109,823 69.2 1,249,694 33.8 789,989 63.2
post 51,473 37,357 72.6 23,562 63.7 11,985 23.3 6,697 55.9

I'm looking at account-level statistics, which should be close to measuring the number of people who have taken the various actions. The design encourages people to search and then proceed to check the "thanks, I will keep these tips in mind when asking" checkbox if they don't get useful results. The "Asked" columns count people who asked after clicking the "Thanks" checkbox and after doing a search. It's two separate yet overlapping funnels. Finally, people who have visited the page since the wizard launched would have first opted out of the wizard to use the traditional asking UI. (For the rest of the post, I'm going to lump both eras together since the results are not significantly different.)

As you can see, people who search first are less likely to go ahead and ask. Presumably, some people find the answer to their question and don't need to post it after all. (We've been experimenting with the way potential duplicate questions are surfaced to achieve that result without an explicit search.) Also notice that quite a few people don't end up asking a question even if they don't do a search. The nature of adding a step in any process is such that some people will stop right there. Of course, it's impossible to know if the questions that aren't asked would have been more or less interesting than the ones there are.

We can tell whether questions asked after using search were better or worse according to question grade:

Search first? Close % Dupe % Bad (%) Neutral (%) Good (%) Avg. Score
true 19.5 5.7 29.67 22.84 49.55 -0.18
false 13.9 4.5 29.87 21.31 50.86 -0.16

Counter-intuitively, askers seem to do worse if they searched first. It's hard to imagine searching is causing people to ask duplicate questions. Instead there's likely some set of confounding factors. One might imagine, for instance, some people prepare to ask their question and have already searched for duplicates before hitting this page. That group of people wouldn't benefit from an extra search and seem prone to ask better questions. Alternatively, it could be that people who have not acquired skill at searching try to compensate by searching more often. Or, of course, I could have an error in my queries. (Joining chronological events in SQL isn't always straightforward.)

We can infer that people who search are looking at the results:

Search first? Median read time (s)
true 935
false 131

Obviously, some people read a lot more quickly and some people leave the tab and don't come back to ask for quite some time. That said, many people seem to be taking the time to read the advice thoroughly. My biggest concern with the interstitial page has always been that it can be hard to internalize a wall of text. One thing that seems to be working well with the wizard is that it parcels out guidance in a more focused way. I'm not convinced it's a good thing that people are spending 15+ minutes on the interstitial. Learning a skill (such as asking a programming question) is often better in the context of working on a task and getting timely feedback.

6
  • 1
    askers doing worse if they searched first can also be explained by some kind of negative selection. One can assume that better part of those who searched ended up finding their answer and abstained from asking because of that - so that many among the rest (who had to resort to asking) could be those who did worse (unable to grok that search results contained what they needed)
    – gnat
    Commented Aug 8, 2019 at 15:24
  • 1
    @gnat: Sure. It's also important to remember that some duplicate closures are not between identical questions. So a user might look at their results, not see the solution to their problem and still have it closed as a duplicate of a question that solves one part of their problem. Still. These results suggest that prompting people to search first is not some sort of silver bullet.
    – Jon Ericson Staff
    Commented Aug 8, 2019 at 15:42
  • "As you can see, people who search first are less likely to go ahead and ask", from that aggregated data we can't see that. Is there a missing table?
    – Braiam
    Commented Aug 8, 2019 at 17:52
  • 1
    @Braiam: No. If you look at the two "Asked" columns in the first table, you can see the percentage of people who asked after clicking the "thanks" checkbox or after using search. Since the first percentage is higher than the second, that means people who use search drop out of the funnel at a higher rate than people who don't use search. (Of course, using search adds a step. They still need to click "thanks" if they want to ask. So it might mean that people are giving up because every time there's an extra step some people don't do it.)
    – Jon Ericson Staff
    Commented Aug 8, 2019 at 18:12
  • Ok, so the table should read Thanks and Asked vs Search and Asked, correct?
    – Braiam
    Commented Aug 12, 2019 at 14:07
  • @Braiam: Correct. Sometimes I wish we had real table support.
    – Jon Ericson Staff
    Commented Aug 12, 2019 at 17:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .