Update Mar 21: TL;DR: Today, we graduated the experiment for Stack Overflow question pages with zero answers, specifically the variant where three related questions are shown by default, with a link to view more.
During the experiment, we observed a statistically significant 900%+ increase in clickthrough rates (CTR) in both experiment groups. Users in the experiment were engaging with related questions at an exponential rate compared to those in the control group where related questions were shown in the sidebar.
Update Feb 24: As of Feb 23, testing has concluded
Update Feb 14: Testing is now active
Background
As part of the Content Discovery initiative, we’re announcing the next experiment. In our previous experiment to reorder related questions, we mentioned that most users are not seeing related questions above the fold. We also believe that banner blindness is a common web behavior and pattern, wherein users have a high tendency to ignore all forms of advertising that typically appear in the sidebar.
We believe displaying related questions within the answers list will allow users to get to the most relevant content sooner. If users typically ignore the sidebar, this new option could allow them to naturally discover related questions. In this post, we explain the goals of the experiment, success criteria, and how we will conduct the test.
Experiment goals and success criteria
Given that this is our first attempt to display additional content within the main content area, we’re interested in learning how users will engage with it and whether this will provide any incremental value when trying to find relevant content to get closer to their just-in-time needs.
Our null hypothesis is that engagement with related questions remains the same in both experiment groups. We will determine whether the variant is a winner if there is a statistical significant lift to clicks on related questions, thus allowing us to reject the null hypothesis.
How we’re conducting the test
The experiment will be conducted on Stack Overflow question pages for anonymous users only. It will target 20% of the traffic that will be split evenly between the control and two variant groups.
We learned from recent user research that users tend to check for other answers and comments right away after reading a question and try solutions offered in Q&A. If users aren’t able to find an answer, we think they are more likely to bounce as this prevents them from being able to test potential solutions. We are initially focusing our efforts on questions without answers in order to validate whether the inline placement results in better engagement before we iterate on question pages with at least one answer.
Here are screenshots of how related questions will appear in each variant group. In the first variant, users will see three related questions below the question post with an option to view more by clicking on the “See more related questions” link. In the second variant, users will see up to five related questions below the question page. Users who are opted into either variant group will only be seeing inline related questions on question pages with no answers. The sidebar related questions on those pages will also be hidden. Question pages with at least one answer will not be impacted by this experiment. Users in the control group will continue to see related questions in the sidebar.
Figure 1: Three related questions displayed below the question post with a link to view more
Figure 2: Up to 5 related questions displayed below the question post
The experiment will run for at least a week in order to account for the newness effect. Once the experiment concludes, we will analyze the results and share them with Meta.