Update Nov 22: V2 experiment has concluded. We will share results and next steps soon.
Update Nov 17: V2 of the experiment is now active. The experiment will conclude tomorrow. We will update Meta when it's concluded.
Update Nov 8: We will be conducting another experiment next week (week of Nov 14) wherein the Related Questions module will display a maximum of 5 links instead of 10 for anonymous users in the experiment group. We will update the post once the experiment is live.
Update Oct 31: This experiment has concluded. We will share results and next steps soon.
Background
Following on the announcement of the Content Discovery Initiative, we are kicking off the very first experiment this week.
In this test, we will be moving Related Questions further up in the sidebar for anonymous users. While community members can find value in the community bulletins, this information is less useful for someone looking for a just-in-time answer for one of their problems.
Given the average browser size, most users are not seeing the Related Questions above the fold. By bringing this module above the Community Bulletin and raising its profile, we expect it will help visitors to find relevant content.
Experiment goals and success criteria
The goal of this experiment is to increase the prominence and visibility of the Related Questions module. We believe that by moving the module higher up on question pages, users are more likely to discover relevant content that could help address their just-in-time needs. The assumption is that anonymous users may find more value in related questions compared to the Community Bulletin. For this experiment, we will be measuring the clickthrough rate (CTR) of the module. Similarly, we will monitor the clicks to the Community Bulletin to determine the impact of moving it further down the page.
How we’re conducting the test
This experiment should not affect any logged-in users and will only run on stackoverflow.com. It will target 1% of the traffic that will be split evenly between the control and variant groups. 50% of anonymous users will see the current sidebar where the first module displayed is the Community Bulletin, which includes links to the Stack Overflow Blog, Featured on Meta posts and Hot Meta Posts. The other 50% will see the Related Questions module displayed first. No changes have been made to how the questions are selected in the module.
The experiment will run for at least three days. For both groups, we will be measuring the click-through rate (CTR) on Related Questions and the Community Bulletin.
Here is a mockup of the experience for anonymous users who are bucketed in the variant group:
Once the experiment concludes, we will analyze the results and share with Meta.
Related Questions
sideBar altogether. They are usually of so little relevance that the section should be burminated. Is there any telemetry on how often they are clicked on?Linked
CTR would have been higher, but am not surprised by the CTR ofRelated
. I just looked through about 10 questions andRelated
links in my area of knowledge and there were only 4Related
links on two questions that sounded on topic (and they were like 5th and 6th in the list of ten so that is troublesome). So that was 4 helpful links out of 100. (I wonder if higher scoring answers are being over-weighted in the results?) I frequently look at theRelated
list and the first 3 or 4 are so far off-topic I don't look further.