30

Update Nov 22: V2 experiment has concluded. We will share results and next steps soon.

Update Nov 17: V2 of the experiment is now active. The experiment will conclude tomorrow. We will update Meta when it's concluded.

Update Nov 8: We will be conducting another experiment next week (week of Nov 14) wherein the Related Questions module will display a maximum of 5 links instead of 10 for anonymous users in the experiment group. We will update the post once the experiment is live.

Update Oct 31: This experiment has concluded. We will share results and next steps soon.


Background

Following on the announcement of the Content Discovery Initiative, we are kicking off the very first experiment this week.

In this test, we will be moving Related Questions further up in the sidebar for anonymous users. While community members can find value in the community bulletins, this information is less useful for someone looking for a just-in-time answer for one of their problems.

Given the average browser size, most users are not seeing the Related Questions above the fold. By bringing this module above the Community Bulletin and raising its profile, we expect it will help visitors to find relevant content.

Experiment goals and success criteria

The goal of this experiment is to increase the prominence and visibility of the Related Questions module. We believe that by moving the module higher up on question pages, users are more likely to discover relevant content that could help address their just-in-time needs. The assumption is that anonymous users may find more value in related questions compared to the Community Bulletin. For this experiment, we will be measuring the clickthrough rate (CTR) of the module. Similarly, we will monitor the clicks to the Community Bulletin to determine the impact of moving it further down the page.

How we’re conducting the test

This experiment should not affect any logged-in users and will only run on stackoverflow.com. It will target 1% of the traffic that will be split evenly between the control and variant groups. 50% of anonymous users will see the current sidebar where the first module displayed is the Community Bulletin, which includes links to the Stack Overflow Blog, Featured on Meta posts and Hot Meta Posts. The other 50% will see the Related Questions module displayed first. No changes have been made to how the questions are selected in the module.

The experiment will run for at least three days. For both groups, we will be measuring the click-through rate (CTR) on Related Questions and the Community Bulletin. 

Here is a mockup of the experience for anonymous users who are bucketed in the variant group:

reorder experiment mockup

Once the experiment concludes, we will analyze the results and share with Meta.

18
  • 33
    What about Linked questions, are they also moved up? I often find those at least as relevant as the Related questions.
    – Marijn
    Oct 25, 2022 at 21:25
  • 36
    @Marijn They are almost always more relevant than Related Questions...
    – TylerH
    Oct 25, 2022 at 21:27
  • 3
    Since the related questions search is so bad, you should collect data to determine how long it takes any given user's increased use of the related questions sidebar to be extinguished.
    – philipxy
    Oct 26, 2022 at 0:36
  • 2
    As a logged-in user (or well, a mod on another site), I'd prefer if the Related Questions can be placed above the ad...
    – Andrew T.
    Oct 26, 2022 at 2:50
  • 3
    A lifetime on the internet has trained me to ignore the right sidebar of any website as advertisement space. Expectations and learned behavior aside, you actually do put ads there...
    – canon
    Oct 26, 2022 at 13:37
  • 8
    @Marijn, linked questions are not moving up as part of this experiment but this is great feedback for a potential future experiment if the community finds this more valuable because they are links to other questions in comments and answers. Thank you for the feedback!
    – tanj92 Staff
    Oct 26, 2022 at 14:24
  • 1
    @philipxy can you elaborate a bit more about your experience with using the related questions module?
    – tanj92 Staff
    Oct 26, 2022 at 14:26
  • 2
    @TylerH can you elaborate on why you think they are more relevant than Related Questions? Do the linked questions help you get to your solution faster because someone else solved a similar problem and directed you to a post that contained a similar answer?
    – tanj92 Staff
    Oct 26, 2022 at 20:22
  • 10
    @tanj92 Because someone intentionally linked to them. Usually when a human says "this is related", it is more accurate/useful info than whatever the site's automated processes say is related. That's just true in general, but especially in SO's case, where the auto-generated "similar" questions are almost entirely unrelated... often the only relation is the language, and sometimes even that relationship is missing.
    – TylerH
    Oct 26, 2022 at 20:56
  • 4
    @tanj92 Take for example, this random recently active post I just clicked on. The question is about TCP/SMTP in C#. Now look at the auto-generated 'related' questions list... Java and SSL is the top result on there. Compared to the linked list... which list would you say contains more relevant/useful info for readers of this question?
    – TylerH
    Oct 26, 2022 at 20:59
  • 7
    @tanj92 The one place where "linked questions" would actually really benefit is splitting it out or delineating between "this is linked from somewhere on this page and might be useful to you" and "this is another question that links from itself to the one you're reading now, e.g. a reference to this page or a case where this question was used as a duplicate". I know I'm getting rather into the weeds about a different feature than what this post is supposed to be about, but, since you asked...
    – TylerH
    Oct 26, 2022 at 21:01
  • 3
    Thanks @TylerH super helpful context. I'll take this into consider for a future experiment and how the data from this experiment compares to the linked questions module.
    – tanj92 Staff
    Oct 26, 2022 at 21:22
  • 2
    I was actually thinking of suggesting to get rid of the Related Questions sideBar altogether. They are usually of so little relevance that the section should be burminated. Is there any telemetry on how often they are clicked on?
    – Mark
    Oct 28, 2022 at 18:03
  • 2
    @Mark I normally look at clickthrough rate (CTR) in terms of clicks / impressions (page views). For last month, related questions had a CTR of 0.07%, while Linked had a CTR of 0.05%.
    – tanj92 Staff
    Oct 28, 2022 at 21:59
  • 3
    @tanj92 Wow, I would have thought that the Linked CTR would have been higher, but am not surprised by the CTR of Related. I just looked through about 10 questions and Related links in my area of knowledge and there were only 4 Related links on two questions that sounded on topic (and they were like 5th and 6th in the list of ten so that is troublesome). So that was 4 helpful links out of 100. (I wonder if higher scoring answers are being over-weighted in the results?) I frequently look at the Related list and the first 3 or 4 are so far off-topic I don't look further.
    – Mark
    Oct 29, 2022 at 1:37

2 Answers 2

10

What will the position be if a question is part of a collective? There is a "collective" banner on the top of the sidebar, will that remain in place, or will it also shift down (I'm not having any strong opinion on what's preferable here).

Besides that, I have a fairly wide screen. If I'd had to guess only about 1/4th to 1/3rd of the page is used for the actual content I would be interested in, if I were here to find an answer. So there is plenty of room to make that side bar wider, so that the linked questions would show up, at least partially (when you serve only 1 add in the sidebar). Let alone make more use of the actual real-estate here and serve the question in a broader form.

1
  • 7
    Great question - I was able to confirm in a dev environment that the Collective banner will remain in it's current location in the variant group.
    – tanj92 Staff
    Oct 26, 2022 at 14:22
6

Update Nov 22: V2 experiment has concluded. We will share results and next steps soon.

When?

2
  • 1
    Presumably sometime between 2022-11-22 and the heat death of the universe (although I guess we can discount a few of the possible dates by now) ;)
    – Dan Mašek
    Mar 24 at 2:32
  • That's why one should not trust statements about the future. "We will ..." may as well mean "We won't ..." and adding "soon" at the end doesn't add useful information. The correct wording would be "We might ...." Mar 27 at 6:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .