Update
Our experiment incorporating machine learning (ML) to improve the relevance of related questions has concluded. Thank you to everyone who provided valuable feedback on that experiment. It is vital data that help us ensure we are on the right track with this initiative.
Results
This is the newest data that we received from the last experiment incorporating machine learning (ML) to improve the relevance of related questions:
- Related questions recommended from the Machine Learning (ML) model generated a whopping 155% increase in clickthrough rate (CTR) in the aggregate. Please note, the variant group includes both questions from the model and Elasticsearch because newer/recent questions are not factored in, thus they fall back to being recommended by Elasticsearch. However, the majority of the clicks in the variant were from questions recommended by the model.
What about other metrics?
Are users voting more?
- Yes. Of the users who are able to vote, there was a 23% lift in overall votes combined (upvotes and downvotes).
Are users commenting more?
- Yes, we saw a similar trend as well with a 26% lift.
What about voting or commenting attempts? (We defined attempts as users who aren’t able to vote due to being anonymous or lack reputation. This is used to measure intent)
- We saw a 68-70% lift in users who attempted to engage (vote/comment) on related question pages in the variant group.
What happened to our postponed plan for the second experiment?
We are graduating the "Related questions using Machine Learning" experiment (without doing a second experiment)
What happened to our original plan for the second experiment?
We have decided to postpone the next experiment. When we updated the model we found something that was not acting as expected and did not want to move forward until it was resolved. We will update the community soon about when we plan to reschedule.
What was our original plan after the first experiment?
Our next experiment will involve updating the sort order of related questions recommended by utilizing the machine learning (ML) model.
How it works is that questions recommended by the ML model will be sorted by similarity score first, which means the questions with the highest relevance will appear first in the list. Just like the last experiment, we calculate similarity or relevance by leveraging an embedding model using titles and tags which converts text into a numerical vector. We then measure of distance between the two to populate the results. In the previous experiment, the recommended questions appeared in a randomized order, so we hope that this will improve the reliability of the list of recommended questions.
We agree that clickthrough rate (CTR) is not enough of a metric to judge relevancy. In this experiment in addition to measuring CTR, we will also be measuring a number of new funnels including:
- Whether or not a user copied (i.e. code snippet) from the page
- Whether engagement attempts were made (i.e. anonymous users or registered users with or without enough reputation attempted to vote or comment)
- The amount of time spent on pages
We heard your feedback regarding including inline feedback during these experiments. While we will not be collecting that data with this experiment, we are working incorporating a prompt to collect qualitative feedback when users interact with related questions.
Just like the last experiment, we will be asking that you share your feedback on a linked question (link to be added soon) that we will title with the experiment name that we are currently running.
Thanks to your invaluable contributions, we have already started talks on ways to incorporate some of the ideas on how better to achieve relevancy, so please continue to offer your suggestions as much as possible.
Background
As part of our Content Discovery initiative, we've undertaken several experiments on Stack Overflow, and have several more planned. For each of those experiments, we’ve created a post here on meta that discusses the experiment and links back to the initiative.
While this simplifies things in the sense that each topic has a dedicated post, it also leaves feedback dispersed. Additionally, the central initiative post isn't serving as a single source for updates in the way that we had hoped.
In order to centralize feedback and provide a single source for sharing experiments and updates — we're rolling out this post as the new format for informing the community on this initiative.
Overview and Objective
Site satisfaction surveys have made it clear that discovering helpful content is a pain point for many visitors to Stack Overflow. Out of the 21,595 responses to the survey year to date, only 11% of participants mentioned that discoverability was one of the most valuable aspects of Stack Overflow. However, when asked “What would you most like to improve about using Stack Overflow?”, discoverability ranked third in responses, indicating that it’s important to prioritize.
The objective of these experiments, then, is to make finding content easier/more reliable/more successful for more users. Not only is "find an answer" the core service but finding quality related content means that we can drive learning and exploration.
Moving Forward
These experiments are largely focused on small changes over time. The current focus of our attention is bettering the Related Questions module, from moving where it appears on the page to improving the content shown there. We may also investigate other ways to recommend content to users, such as using browsing history to point users to pages that could be relevant to their searches.
To summarize the current focus of Content Discovery, we are:
- Continuing to iterate on related questions (i.e. improve relevance)
- Introducing user-to-content recommendation modules
Related Questions will go through several iterations, especially once we've launched the first machine learning experiment in the near future. Internal content recommendations (i.e. recommended for you because we think you like "python") will be another area of focus for experimentation that will be prioritized next quarter.
Moving forward, we'll announce new experiment releases as an update on this post and the table below, instead of a standalone Meta post for each experiment.
The Experiments
We’ll be updating the below chart when experiments have been released or have concluded. That way you’ll be able to easily see updates and next steps all in one location. Note that the experiments listed on this table are listed from newest to oldest experiment.
Summary | Start Date | End Date | Description | Measure of Success | Status |
---|---|---|---|---|---|
Updating the sort order of related questions utilizing the machine learning (ML) model | – | – | Using machine learning to improve the reliability of the list of recommended questions | Clickthrough rate (CTR), and other engagement data | Postponed |
Related questions using a Machine Learning model | March 29, 2023 | April 12, 2023 | Using machine learning to recommend more relevant related questions using the question title and tags | Clickthrough rate (CTR) | Concluded |
“Most asked in [Tag]” sidebar module | Mar 8, 2023 | Mar 16, 2023 | Displaying frequently visited/asked questions above Hot Network Questions | Clickthrough rate (CTR) | Concluded |
Related questions within answers list | Feb 14, 2023 | Feb 24, 2023 | Displaying related questions on pages without answers | Clickthrough rate (CTR) for Related Questions | Graduated |
Separating Overflow Blog and Community Bulletin | Jan 30, 2023 | Feb 9, 2023 | Two experiments were conducted for creating a dedicated space for blog content | Variant does not perform worse than control, CTR for Community Bulletin | Concluded |
Moving Related Questions higher on question pages | Oct 26, 2022 | Nov 22, 2022 | Two experiments were conducted to increase the prominence of related questions in the sidebar | Clickthrough rate (CTR) for Related Questions and Community Bulletin | Concluded |
Feedback
When new experiments are released or concluded we’ll update this central post with a call to action for the current community feedback we’re looking for. We ask that you share your feedback on a linked question (link to be added soon) that we will title with the experiment name that we are currently running. We are doing it this way so that we can more easily compartmentalize the feedback for each experiment and take action on the information you’re providing us with.
Our goal in shifting communication on this project is to keep you better informed and make it easier to surface your valuable feedback to Product and Engineering.
Ad Revenue
for transparencies sake.