This experiment has concluded. We've gathered and analyzed the results in Results of the Trending sort experiment.
As we’ve been moving forward on the Outdated Answers project, one of the things that’s been clear is that on questions with multiple answers, we need a way to surface newer answers that may be more current, while reducing the visibility of older answers that may no longer be correct or relevant. Over the past few months we’ve been working on developing an algorithm that does exactly that — a way of identifying answers that are “trending” — meaning that they’ve received more recent votes.
In January, we let you know that this project was coming soon, and you seemed excited about it and had a lot of ideas and questions. This post is here to answer those questions and give you a peek at the research we’ve done to get to this point, along with the variations we’ve tried and their respective pros and cons. I’ll also be discussing the algorithms we’re going to test and the timeline for testing.
What is the goal of this new sort feature?
One of the promises in our Tour is that great answers are voted up and will rise to the top of the list of answers. While this is certainly something that we see in short-term situations, as a post ages and collects more answers that’s not always the case. In some cases, there are so many votes on established answers that it’s likely impossible to expect a newer, better answer to ever become visible. The disparity is sometimes huge — the top answer has had a decade of visibility, while a newer answer has only had a few months. While we can encourage users to sort answers using something other than answer score — such as Newest (coming within the next few weeks) — a new answer doesn’t always mean it’s a good one.
We had previously found that the signal of an accepted answer was outdated. We saw great benefit from using score instead of the accepted answer that the original asker found most helpful. These scores are calculated using the years of votes on posts showing what the community finds most valuable. However, just like accepted answers, votes don't change much over time, and so the signal from those older votes may not be as relevant as a vote from today. Right now, a vote from the first day Stack Overflow was created carries the same signal as a vote from today, and especially for older or more popular posts, there aren't enough downvotes to counteract the years of upvotes — which effectively prevents newer and potentially more accurate answers from gaining traction and visibility.
But there is one piece of information that should usually be a good signal: recent votes. If the top answer starts getting downvoted while a newer answer seems to be preferred, that’s something we think makes sense to recognize and turn into a sort option. This is where the concept of trending votes came from initially. We want to use the data we have to help users find answers that have been identified to be useful… recently.
The goal is central to Stack Overflow: ensure that the best answers show up first so that they’re easy to find. By amplifying the score of votes cast more recently, we believe that people searching for answers will find more up-to-date answers higher up in the sort.
History of the discovery phase
We saw in our research that the concept of recency is important to developers and the technology industry. We found that users often find solutions that still work, but there is also a newer way that things are done that’s preferred or simpler. Users can't quickly find working solutions for technology and frameworks as they exist today. This led to the solution of a Trending sort that focused on exposing these kinds of answers.
When we proposed the idea of a Trending sort option in a survey study, the majority of respondents reported that they would be likely to use it. The rest of respondents thought that the current sorting options were sufficient because most questions only have a couple of answers. This pointed at Trending sort being an extra option — not a replacement for our existing sorts.
When we looked at the voting data on positively scored answers on Stack Overflow, our findings pointed to a Trending sort that occurred over long periods of time:
- Upvotes happen over long periods of time. While around a fifth of upvotes happen on the first day an answer is created, the majority of upvotes happen after the post has reached two years old.
- Downvotes behave similarly, with a quarter of downvotes happening on the first day an answer is created, and a third of them happening after the post has reached two years old.
- 45% of all votes on these answers have happened in the past five years.
What’s the process for identifying a good algorithm?
We want to find a descending mathematical function that takes an individual vote's age and decays the vote's value. This function should output a result between 0 and 1 inclusive. When the vote hasn't aged, it should have the full value of 1. When the vote has aged sufficiently, it should fall to a value of 0. The function should be continuous without any abrupt changes in values so that newer votes are always worth more than older votes.
We'll then apply this function to each vote on the answer to get each vote's decayed score. We'll then sum up these scores to come up with the answer's decayed score. This decayed score will be used to sort answers under the Trending sort.
Because these sorts would need to perform at scale, we need to make sure that these are quickly calculated or otherwise cached.
The process to find good algorithm candidates involved measuring their impact on the history of Stack Overflow answers. The initial algorithms we proposed affected a small minority of answers, with only 5% of top answers changing. We wanted this trending sort to have a more significant impact on the answer order.
We used the following metrics to compare algorithms to the Score sort:
- How many positively scoring answers would be sorted differently?
- How many top answers would be different?
- How many estimated views on these questions would this sort impact?
We also knew that we wanted to test several different algorithms in an A/B test, so we wanted to make sure that algorithms we proposed were diverse enough from each other and didn't always result in the same sort.
The Decay functions we're proposing
We tried many different functions and came up with four candidate decay functions out of this analysis. We've named them based on how strongly they decay votes. Here's a sample of how they perform on different time periods for an upvote:
Value of an upvote… | 50% Decay | 82% Decay | 97% Decay | 100% Decay |
---|---|---|---|---|
on the first day | 1.00000 | 1.00000 | 1.00000 | 1.00000 |
after the first month | 0.94462 | 0.86725 | 0.75212 | 0.56123 |
after the sixth month | 0.71047 | 0.42547 | 0.18102 | 0.03125 |
after the first year | 0.50000 | 0.17678 | 0.03125 | 0.00089 |
after the second year | 0.25000 | 0.03125 | 0.00098 | A small non-zero value |
after the third year | 0.12500 | 0.00552 | 0.00003 | A very small non-zero value |
after the fifth year | 0.00000 | 0.00000 | 0.00000 | 0.0000 |
Downvotes share the exact same decay formula, but they will start at -1 point and decay with the same decay function towards 0.
In these four algorithms, an answer can be assigned a decay score if it meets the following criteria:
- The answer is positively scored
- The answer is not deleted
- The answer has had votes in the past five years
- The parent question is positively scored
- The parent question has more than one answer
- The parent question is not deleted
- The parent question is at least a day old
This eligibility is subject to change in future iterations of the Trending sort.
Some additional implementation details on the Trending sort:
- We're only using votes that are less than five years old at the time of the sort. An answer with no votes or only votes older than five years has a Trending score of 0.
- Even though older votes are completely discarded here, answers that have been sorted to the top of the default sort have been more visible through their lifetime and have received a steady stream of votes. We expect that if an answer is still correct, it will still be getting enough recent votes to stay on top, even in a trending sort.
- We expect these algorithms to perform better on older questions with active voting histories. These algorithms may be inaccurate when there's a low number of votes over its lifetime. For example, questions that are only a few weeks old may not have aged enough for there to be a useful Trending sort. We'll look at this in our data analysis and may modify the eligibility criteria to account for this.
- Any answers that have the same Trending score will use the default Score sort behavior as a tiebreaker.
- We're using the usual upvotes and downvotes for our voting data, but we're not using the anonymous voting data. There's some technical limitations that prevent us from using this data. Additionally, anonymous votes don't have the same protections as our normal voting, so we're hesitant to use this signal for the Trending sort.
Here's how the different algorithms performed in our analysis on questions and answers that met this criteria:
Algorithm | Affected Positive Answers |
Affected Top Answers |
Estimated Affected Views |
Average Age of New Top Answers |
---|---|---|---|---|
50% Decay | 19.6% (0.98m) | 7.5% (182k) | 8.7% | 341 days newer |
82% Decay | 27.8% (1.39m) | 12.7% (307k) | 15.8% | 334 days newer |
97% Decay | 28.2% (1.41m) | 13.2% (321k) | 19.8% | 380 days newer |
100% Decay | 24.4% (1.22m) | 11.4% (277k) | 21.5% | 456 days newer |
There are some things to note about these results:
- Estimated views are calculated naively by taking the all-time views of the parent question that would be affected. There's a large bias towards older questions that have many years of views, but this is exactly the kind of focus we're looking for. We want to impact established and popular posts on the site.
- 100% Decay affects a smaller number of top answers, but a larger number of estimated views. It's the algorithm that impacts more popular answers on Stack Overflow, but doesn't affect less popular answers as much.
- The average age is just an average calculated with the SQL
AVG()
aggregate on all changed top answers. It's not a median and it doesn't include standard deviation. Each algorithm affects a different sample of top answers. For these reasons, these numbers aren't directly comparable, but provide a decent approximation.
What are we testing?
We have done an initial analysis of the impact the proposed algorithms would have on this site, but it's only an estimate. We know that these algorithms would impact the site, but we're not sure on the quality of their impact. We want to collect real user data on how these algorithms impact users finding answers on the site to make sure we're meeting our goals of putting the trending answers on top:
- We're going to measure how copy and voting behavior changes between the default Score sort and the four Trending sorts.
- We're going to look at these metrics on the top sorted answer, the highest scored answer, the accepted answer, and all other answers on each question.
- We're also watching the rate at which users change between the different Sort options.
Our user research also indicated that experienced users read many answers in order to determine which one works, while other users may be more likely to just take the top answers and/or the accepted answer if any of them work. While we're running this test, we're also going to look at a few other exploratory metrics to help us understand this behavior. These metrics may not impact our final choice of algorithm:
- We're going to attempt to measure the bias of accepted/top answers. These answers are more visible than other answers.
- We're also going to try measuring "answer view depth", which is a measure of how many answers you've viewed when you leave the page.
The subjective performance of these algorithms is also important. We'll be prompting some of you with a qualitative survey to help us understand how effective the sort was. You'll see this survey if you're part of the test and visit a question that would be impacted by these sorts, or if you're part of the baseline and use the default Score sort. You'll only be prompted to fill out this survey once.
How will the test work?
The goal of the test is to help us compare the performance of these four algorithms and identify the best candidate to use. We want to ship one of these four algorithms.
We are going to run several A/B tests on Stack Overflow that compare these different trending algorithms against the default highest-score sort. If (a) your current sort option is Score, (b) you're part of this test, and (c) the question you're viewing would have a different sort order under Trending, then you'll see "Trending (recent votes count more)" as your new selected sort option for that question. You'll receive any one of the four proposed algorithms, but you won't know at the time which one you're getting. You won't be able to see the Trending score of answers, and you'll continue to see its normal non-decayed score.
There is no way to opt-in to the test. If you're in the test, you can opt-out of the test at any time by changing to another sorting option, but you will not be able to choose between or compare different trending sort options. As always, if you've opted out of analytics with your cookie settings, we won't be collecting your data during this test.
One technical limitation of the test is that the decay scores will be cached aggressively for 24-36 hours. If you vote after viewing the page, you won't see an immediate change in the Trending sort order of the questions until the cached values expire and new decay scores are calculated. We don't expect this limitation to be there when we ship it.
We're planning on starting the test in 4-6 weeks. We expect to run the test for a couple of weeks until we get a sufficient sample size. When the test is over, you won't see Trending as an available option anymore until we launch it permanently in production.
The ask from the community
We ask that if you're part of the test and you see the Trending option, please try it out! It won't be perfect — any Trending algorithm we propose will be occasionally wrong; that's expected. What we want for this to tend to show better answers than the pure, score-based sort. This also means it's really important for you to help the sorting algorithm show better content by voting on the answers you see — either up or down.
If you see a bad answer show up at the top, please confirm that it really is a bad answer and then vote! Your downvote will help tell the algorithm that this question may not be as good as the existing votes show.
If you see the qualitative survey, please consider filling it out. This is the best place for you to provide feedback on how the answers you viewed were sorted. There's room for you to comment on how the sort worked for your specific case as well as provide detailed feedback on your experience. Note that we plan to only show you the survey once, so you won't be able to fill it out multiple times.
After the test is over, we'll spend some time analyzing the results. Just like we did for the unpinning experiment, we'll share the findings that come out of this test. We'll also share the exact details of each proposed decay function. Our plan is to permanently ship the single best algorithm, provided that it performs similarly or better than our default Score sort.
Right now, we're only running tests and implementing this on Stack Overflow. Just like the unpinning experiment, later we'll request some network-wide feedback on Meta Stack Exchange and identify some sites that the Trending sort could also be available on. The Trending sort requires sustained voting on answers over time, so it may not be useful or impactful on sites with lower voting engagement.
806, 5, 16, 39, 90, 6, 7, 1
. I understand that's the point, but it would be easier to know how much I feel I should trust the "trending" answers if you'd display their "trending" weighted scores, alongside, perhaps, their conventional upvote count.