In this post, we'll go over how the Trending sort experiment went and why we chose Decay-50 as our preferred choice as announced in Trending: A new answer sorting option. We'll summarize both the objective metrics we measured through our experiment data as well as the response to the survey we prompted during the first time you used Trending sort. You can see the definition of the experiment in A/B testing of a "Trending" sort option for answers.
How the algorithms work
The experiment featured these four Trending algorithms:
Algorithm | Decay Function |
---|---|
Decay-100 | decay(ageInDays) = (1/32) ^ (ageInDays/180) |
Decay-97 | decay(ageInDays) = (1/32) ^ (ageInDays/365) |
Decay-82 | decay(ageInDays) = (1/32) ^ (ageInDays/760) |
Decay-50 | decay(ageInDays) = (1/2) ^ (ageInDays/365) |
Experiment results
Through a combination of several metrics we measured during the test, we found that Decay-50 outperformed all other options. We analyzed this change similarly to the Unpinning experiment. The specific metrics we tracked were:
- A post was copied from or voted on
- An answer was copied from
- The first answer was copied from
- Users upvoted on any answer.
We did an A/B test on the four decay algorithms compared to a baseline of score sort. The tables below are results from copies and votes under various conditions, measuring the lift compared to Score sort.
Copies or votes on all posts
We measured whether or not a post got a copy or vote event. Decay-50 saw a 4.46% lift compared to the baseline Score sort.
Algorithm | p-value | Lift | Confidence Interval |
---|---|---|---|
Decay-100 | <0.0001 | 1.98% | [1.89%, 2.07%] |
Decay-97 | <0.0001 | 2.14% | [2.05%, 2.23%] |
Decay-82 | <0.0001 | 3.06% | [2.98%, 3.15%] |
Decay-50 | <0.0001 | 4.46% | [4.37%, 4.55%] |
Copies on an answer
This funnel measured when a copy occurred on an answer anywhere in the post. Decay-50 saw a 4.88% lift compared to baseline.
Algorithm | p-value | Lift | Confidence Interval |
---|---|---|---|
Decay-100 | 0.12 | 0.10% | [-0.03%, 0.23%] |
Decay-97 | <0.0001 | 1.39% | [1.26%, 1.52%] |
Decay-82 | <0.0001 | 3.17% | [3.04%, 3.31%] |
Decay-50 | <0.0001 | 4.88% | [4.74%, 5.01%] |
Copies on the first shown answer
We know that many users value the first answer shown on the page (part of the reason we are creating Trending Sort!). This funnel measures copies on that first shown answer only. The lift compared to baseline is significantly lower than copies overall, but we still saw a significant lift across all four algorithms.
Algorithm | p-value | Lift | Confidence Interval |
---|---|---|---|
Decay-100 | <0.0001 | 0.16% | [0.12%, 0.19%] |
Decay-97 | 0.034 | 0.04% | [0.00%, 0.08%] |
Decay-82 | <0.0001 | 0.18% | [0.14%, 0.21%] |
Decay-50 | <0.0001 | 0.14% | [0.10%, 0.18%] |
Position of Answer When Copied - % of Population by Algorithm
Much like the copies on the first shown answer, copies beyond position one are also important for sort. Most will copy the answer in position one, and that is true regardless of the algorithm.
This table looks at the percent of all copy events based on the position on the page. 55.8% of all score sort copies happen in position one, and we see a similar percentage across most of the other algorithms. That distribution to position two and beyond also looks similar between the algorithms.
Algorithm | Position 1 | Position 2 | Position 3+ |
---|---|---|---|
Baseline | 55.8% | 20.2% | 24.0% |
Decay-100 | 54.6% | 21.6% | 23.8% |
Decay-97 | 55.2% | 21.2% | 23.5% |
Decay-82 | 55.6% | 20.8% | 23.6% |
Decay-50 | 55.7% | 20.1% | 24.2% |
Upvotes on an answer
Upvotes are more rare than copies. We measured them as part of this experiment, but didn’t find any significant lifts in either direction compared to score sort.
Algorithm | p-value | Lift | Confidence Interval |
---|---|---|---|
Decay-100 | 0.67 | -0.16% | [-0.91%, 0.58%] |
Decay-97 | 0.36 | 0.35% | [-0.40%, 1.10%] |
Decay-82 | 0.19 | -0.51% | [-1.28%, 0.26%] |
Decay-50 | 0.14 | 0.59% | [-0.19%, 1.37%] |
Copies on Answers Over Time
In the graph below we look at the copies for each sort over time, based on the age of the answer in months. Newer answers tended to benefit from more aggressive decay algorithms. As we approach the five year mark, we noticed that older answers are indistinguishable from baseline.
The drop in performance for 5 year old answers lined up with the condition we had that excluded votes older than 5 years old. For this reason, we are not excluding any old votes in the version of Trending sort we launched.
Survey Data
The first time you viewed a Trending question, you were able to respond to a survey on the given sort's effectiveness. We collected responses for both the baseline of Score as well as each Trending sort variant. It's difficult to summarize all this subjective data, however in general Decay-50 was subjectively the best option.
We also want to recognize everyone who participated in the survey. We received over 11,000 survey responses and we want to sincerely thank you for participating - your feedback helped us choose the best candidate. (11,000 survey responses is a lot!)
Demographics
Reputation | Percentage of population |
---|---|
Anonymous (no reputation) | 33.8% |
Less than 10 | 13.7% |
10 - 124 | 14.6% |
125 - 1,999 | 22.3% |
2,000 - 9,999 | 9.9% |
10,000+ | 5.7% |
Best Answer
The survey prompt was "Are the answers sorted in a way that puts the best answer at or near the top?". Decay-50 did not have a significant difference from the baseline of Score and the more aggressive algorithms got progressively worse.
Algorithm | Lift |
---|---|
Decay-100 | -9.0% |
Decay-97 | -3.4% |
Decay-82 | -2.1% |
Decay-50 | -0.6% |
Effectiveness
We asked about the effectiveness in a few ways:
- Why is the answer sorting effective?
- Why is the answer sorting not effective?
- Based on the question you were just viewing, what do you like or dislike about how the answers are sorted?
Score performed as we expected. When it works, people find that highly upvoted or accepted and/or accepted answers have good quality and can be trusted. When it doesn't work, the top answer was outdated, low quality, or a better answer was found further down the page.
Decay-50 performed similarly to Score and tended to perform better than the other algorithms. When it worked, it surfaced newer answers at the top, saved time on finding the best answer, and was friendly to newer users since they didn't have to read more answers for relevance, and had good answer quality. When it didn't work, it was difficult to tell why answers were sorted the way they were, they had trouble finding the accepted or highest score answers, and they had to compare answers to find the best one.
The other more aggressive algorithms were successful at putting new and relevant answers at the top and made it easier for newer answers to get a chance to rise to the top. However, when they didn't work, they would remove the best answer from the top and would put low quality answers before the best answer. Old high quality answers were being listed at the bottom.
Conclusion
All data pointed towards Decay-50 being the best candidate to ship for our launch of Trending sort. It did not perform significantly worse than our Score sort and outperformed all other algorithms. We are confident that this option will be successful and will tend to put relevant answers at the top.
decay-25
? The currentdecay-50
gives a vote a half-life of barely more than a year, whereas I would estimate that the first ~3 years it's unlikely to be outdated (the answer might still be bad, but not no longer work so age is not a factor here). If I am correct and I'm eyeballing the graph right, somewhere arounddecay-18
should be the sweet spot, i.e. putting the best answers on top for googlers landing on your pages.