45

In this post, we'll go over how the Trending sort experiment went and why we chose Decay-50 as our preferred choice as announced in Trending: A new answer sorting option. We'll summarize both the objective metrics we measured through our experiment data as well as the response to the survey we prompted during the first time you used Trending sort. You can see the definition of the experiment in A/B testing of a "Trending" sort option for answers.

How the algorithms work

The experiment featured these four Trending algorithms:

All four Trending algorithms from the experiment are shown to demonstrate the Trending score of a single vote by algorithm strength. All algorithms start at a score of 1.0 on the first day, but decay quickly towards 0.0 as days pass. Decay-50 halves its value each year. Decay-82 becomes 1/32 of its original value every two years. Decay-97 becomes 1/32 of its original value each year. Decay-100 becomes 1/32 of its original value every six months.

Algorithm Decay Function
Decay-100 decay(ageInDays) = (1/32) ^ (ageInDays/180)
Decay-97 decay(ageInDays) = (1/32) ^ (ageInDays/365)
Decay-82 decay(ageInDays) = (1/32) ^ (ageInDays/760)
Decay-50 decay(ageInDays) = (1/2) ^ (ageInDays/365)

Experiment results

Through a combination of several metrics we measured during the test, we found that Decay-50 outperformed all other options. We analyzed this change similarly to the Unpinning experiment. The specific metrics we tracked were:

  • A post was copied from or voted on
  • An answer was copied from
  • The first answer was copied from
  • Users upvoted on any answer.

We did an A/B test on the four decay algorithms compared to a baseline of score sort. The tables below are results from copies and votes under various conditions, measuring the lift compared to Score sort.

Copies or votes on all posts

We measured whether or not a post got a copy or vote event. Decay-50 saw a 4.46% lift compared to the baseline Score sort.

Algorithm p-value Lift Confidence Interval
Decay-100 <0.0001 1.98% [1.89%, 2.07%]
Decay-97 <0.0001 2.14% [2.05%, 2.23%]
Decay-82 <0.0001 3.06% [2.98%, 3.15%]
Decay-50 <0.0001 4.46% [4.37%, 4.55%]

Copies on an answer

This funnel measured when a copy occurred on an answer anywhere in the post. Decay-50 saw a 4.88% lift compared to baseline.

Algorithm p-value Lift Confidence Interval
Decay-100 0.12 0.10% [-0.03%, 0.23%]
Decay-97 <0.0001 1.39% [1.26%, 1.52%]
Decay-82 <0.0001 3.17% [3.04%, 3.31%]
Decay-50 <0.0001 4.88% [4.74%, 5.01%]

Copies on the first shown answer

We know that many users value the first answer shown on the page (part of the reason we are creating Trending Sort!). This funnel measures copies on that first shown answer only. The lift compared to baseline is significantly lower than copies overall, but we still saw a significant lift across all four algorithms.

Algorithm p-value Lift Confidence Interval
Decay-100 <0.0001 0.16% [0.12%, 0.19%]
Decay-97 0.034 0.04% [0.00%, 0.08%]
Decay-82 <0.0001 0.18% [0.14%, 0.21%]
Decay-50 <0.0001 0.14% [0.10%, 0.18%]

Position of Answer When Copied - % of Population by Algorithm

Much like the copies on the first shown answer, copies beyond position one are also important for sort. Most will copy the answer in position one, and that is true regardless of the algorithm.

This table looks at the percent of all copy events based on the position on the page. 55.8% of all score sort copies happen in position one, and we see a similar percentage across most of the other algorithms. That distribution to position two and beyond also looks similar between the algorithms.

Algorithm Position 1 Position 2 Position 3+
Baseline 55.8% 20.2% 24.0%
Decay-100 54.6% 21.6% 23.8%
Decay-97 55.2% 21.2% 23.5%
Decay-82 55.6% 20.8% 23.6%
Decay-50 55.7% 20.1% 24.2%

Upvotes on an answer

Upvotes are more rare than copies. We measured them as part of this experiment, but didn’t find any significant lifts in either direction compared to score sort.

Algorithm p-value Lift Confidence Interval
Decay-100 0.67 -0.16% [-0.91%, 0.58%]
Decay-97 0.36 0.35% [-0.40%, 1.10%]
Decay-82 0.19 -0.51% [-1.28%, 0.26%]
Decay-50 0.14 0.59% [-0.19%, 1.37%]

Copies on Answers Over Time

In the graph below we look at the copies for each sort over time, based on the age of the answer in months. Newer answers tended to benefit from more aggressive decay algorithms. As we approach the five year mark, we noticed that older answers are indistinguishable from baseline.

A line graph showing the copy events on answers over time, segmented by algorithm. The age is based on the answer’s age. Decay algorithms outpace the baseline score sort for answers younger than five years old. Copies older than five years old are similar across all algorithms and baseline, and trend downward the older the answer is.

The drop in performance for 5 year old answers lined up with the condition we had that excluded votes older than 5 years old. For this reason, we are not excluding any old votes in the version of Trending sort we launched.

Survey Data

The first time you viewed a Trending question, you were able to respond to a survey on the given sort's effectiveness. We collected responses for both the baseline of Score as well as each Trending sort variant. It's difficult to summarize all this subjective data, however in general Decay-50 was subjectively the best option.

We also want to recognize everyone who participated in the survey. We received over 11,000 survey responses and we want to sincerely thank you for participating - your feedback helped us choose the best candidate. (11,000 survey responses is a lot!)

Demographics

Reputation Percentage of population
Anonymous (no reputation) 33.8%
Less than 10 13.7%
10 - 124 14.6%
125 - 1,999 22.3%
2,000 - 9,999 9.9%
10,000+ 5.7%

Best Answer

The survey prompt was "Are the answers sorted in a way that puts the best answer at or near the top?". Decay-50 did not have a significant difference from the baseline of Score and the more aggressive algorithms got progressively worse.

Algorithm Lift
Decay-100 -9.0%
Decay-97 -3.4%
Decay-82 -2.1%
Decay-50 -0.6%

Effectiveness

We asked about the effectiveness in a few ways:

  • Why is the answer sorting effective?
  • Why is the answer sorting not effective?
  • Based on the question you were just viewing, what do you like or dislike about how the answers are sorted?

Score performed as we expected. When it works, people find that highly upvoted or accepted and/or accepted answers have good quality and can be trusted. When it doesn't work, the top answer was outdated, low quality, or a better answer was found further down the page.

Decay-50 performed similarly to Score and tended to perform better than the other algorithms. When it worked, it surfaced newer answers at the top, saved time on finding the best answer, and was friendly to newer users since they didn't have to read more answers for relevance, and had good answer quality. When it didn't work, it was difficult to tell why answers were sorted the way they were, they had trouble finding the accepted or highest score answers, and they had to compare answers to find the best one.

The other more aggressive algorithms were successful at putting new and relevant answers at the top and made it easier for newer answers to get a chance to rise to the top. However, when they didn't work, they would remove the best answer from the top and would put low quality answers before the best answer. Old high quality answers were being listed at the bottom.

Conclusion

All data pointed towards Decay-50 being the best candidate to ship for our launch of Trending sort. It did not perform significantly worse than our Score sort and outperformed all other algorithms. We are confident that this option will be successful and will tend to put relevant answers at the top.

4
  • 1
    @DalijaPrasnikar That table is comparing each of the four Trending variants to the baseline of Score sort, which sorts by the highest voted answers. Maybe I'm misunderstanding - are we missing a set of data you'd like to see?
    – Kyle Pollard StaffMod
    Commented Jun 17, 2022 at 7:05
  • 12
    Very interesting work, thanks for sharing the data. It's a pity it solely relies on copies, a dirty metric since the problems I run into nearly never have me copy something from an answer, but look at the underlying problem and solution. However, finding a good metric for ordering quality (without surveying users directly) is hard. I hope we're not optimizing for answers that dump code instead of properly explaining, it is worrisome that all algorithms did worse than score when surveying users.
    – Erik A
    Commented Jun 17, 2022 at 11:24
  • 2
    For the p-values, what is the null hypothesis? Also, what is the confidence level for the given intervals? (I am no statistician, I am just curious about those)
    – Didier L
    Commented Jun 17, 2022 at 12:18
  • Given that the best results came from the furthest-out value, are you considering another experiment with a farther-out value, such as decay-25? The current decay-50 gives a vote a half-life of barely more than a year, whereas I would estimate that the first ~3 years it's unlikely to be outdated (the answer might still be bad, but not no longer work so age is not a factor here). If I am correct and I'm eyeballing the graph right, somewhere around decay-18 should be the sweet spot, i.e. putting the best answers on top for googlers landing on your pages.
    – Luc
    Commented Aug 24, 2022 at 22:33

2 Answers 2

24

Thanks for the analysis. The chosen final decay value of 50 is at the border of the investigated interval (50-100), so it might be that non-investigated values below 50 (say Decay-30) perform even better. However, at some point there would not be enough difference to basic score sort to warrant another sort option, so I think that this is the absolutely right choice and people can now personally try them both out and see what works better for them. Looking forward to it.

The observed differences in behavior seems to be relatively small overall except for copying of code, indicating that in most cases maybe there aren't really many newer, more up-to-date, better answers and people do instead edit old, popular answers. Copying as a metric in itself is not ideal, because many good answers do not need to offer code for copying. More copying doesn't necessarily equal higher quality of the content.

In any case, it underscores that voting is not a magic bullet to define content quality. It's still the best available thing but far from perfect and weighting of votes can in some cases bring improvement and not so in others.

For further fine-tuning, maybe concentrate on answers with very low number of votes, where Decay-50 might increase noise (due to small numbers) in the ordering and maybe trending sort should be more conservative there and on answers with high number of votes, where a more aggressive decay might help surface updated content better without increasing variance of the sort order too much.

Finally, would it be possible to separate the existing results by major tags and look for differences? Some technologies might change faster than others and so optimal decay rates might be different for different tags.

However, since voting didn't really significantly change, maybe the whole problem with outdated answers wasn't so big after all either because the top most scored answers are most of the time the best (with notable exceptions) or because people do actually scroll down and do compare with newer answers.

1
  • 2
    On questions with a good long answer that explains the concepts but doesn't show fully worked-out concrete examples in code, it's not rare to see later answers that are mostly code implementing the same idea. Those do have some value for future readers, especially ones very new to a language overall. For such answers to get any / many upvotes, they usually need to not be total clutter (by the standards of experienced users in the tags), so this decay weighting won't sort them too high. But that's just one specific type of Q&A. Commented Jun 18, 2022 at 2:56
7

Thanks for releasing this analysis.

There's something in this data that seems really strange: copies overall vs copies from first position.

If you look at the data under the "Copies on an answer" section, the magnitude of the change vs baseline is around 1-5%. If you look at the data under the "Copies on the first shown answer" section, the magnitude of the change is around 0.1-0.2%. This is the opposite of what I'd expect. I would expect the change in copies for the first answer to be bigger than the change in copies overall. After all, the goal of changing the sort order is to put a helpful answer closer to the top.

Do you know what is driving the change in copies if not copies from the first position? I see the table under "Position of Answer When Copied," but this doesn't really clarify it - this is the position of the copy assuming that a copy was made. Could we see a table of how Decay-50 did relative to baseline, broken down by answer position?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .