158

This experiment has concluded. We've gathered and analyzed the results in Results of the Trending sort experiment.


As we’ve been moving forward on the Outdated Answers project, one of the things that’s been clear is that on questions with multiple answers, we need a way to surface newer answers that may be more current, while reducing the visibility of older answers that may no longer be correct or relevant. Over the past few months we’ve been working on developing an algorithm that does exactly that — a way of identifying answers that are “trending” — meaning that they’ve received more recent votes.

In January, we let you know that this project was coming soon, and you seemed excited about it and had a lot of ideas and questions. This post is here to answer those questions and give you a peek at the research we’ve done to get to this point, along with the variations we’ve tried and their respective pros and cons. I’ll also be discussing the algorithms we’re going to test and the timeline for testing.

What is the goal of this new sort feature?

One of the promises in our Tour is that great answers are voted up and will rise to the top of the list of answers. While this is certainly something that we see in short-term situations, as a post ages and collects more answers that’s not always the case. In some cases, there are so many votes on established answers that it’s likely impossible to expect a newer, better answer to ever become visible. The disparity is sometimes huge — the top answer has had a decade of visibility, while a newer answer has only had a few months. While we can encourage users to sort answers using something other than answer score — such as Newest (coming within the next few weeks) — a new answer doesn’t always mean it’s a good one.

We had previously found that the signal of an accepted answer was outdated. We saw great benefit from using score instead of the accepted answer that the original asker found most helpful. These scores are calculated using the years of votes on posts showing what the community finds most valuable. However, just like accepted answers, votes don't change much over time, and so the signal from those older votes may not be as relevant as a vote from today. Right now, a vote from the first day Stack Overflow was created carries the same signal as a vote from today, and especially for older or more popular posts, there aren't enough downvotes to counteract the years of upvotes — which effectively prevents newer and potentially more accurate answers from gaining traction and visibility.

But there is one piece of information that should usually be a good signal: recent votes. If the top answer starts getting downvoted while a newer answer seems to be preferred, that’s something we think makes sense to recognize and turn into a sort option. This is where the concept of trending votes came from initially. We want to use the data we have to help users find answers that have been identified to be useful… recently.

The goal is central to Stack Overflow: ensure that the best answers show up first so that they’re easy to find. By amplifying the score of votes cast more recently, we believe that people searching for answers will find more up-to-date answers higher up in the sort.

History of the discovery phase

We saw in our research that the concept of recency is important to developers and the technology industry. We found that users often find solutions that still work, but there is also a newer way that things are done that’s preferred or simpler. Users can't quickly find working solutions for technology and frameworks as they exist today. This led to the solution of a Trending sort that focused on exposing these kinds of answers.

When we proposed the idea of a Trending sort option in a survey study, the majority of respondents reported that they would be likely to use it. The rest of respondents thought that the current sorting options were sufficient because most questions only have a couple of answers. This pointed at Trending sort being an extra option — not a replacement for our existing sorts.

When we looked at the voting data on positively scored answers on Stack Overflow, our findings pointed to a Trending sort that occurred over long periods of time:

  • Upvotes happen over long periods of time. While around a fifth of upvotes happen on the first day an answer is created, the majority of upvotes happen after the post has reached two years old.
  • Downvotes behave similarly, with a quarter of downvotes happening on the first day an answer is created, and a third of them happening after the post has reached two years old.
  • 45% of all votes on these answers have happened in the past five years.

What’s the process for identifying a good algorithm?

We want to find a descending mathematical function that takes an individual vote's age and decays the vote's value. This function should output a result between 0 and 1 inclusive. When the vote hasn't aged, it should have the full value of 1. When the vote has aged sufficiently, it should fall to a value of 0. The function should be continuous without any abrupt changes in values so that newer votes are always worth more than older votes.

We'll then apply this function to each vote on the answer to get each vote's decayed score. We'll then sum up these scores to come up with the answer's decayed score. This decayed score will be used to sort answers under the Trending sort.

Because these sorts would need to perform at scale, we need to make sure that these are quickly calculated or otherwise cached.

The process to find good algorithm candidates involved measuring their impact on the history of Stack Overflow answers. The initial algorithms we proposed affected a small minority of answers, with only 5% of top answers changing. We wanted this trending sort to have a more significant impact on the answer order.

We used the following metrics to compare algorithms to the Score sort:

  • How many positively scoring answers would be sorted differently?
  • How many top answers would be different?
  • How many estimated views on these questions would this sort impact?

We also knew that we wanted to test several different algorithms in an A/B test, so we wanted to make sure that algorithms we proposed were diverse enough from each other and didn't always result in the same sort.

The Decay functions we're proposing

We tried many different functions and came up with four candidate decay functions out of this analysis. We've named them based on how strongly they decay votes. Here's a sample of how they perform on different time periods for an upvote:

Value of an upvote… 50% Decay 82% Decay 97% Decay 100% Decay
on the first day 1.00000 1.00000 1.00000 1.00000
after the first month 0.94462 0.86725 0.75212 0.56123
after the sixth month 0.71047 0.42547 0.18102 0.03125
after the first year 0.50000 0.17678 0.03125 0.00089
after the second year 0.25000 0.03125 0.00098 A small non-zero value
after the third year 0.12500 0.00552 0.00003 A very small non-zero value
after the fifth year 0.00000 0.00000 0.00000 0.0000

Downvotes share the exact same decay formula, but they will start at -1 point and decay with the same decay function towards 0.

In these four algorithms, an answer can be assigned a decay score if it meets the following criteria:

  • The answer is positively scored
  • The answer is not deleted
  • The answer has had votes in the past five years
  • The parent question is positively scored
  • The parent question has more than one answer
  • The parent question is not deleted
  • The parent question is at least a day old

This eligibility is subject to change in future iterations of the Trending sort.

Some additional implementation details on the Trending sort:

  • We're only using votes that are less than five years old at the time of the sort. An answer with no votes or only votes older than five years has a Trending score of 0.
  • Even though older votes are completely discarded here, answers that have been sorted to the top of the default sort have been more visible through their lifetime and have received a steady stream of votes. We expect that if an answer is still correct, it will still be getting enough recent votes to stay on top, even in a trending sort.
  • We expect these algorithms to perform better on older questions with active voting histories. These algorithms may be inaccurate when there's a low number of votes over its lifetime. For example, questions that are only a few weeks old may not have aged enough for there to be a useful Trending sort. We'll look at this in our data analysis and may modify the eligibility criteria to account for this.
  • Any answers that have the same Trending score will use the default Score sort behavior as a tiebreaker.
  • We're using the usual upvotes and downvotes for our voting data, but we're not using the anonymous voting data. There's some technical limitations that prevent us from using this data. Additionally, anonymous votes don't have the same protections as our normal voting, so we're hesitant to use this signal for the Trending sort.

Here's how the different algorithms performed in our analysis on questions and answers that met this criteria:

Algorithm Affected
Positive Answers
Affected
Top Answers
Estimated
Affected Views
Average Age of
New Top Answers
50% Decay 19.6% (0.98m) 7.5% (182k) 8.7% 341 days newer
82% Decay 27.8% (1.39m) 12.7% (307k) 15.8% 334 days newer
97% Decay 28.2% (1.41m) 13.2% (321k) 19.8% 380 days newer
100% Decay 24.4% (1.22m) 11.4% (277k) 21.5% 456 days newer

There are some things to note about these results:

  • Estimated views are calculated naively by taking the all-time views of the parent question that would be affected. There's a large bias towards older questions that have many years of views, but this is exactly the kind of focus we're looking for. We want to impact established and popular posts on the site.
  • 100% Decay affects a smaller number of top answers, but a larger number of estimated views. It's the algorithm that impacts more popular answers on Stack Overflow, but doesn't affect less popular answers as much.
  • The average age is just an average calculated with the SQL AVG() aggregate on all changed top answers. It's not a median and it doesn't include standard deviation. Each algorithm affects a different sample of top answers. For these reasons, these numbers aren't directly comparable, but provide a decent approximation.

What are we testing?

We have done an initial analysis of the impact the proposed algorithms would have on this site, but it's only an estimate. We know that these algorithms would impact the site, but we're not sure on the quality of their impact. We want to collect real user data on how these algorithms impact users finding answers on the site to make sure we're meeting our goals of putting the trending answers on top:

  • We're going to measure how copy and voting behavior changes between the default Score sort and the four Trending sorts.
  • We're going to look at these metrics on the top sorted answer, the highest scored answer, the accepted answer, and all other answers on each question.
  • We're also watching the rate at which users change between the different Sort options.

Our user research also indicated that experienced users read many answers in order to determine which one works, while other users may be more likely to just take the top answers and/or the accepted answer if any of them work. While we're running this test, we're also going to look at a few other exploratory metrics to help us understand this behavior. These metrics may not impact our final choice of algorithm:

  • We're going to attempt to measure the bias of accepted/top answers. These answers are more visible than other answers.
  • We're also going to try measuring "answer view depth", which is a measure of how many answers you've viewed when you leave the page.

The subjective performance of these algorithms is also important. We'll be prompting some of you with a qualitative survey to help us understand how effective the sort was. You'll see this survey if you're part of the test and visit a question that would be impacted by these sorts, or if you're part of the baseline and use the default Score sort. You'll only be prompted to fill out this survey once.

How will the test work?

The goal of the test is to help us compare the performance of these four algorithms and identify the best candidate to use. We want to ship one of these four algorithms.

We are going to run several A/B tests on Stack Overflow that compare these different trending algorithms against the default highest-score sort. If (a) your current sort option is Score, (b) you're part of this test, and (c) the question you're viewing would have a different sort order under Trending, then you'll see "Trending (recent votes count more)" as your new selected sort option for that question. You'll receive any one of the four proposed algorithms, but you won't know at the time which one you're getting. You won't be able to see the Trending score of answers, and you'll continue to see its normal non-decayed score.

There is no way to opt-in to the test. If you're in the test, you can opt-out of the test at any time by changing to another sorting option, but you will not be able to choose between or compare different trending sort options. As always, if you've opted out of analytics with your cookie settings, we won't be collecting your data during this test.

One technical limitation of the test is that the decay scores will be cached aggressively for 24-36 hours. If you vote after viewing the page, you won't see an immediate change in the Trending sort order of the questions until the cached values expire and new decay scores are calculated. We don't expect this limitation to be there when we ship it.

We're planning on starting the test in 4-6 weeks. We expect to run the test for a couple of weeks until we get a sufficient sample size. When the test is over, you won't see Trending as an available option anymore until we launch it permanently in production.

The ask from the community

We ask that if you're part of the test and you see the Trending option, please try it out! It won't be perfect — any Trending algorithm we propose will be occasionally wrong; that's expected. What we want for this to tend to show better answers than the pure, score-based sort. This also means it's really important for you to help the sorting algorithm show better content by voting on the answers you see — either up or down.

If you see a bad answer show up at the top, please confirm that it really is a bad answer and then vote! Your downvote will help tell the algorithm that this question may not be as good as the existing votes show.

If you see the qualitative survey, please consider filling it out. This is the best place for you to provide feedback on how the answers you viewed were sorted. There's room for you to comment on how the sort worked for your specific case as well as provide detailed feedback on your experience. Note that we plan to only show you the survey once, so you won't be able to fill it out multiple times.

After the test is over, we'll spend some time analyzing the results. Just like we did for the unpinning experiment, we'll share the findings that come out of this test. We'll also share the exact details of each proposed decay function. Our plan is to permanently ship the single best algorithm, provided that it performs similarly or better than our default Score sort.

Right now, we're only running tests and implementing this on Stack Overflow. Just like the unpinning experiment, later we'll request some network-wide feedback on Meta Stack Exchange and identify some sites that the Trending sort could also be available on. The Trending sort requires sustained voting on answers over time, so it may not be useful or impactful on sites with lower voting engagement.

39
  • 92
    I occasionally find myself at a question I already upvoted, years prior. I know this might not be a high priority item, but have you considered having some method of re-upvoting an old answer that I found useful again?
    – Cullub
    Commented Mar 9, 2022 at 20:28
  • 24
    @Cullub It's something that came up in our discovery. I also experience finding old questions I already upvoted and agree that having a way to refresh upvotes could be useful. For now, we're not prioritizing this functionality and we're keeping the existing functionality around voting.
    – Kyle Pollard StaffMod
    Commented Mar 9, 2022 at 20:37
  • 10
    @KevinB Correct, the Trending experiment is at least a few weeks away. The dropdown design was in a separate A/B test that ended sometime earlier today. The dropdown data will be analyzed separately and I expect we'll have made a decision by the time we launch the Trending test. I'll defer discussion about the dropdown to its announcement post.
    – Kyle Pollard StaffMod
    Commented Mar 9, 2022 at 20:55
  • 27
    Astounding work! This is what (and I hope I do not speak only for myself here) large projects affecting the community at large should preferrably look like. Pretty much nothing else to say - eagerly awaiting the test and the results afterwards.
    – 0Valt
    Commented Mar 9, 2022 at 21:24
  • 29
    I'm not sure having actual zeroes in the decay algorithm would be a good idea, I'd prefer to still have a very, very small value, so if two answers scored equally recently, but one has more old votes, that still counts for something. Specifically for very-low activity old questions, the entire score of all answers may be ignored. It's odd that all options have a zero at 5 years, so no differences are evaluated for decay at that timepoint in this test. Is there a technical reason I'm missing?
    – Erik A
    Commented Mar 10, 2022 at 10:28
  • 9
    @ErikA It's already built-in "Any answers that have the same Trending score will use the default Score sort behavior as a tiebreaker.". But just using the same exponential relation for all time intervals would also work to solve the problem. Commented Mar 10, 2022 at 12:11
  • 8
    @ErikA My first implementation used all-time votes for exactly that case. I agree that the comparison of Trending scores would order all answers by recent votes. However, an answer with 5 year old votes beating an answer with 6 year old votes doesn't feel like the winning answer is Trending, just that it's less old. Here, we'd prefer to use a limited time span of votes and picked the 5 year range with some data analysis. (This does also help with performance, but that wasn't why we chose to do this)
    – Kyle Pollard StaffMod
    Commented Mar 10, 2022 at 19:47
  • 24
    It says “Downvotes share the exact same decay formula” Why though? A bad (or even outdated) approach does not become better over time, quite the contrary. Having downvotes decay to 0 makes no sense.
    – Bolpat
    Commented Mar 21, 2022 at 10:19
  • 8
    After visiting some questions with trending sort, I can only say this is a mess. This sorting order does not help in finding better answers and in some cases it only pushes up newer meh or me too answers that add nothing new - for instance if answer with 4 upvotes and a downvote comes before answer with more than 50 upvotes, there is something very wrong here. In very rare cases where it actually brings more appropriate answer to the top it is not very hard to find that answer based on score alone. This will be either completely useless or will give emphasis to worse answers.
    – Dalija Prasnikar Mod
    Commented Apr 13, 2022 at 12:41
  • 5
    @DalijaPrasnikar I only saw a couple of questions with trending sort but I agree it's not exactly what I'd expect. The oldest, most complete, most upvoted answer was last. Apparently because it wasn't upvoted recently. The other answers which were above it weren't bad but also weren't as complete as that one.
    – VLAZ
    Commented Apr 13, 2022 at 13:35
  • 3
    I took a cursory glance over some Javascript algorithm questions yesterday, and really liked the trending results! I like how it brings newer, fresher solutions higher up, especially on questions where older, pre-ES5/6 solutions have long dominated the upper slots even when newer, more succinct, more efficient solutions are available. This sort brings those newer answers further up in the ranking, while still keeping the long-standing high-quality answers at or near the top.
    – zcoop98
    Commented Apr 13, 2022 at 16:29
  • 9
    Whenever I visit a question that's A/B-enabled, that question is forcibly switched over to trending sort instead of my preferred sort method, even though I've changed it away from trending sort twice now. Whatever system is used is local to each question. Could you please either make an opt-out button, or make the sort system respect that I've switched away from trending sort globally? Resetting it on every enabled question I stumble into for 6 weeks straight is going to be annoying Commented Apr 14, 2022 at 18:26
  • 26
    Interesting, but good heavens the UI looks drunk now. In the question where I just got the survey, the votes are, in order of presentation: 806, 5, 16, 39, 90, 6, 7, 1. I understand that's the point, but it would be easier to know how much I feel I should trust the "trending" answers if you'd display their "trending" weighted scores, alongside, perhaps, their conventional upvote count.
    – ruffin
    Commented Apr 14, 2022 at 21:55
  • 6
    Correction re: my previous comment: switching away from trending isn't just per question, it's per tab. The disabling is written as a part of the URL, and if removed, switches back to trending. Please make it possible to disable without going down a userscript path for 6 weeks Commented Apr 15, 2022 at 15:22
  • 5
    I feel the explanation of the algorithm could be clearer by mentioning that you use an exponential decay with different half lives - assuming that's what it is. You make something trivial (to a Physicist) sound complicated. Commented Apr 18, 2022 at 9:10

22 Answers 22

127

This still has the same issue — pointed out previously — of assuming that upvotes and downvotes have equal weight, when we know that this is manifestly not the case: in fact, downvotes weigh much heavier due to being used a lot less, and due to carrying a cost for the vote caster.

I know I keep harping on about this but please, please account for this in your “trending” algorithm. The current draft is simply based on an utterly false premise about how votes are cast. Even just multiplying downvotes by a fixed constant factor (>1) will almost certainly be an immediate improvement compared to any algorithm that weighs up- and downvotes equally.

61
  • 3
    I wonder how vote distribution over time factors into this. I would guess that most upvotes tend to be more clustered around the time of the post’s creation (especially for popular questions), whereas downvotes are more evenly distributed. If so, then perhaps this doesn’t need that much correcting after all, since the decay will affect upvotes more than it does downvotes.
    – dumbass
    Commented Mar 13, 2022 at 20:44
  • 2
    But wouldn't you need another independent measure of content quality to gauge the weight of up/downvotes? Some people might say there are sufficiently many downvotes given, others might say something else. Who can decide who is right? Commented Mar 14, 2022 at 11:13
  • 36
    @Trilarion As for “who can decide who is right”, I can off the top of my head point to several highly upvoted answers which are objectively wrong, and for which upvotes are skewed because the mistake is subtle, or because it’s a common misconception. These examples lead me to believe that downvotes should probably be weighted at least 5×, possibly as much as 10×, compared to upvotes. Commented Mar 14, 2022 at 11:38
  • 2
    I find it an interesting sentiment that people would rather allow old downvotes to outstrip the value of old upvotes.
    – jxh
    Commented Mar 15, 2022 at 19:10
  • 4
    @jxh If that’s how you interpret this answer let me tell you that you have completely misunderstood it. Commented Mar 15, 2022 at 19:16
  • 17
    I think the right approach is for downvotes by gold-badge holders (in any on the question) to never decay. Someone who's taken the time to earn a gold tag badge is already trusted to instantly close questions, is unlikely to be using shotgun downvotes, and generally knows when they see a fatally flawed answer that should never be followed.
    – Ben Voigt
    Commented Mar 15, 2022 at 20:51
  • 3
    @BenVoigt The badness was already accounted for in the original vote. If the post was changed, generating newer upvotes, then the downvotes also become less relevant.
    – jxh
    Commented Mar 15, 2022 at 21:09
  • 5
    @jxh: Also, "downvoters rarely change their vote, even if the post is improved to address any reasonable criticism" is often intentional. I don't reward a FGITW user for copying from the correct answers into his.
    – Ben Voigt
    Commented Mar 15, 2022 at 21:15
  • 5
    @jxh My answer isn’t about “old” votes, that seems to be a misconception. All votes should be weighted, in addition to (or even instead of) decaying. In a way my answer is actually completely orthogonal to the issue of decay over time. Commented Mar 15, 2022 at 21:27
  • 6
    @KonradRudolph I am not trying to misrepresent your position, so apologies. What I mean is that if you are using statistics to determine an empirical result, the result itself is the empirical fact, not the conclusion. The conclusion of "weighing heavier" would be an interpretation of the result. Treating it as fact would be a belief or a bias that it is inherent.
    – jxh
    Commented Mar 16, 2022 at 15:41
  • 5
    @Flimm Yes, it almost certainly matters tremendously. Of course we won‘t know for sure until it has been tried but based on the evidence from other cases where similar sort orders are used, it has a substantial effect. — You seem to misunderstanding the point, however. It’s not about “difficulty”, it’s about motivation: people upvote haphazardly (often without really understanding whether something’s correct or good). But people downvote much more rarely, and (usually) only when something’s actually wrong/bad. This makes downvotes much more informative. Commented Apr 13, 2022 at 10:27
  • 4
    It's probably covered by this answer (which is why I'm not posting another one), but I saw this gem before I realised that hashtag trending hashtag sorting was silently enabled for me. -5-scored post above 0-scored post. Very trending. Commented Apr 18, 2022 at 21:38
  • 4
    One issue with this is that good answers sometimes pick up a downvote or two during their initial lifespan (while they're still being refined or clarified) and the people who drop these downvotes often never retract them. If old downvotes count more than old upvotes, answers with a few early downvotes get doomed forever.
    – Zags
    Commented Apr 20, 2022 at 22:03
  • 4
    @BenVoigt to be a bit contrarian on the whole "treat gold badge downvotes differently", if I as a low rep user downvote, it means I'm willing to take a lot more significant portion of my rep than a 30ker to send a message. There was a point in time where I had to hold off downvoting because I'd downvote right out of the privilege (which is where I still sit on Meta.SE). I know what you mean about "proven expertise" (or perhaps a proven trend of FGITW quick draw to homework dump answers in 2009) but that doesn't mean there aren't any low rep experts.
    – jrh
    Commented Apr 21, 2022 at 15:42
  • 5
    eh, the majority of these voting rings upvote, not downvote. There's little to no benefit to such rings casting downvotes. Regardless, voting rings aren't a reason to not alter voting weights if altering voting weights will benefit the site as a whole assuming voting rings are dealt with.
    – Kevin B
    Commented Apr 21, 2022 at 16:10
40

This needs an accompanying reexamination of the locking of old votes (you can't retract a vote you have given more than some amount in the past unless the post is edited).

Locked Vote

This is for two reasons:

  1. If I revisit a question where I have previously voted on an answer and that answer still is the best, I should be able to reissue that signal. As implemented, my vote counts less because I already gave it.

  2. If I revisit a question where I have previously voted on an answer, I should be able to retract my vote if that answer is no longer good. Under current logic, I can't retract my upvote unless the answer is edited.

I've posted this as a separate feature request: Get Rid of Vote Locking

7
  • 2
    Yes, a lock that times out should be enough to cover the scenario that it is designed for. Commented Apr 15, 2022 at 17:22
  • 4
    Not serious: Put a skull and cross bones icon on the triangular vote icon to indicate that the old vote has "perished". Allow clicking the triangle again to resurrect/revive its weight on the post. Commented Apr 19, 2022 at 1:24
  • 6
    This is important. It was an ugly kludge even when we pretended the situation was static and that answers could be labelled good (or bad) for all time. Yet, time flows and the world changes. Now that the dynamic nature of StackExchange is being officially recognized, we should let go of permanently locked votes.
    – hackerb9
    Commented Apr 19, 2022 at 19:36
  • 1
    "auto-revote when rediscovered" might be even better than having the option to manually revote.
    – enharmonic
    Commented Apr 19, 2022 at 23:21
  • 3
    @enharmonic, I'm not sure what you mean by "auto-revote", but what if you go onto a question that you previously voted on or it's answers, but their relevance has changed? I've been on old questions, but looking for different answers than before. And some of the answers may no longer work, so an automatic same vote wouldn't make sense. A manual vote doesn't assume "nothing has changed" like the current locking system does or an auto-vote system. Commented Apr 21, 2022 at 14:57
  • 3
    Related, I would like to be able to change my vote in general. I might make a mistake (accidentally upvote a post that has a sneaky problem), or I might want to un-upvote an answer that used to be the best solution but has aged badly ("do this on version 2.3.2" isn't valuable for most people if they had to upgrade to 3.5 to avoid a security vulnerability or something).
    – jrh
    Commented Apr 21, 2022 at 15:46
  • 1
    The proposal on MSE: Get rid of vote locking Commented Apr 22, 2022 at 8:34
34

The optimal decay weight might be a compromise of increased variance/noise (with higher decay rates effectively aggregating less information) and increased recency. The noise also depends on the total number of votes. Therefore popular Q&A with high number of votes might benefit from a stronger decay rate emphasizing recency while unpopular Q&A (the long tail) with only a few votes might need a much slower decay rate to remain stable and not dominated by voting noise.

But before going down this road, learn from fixed decay rates first. In the results also correlate with total number of votes given and then maybe try to adapt on it, for example by making the used decay rate a function that increases with the total number of votes or views (or both).

2
  • 14
    Thanks for bringing this up, I like it! I agree that we'll need to measure performance of these algorithms across different post/vote ages. Your idea around choosing different algorithms based on total number of votes is interesting and I'll consider it, though I'm balancing it with a goal on making this algorithm transparent and understandable. Ultimately, we're looking for an algorithm that tends to sort by trending and our goal is not to find a perfect algorithm.
    – Kyle Pollard StaffMod
    Commented Mar 10, 2022 at 19:58
  • 3
    I personally don't mind if the details of the algorithm are a bit hairy, especially in a scenario like this. I expect "trending" to strike a balance between surfacing answers that have recently gained a fair amount of upvotes, while still keeping the order of answers somewhat stable (I wouldn't want a single vote to drastically affect the order of answers). The specific details of how that is done is more of an implementation detail, and it can be as complicated as it needs to be in order to fulfill this intuitive understanding. Commented Apr 20, 2022 at 2:58
18

You may have narrowed down your search space for the optimal decay function too much. I don't quite see the logic here. Quoting from the post:

What’s the process for identifying a good algorithm

We want to find a descending mathematical function that takes an individual vote's age and decays the vote's value. This function should output a result between 0 and 1 inclusive. When the vote hasn't aged, it should have the full value of 1. When the vote has aged sufficiently, it should fall to a value of 0. The function should be continuous without any abrupt changes in values so that newer votes are always worth more than older votes.

While smooth monotonous decay is reasonable, I see no reason why the value of a score should necessarily decay to 0. It would not be unreasonable to have it decay to 0.1 for example.

Rather than making the decay to 0 an axiom that you don't explain, you could have stated that you wanted to find a smooth, monotonous function with the smallest number of parameters to simplify exploration in an A/B test.

Reasonable functions to choose for smooth decay to 0 could be: exponential (1 parameter), power law (2 parameters), bi-exponential (3 parameters).

If we'd allow decay to non-0, that'd add one parameter to all of the above. So the simplest function is exponential decay to 0, which is what you seem to have chosen. Not unreasonable, but it's not the only possible choice.

I actually think that power law decay may be better. Since the value of a vote doesn't decay by the same factor every year. There's no intrinsic time scale over which votes become less valuable. That's an argument for a power law decay which doesn't have such a single time scale.

The disadvantage of a power law decay is that it requires one more parameter. So you may want to explore power law decay after you complete this test.

2
  • 3
    Yes, they definitely have narrowed their "trending score" model without motivating that. With the help of the A/B test now they could however also test other models too, eg. by looking for models that minimize the number of questions where users report that the current or trending sort is wrong while keeping the number high of questions where they report it's right. I also like sigmoidal behaviors (possibly with an offset) where the weight stays close to 1 for a characteristic time and only then goes smoothly down. The characteristic time could for example be tag (technology) dependent. Commented Apr 20, 2022 at 5:30
  • 1
    I really don't see any reason to go to zero in five years already or within some other finite time. A function that has it's asymptote at zero would be way better. Like, why is the 50% decay function zero after five years? I'd expect 1/32 after five years, 1/1024 after ten years and so on.
    – Joooeey
    Commented May 5, 2022 at 13:43
17

I'd like to see the metric used

I like this, I honestly do. I think "recently upvoted answers" is a good metric for identifying good answers that haven't gone stale. But one thing that does bug me. The question I just got asked the survey on had answers in this order: 14, 15, 1, 2. And I did a double take. It took me a minute to realize that some hidden, behind-the-scenes metric was being used instead of the numbers I could see. All of the other sort orders ("high score", "creation date", and "modification date") are readily visible, but the trending metric is not.

Social media platforms are constantly sorting by some vague metric of "trending" or "what we think is best", where "we" is defined as some corporate entity, and I don't want this metric to become (or even be perceived to become) that. It would be nice to have some visual indicator of the metric used. Maybe a smaller number beneath the vote count that indicates the "decay metric" or "decay score". The goal here is to show people good information, not to keep them browsing. So let's make the sorting process as transparent as possible.

1
  • 5
    If we're looking for a name, I think "effective score" is better than being explicit in the name about "decay". It tells the story of what it is, rather than the story of how it was generated.
    – M. Justin
    Commented Apr 19, 2022 at 15:01
16

Give us a reason to trust the algorithm.

I love the idea in principle. In practice, I keep finding myself dis-trusting the algorithm, and scrolling further down a given page to see if other answers exist with more upvotes. I believe the reason for this is simply because I have no reason to trust the algorithm's sort order. It claims that this answer with 51 votes is more relevant than the answer with 185 votes, but why should I believe it? Could it be that the algorithm is just being overly strong in this scenario?

It would be nice to have available a basic graph of the change in a question's score over time. For example, something like this:

enter image description here

In this example, the Y-axis represents the change in score for a given year, and the X axis relates to time.

A full-blown version of this graph could be provided in the question's history, while a miniature version without labels could be embedded within the question, perhaps showing up when you hover over the question's reputation, thus allowing users to easily compare the trends of different questions and gain an intuitive understanding of why it was sorted the way it was.

enter image description here


Edit: Addressing "edge cases"

A graph with few votes over a long period of time would result in a "bumpy" graph as @Trilarion noted. For example, a 10-year-old, 5-vote answer will certainly not have a smooth graph. But, I'm personally fine with this. I'd still like to know if those 5 votes are all old, or if some or all of them are newer, and a bumpy graph will still provide me with this information.

I'll also point out that perhaps some minimal labeling would be necessary on the graphs, perhaps at the extremes of the graph (i.e. what year does this graph start on, and what section of the y-axis is currently visible).

Perhaps a harder scenario to deal with are answers that were recently provided. If the answer is only a year or two old, then perhaps we could change the granularity of the x-axis to be by month instead of by year. And, if it was opened within the past couple of months, we could simply show a message like the following when you hover over the reputation: "All votes for this answer have been cast within the past X months"

12
  • 2
    The change in score over year is basically the derivative or the speed of score change. And that is basically what the trending score adds up to. However, for many answers this curve will look very bumpy. Commented Apr 20, 2022 at 5:39
  • Good point @Trilarion - I edited the response to address this concern. Commented Apr 20, 2022 at 17:02
  • 2
    The fundamental problem is the assumption that an answer has to keep a steady amount of new upvotes to be good or relevant. But even for highly voted answers of mine, I see a vote in a year perhaps, after the initial attention span. Assuming that a new answer was better, just because it recently earned three upvotes (compared to the hundred old upvotes of the old answer) as currently happening, is insane. I think, I’ll start retracting old upvotes and reapply them as new, if this system persists…
    – Holger
    Commented Apr 25, 2022 at 9:15
  • 1
    @Holger "Assuming that a new answer was better, just because it recently earned three upvotes (compared to the hundred old upvotes of the old answer) as currently happening, is insane. I think" I don't know. Maybe this new answer is actually better. We would need to find out by other means, wouldn't we. With your model there could never come up a newer, better answer if there is already a highly upvoted old answer existing. But that should actually be allowed, I think. Commented Apr 26, 2022 at 8:16
  • @Trilarion let me counter with this real life example. The question is from 2009, so it obviously has outdated answers. This late answer from 2016 incorporates a new development and managed to get >400 upvotes without the aid of the system. It’s not necessary to give a few upvotes such an insane weight, good new answers will get some votes anyway. It would be good, if this answer had a chance to outweigh the ancient >1200 votes answer but that doesn’t happen even with the new “Trending” order.
    – Holger
    Commented Apr 26, 2022 at 9:06
  • @Holger But it does and this isn't a counter-example. With the trending sort, the newer (and supposedly better) answer with 400 votes is shown first and the older (and not as good) answer with 1200 votes second, as it should be judging by the quality of the answers alone (I guess). So in this case trending sort was helpful. I'm not very dogmatic. If you tell me, what order you prefer, I can construct a score that achieves that order as best as possible and age of votes would be one aspect to include if it helps me achieve better what you or others want. Commented Apr 26, 2022 at 11:27
  • @Trilarion if you followed the link you had noticed that “Trending” did not change the order. Which is easy to explain. Since both answers are older than five years, the current algorithm gives most of the votes for both answers zero weight. I think, a lot of people already said that ending up at zero weight at all, is bad. Besides that, the weights for newest votes are way too high, but lowering them to a point that copycat answers do not benefit would make weighting pointless. A nonlinear solution is required, e.g. do not apply a benefit to new answers before they got at least ten upvotes.
    – Holger
    Commented Apr 26, 2022 at 11:41
  • @Holger "if you followed the link you had noticed that “Trending” did not change the order." I did and it did. It still does. Do you need a screenshot or do you believe me? Commented Apr 26, 2022 at 12:20
  • @Trilarion this is how it looks for me. If you see something different than the site must use different algorithms for different users, which makes everything even worse. As said, the Q&A we’re discussing states explicitly that votes older than five years get zero weight. Which would mean that ~1500 votes get ignored and a few occasional votes decide what gets to the top. But the point you’re still ignoring is that the new answer got to the 3rd place of 40 answers even without this new feature. In other words, a single new vote doesn’t need such a weight.
    – Holger
    Commented Apr 26, 2022 at 13:23
  • @Holger and for me. It was a misunderstanding, we simply saw different things. What I'm not ignoring is that you argue that the weight decays too fast and too strongly (and it may well be in your case). I fully acknowledge that and think it should decay slower and weights should never reach zero. But in my eyes this doesn't mean that trending sort is useless, just currently not properly adjusted. Currently they try to adjust it. That is where we differ. Commented Apr 26, 2022 at 14:33
  • 1
    @Holger From the OP "You'll receive any one of the four proposed algorithms, but you won't know at the time which one you're getting." That should explain the discrepancy. Commented Apr 26, 2022 at 14:36
  • 2
    @Trilarion I never called it useless. We agree that it is “currently not properly adjusted”. What I doubted, is the usefulness of a simple weight factor derived from the voting time. It will always tend to give too much weight to a single sympathy vote while probably having too little effect against heavily upvoted outdated answers. That’s why I think, nonlinear corrections are unavoidable. Like the suggested threshold for new answers.
    – Holger
    Commented Apr 26, 2022 at 14:44
15

It looks like I got the survey prompt today. A very, very minor issue is that the survey prompt is phrased like a question, but ends with a period instead of a question mark:

Are the answers below sorted in a way that puts the best answer at or near the top.

Screenshot:

Survey prompt

2
  • 7
    Thanks for reporting this issue! I just deployed a fix. Now the sentence ends with a question mark.
    – Shiyao Li StaffMod
    Commented Apr 13, 2022 at 22:05
  • 2
    @ShiyaoLi You're welcome! I can confirm that it now shows me the same text, but with a question mark.
    – M. Justin
    Commented Apr 13, 2022 at 22:16
14

So, this happened:

enter image description here

Note that all votes have almost the same age, so yeah. It seems that it's not normalized to zero, so no amount of downvotes would make it rank lower. This touches on many issues that were noted on other answers here. So, we need to know what the algo is doing, how it's scoring things and that it should take the direction of the votes into account, because this suggest that you feed upvotes and downvotes without differentiation.

1
11

I really like what you're doing here. If anything, I feel you're not being ambitious enough, by just making it another option, along with score, active etc. And the word "trending" feels...weird. Answers shouldn't be trendy, they should be correct. I don't care what the "trending" answer is, I want the best answer.

My suggestion would be to call it "Best" or "Recommended", and leave open the possibility of tweaking the algorithm further over time, and aim to make it the default (once you know it's better than just "score").

9
  • 21
    “Trending” makes sense, though. If there’s a trend towards a positive score, then it’s trending (not necessarily trendy), isn’t it? But you’re probably right that the word choice should be more open to suggestions. Internet users are probably more familiar with the word “Trending” from Reddit, but contributions on Reddit are mostly not like contributions on Stack Exchange. Old answers and questions having a lot of votes is already contributed to higher “popularity” which is seen as problematic, and we probably shouldn’t encourage this “popularity” mindset even more. Commented Mar 16, 2022 at 11:24
  • 11
    “Recommended” can be confused with the Collectives label, unfortunately. “Best” is too ambiguous if it sits right next to “Score”. Commented Mar 16, 2022 at 11:27
  • 10
    ‘Best’ may imply a claim about quality. I don’t believe weighing votes by recency does much to more accurately capture quality, unless incidentally (to the extent vote recency correlates with quality, e.g. by anti-correlating with clustering in time near creation typical to Fastest Gun answers).
    – dumbass
    Commented Mar 17, 2022 at 14:58
  • 7
    "Trending" is the correct word and means " a general direction in which something is developing or changing" and doesn't have the same meaning as "Trendy" which means "very fashionable or up to date in style". Commented Apr 13, 2022 at 12:50
  • 1
    These definitions of "trending" and "trendy" may technically be correct, but I don't think they match the understanding of most users on modern social media platforms, for instance. Twitter uses "trending" to mean "currently most active", for instance. Commented Apr 14, 2022 at 0:19
  • 1
    "Recently recommended" would be more accurate than "Best" or "Recommended" alone. Best is bad because the answer may not be the best, particularly with a high decay rate.
    – mdfst13
    Commented Apr 19, 2022 at 9:01
  • 4
    "Trending" is a very poor choice given current usage of the word. For example, both Google Mobile and Twitter use it to show distracting, irrelevant information. There are other ways to imply a statistical "trend line" without using that word. I suggest Rising. Its meaning is well understood without the baggage of "trending".
    – hackerb9
    Commented Apr 19, 2022 at 19:07
  • @hackerb9 so, considering the current results, “Trending” is the best word to describe it.
    – Holger
    Commented Apr 26, 2022 at 7:07
  • @Holger 😆 If the results show this sort is distracting and irrelevant, then I'm all for naming it Trending. It certainly seems to have a glitch in considering a single up vote to be a "trend". However, I've seen some good use cases (see the Necromancer above), and am hopeful both that this will be a useful sort and that it'll be renamed to Rising.
    – hackerb9
    Commented Apr 26, 2022 at 21:44
11

I've run across a rather extreme example showing this new algorithm doing its job, putting a new answer with 5 upvotes above an old answer with 1485 upvotes.

I'm a thread necromancer (it's the badge I've earned the most in the last decade, at 30 times). While some of these are providing a first answer (or first good answer) to a question, a number of them are providing a new answer based on updated technology, or an answer that addresses gaps in the existing answers.

On questions where I was late to the party but with an answer that is receiving regular upvotes, I've noticed my necromancied answers are much, much nearer the top with the trending sort.

I think the most dramatic instance is this question with 71 answers, originally asked in 2008. Three months ago I posted a solution based on change in Java 17 (released September 2021). Despite only having 5 votes to the top answer's 4168 votes, it's appearing third in the list, above an item with over 1485 votes.

screenshot of trending votes

This and other similar examples:

Question Answer votes (* are my answers)
Java random int 4168/435/5*/1485/178/9
git switch 430/52*/151
git mv 482/6*/1/82/36
Java subpackages 186/20*/63/5
Spring Boot profiles 11/5*/22/6
HttpServletRequest mocking 8*/24

One thing I did observe while going through my necromancied posts. In less frequently upvoted questions, my necromancy answers tend to be more in line with their total vote counts, rather than higher up. Maybe that's where they belong, though I suspect it's a case where the decay function breaks down in questions where there's not a lot of recent activity, but only a slow trickle of activity over time.

14
  • 5
    I'm not sure I understand why you've posted this. Is this supposed to be a problem? Or a good thing?
    – VLAZ
    Commented Apr 19, 2022 at 15:50
  • 2
    @VLAZ An observation of its effect in a rather extreme example.
    – M. Justin
    Commented Apr 19, 2022 at 15:52
  • @VLAZ I can certainly delete it if it's not useful for the discussion.
    – M. Justin
    Commented Apr 19, 2022 at 15:53
  • 6
    It's a good observation, @VLAZ - if the new sort is able to do that, I feel like it might be worthwhile. That said, this answer can benefit from some generalisation - a dozen more examples of where a new answer outpaces an immensely voted one would be nice.
    – 0Valt
    Commented Apr 19, 2022 at 15:59
  • 5
    @OlegValteriswithUkraine I've gone through my own necromancy posts & added 5 more similar examples. It's not the dozen more, but it is more data points.
    – M. Justin
    Commented Apr 19, 2022 at 16:23
  • 1
    @M.Justin nice, thank you for adding some data points!
    – 0Valt
    Commented Apr 19, 2022 at 16:35
  • 4
    I posted an answer, it got one upvote, and jumped ahead of 24 other answers, one of which had 64 votes. Not quite as extreme as what's shown here, but still, I feel this not good. if you see a one-vote answer followed by a 64-vote answer, which one are you going to be more interested in? Which would you want to see first? The answer that a single person out there recently thought was pretty good, or an answer that a fairly large subset of the community found useful at some point in time? Commented Apr 20, 2022 at 3:34
  • 7
    ...I guess I'd like to see a little more stability in the ordering from this algorithm. Right now it feels the answers are too free-floating, able to move far distances with a few votes. Yes I want to see trending answers above dated ones, but I'm also ok if they still have to work a bit to get there. Commented Apr 20, 2022 at 3:38
  • 3
    @ScottyJamison Yeah, a single upvote isn't a trend. 5 upvotes for something on the third page over something on the first page with 1000 upvotes, that's starting to sound more reasonable to float to the top.
    – M. Justin
    Commented Apr 20, 2022 at 3:38
  • 3
    Honestly, it actually feels more extreme to me than what I have here.
    – M. Justin
    Commented Apr 20, 2022 at 3:39
  • These examples show that it still heavily favors the one with more upvotes (or higher score). And while yes, it promotes later comers, the top answer is still the same in all scenarios.
    – Braiam
    Commented Apr 22, 2022 at 10:42
  • I'd say it seems to heavily favor the one with the highest score, which is probably a combination of it being a good answer and the first thing users see (so they upvote it if it looks good). In all these cases, there are ones with higher scores that are further down on the list. Though 2 of the 6 answers had something other than the one with the highest score on top.
    – M. Justin
    Commented Apr 22, 2022 at 15:15
  • 2
    It’s a good thing for an answer like yours, just like intended. Unfortunately, I have seen a lot of bad answers moved to the front by this new algorithm. All it takes, is posting an actually obsolete answer to an old, highly voted question, retelling what other answers said and get a few upvotes and there you are. In fact, we know that a lot of such “late answers” are bad, that’s why they go to the review queue automatically. But a criteria like “obsolete because not telling something new” does not exist. As long as they are not outright plagiarism, they benefit much from the new system.
    – Holger
    Commented Apr 25, 2022 at 9:08
  • 1
    @Holger In the interest of full disclosure, one of those six of mine actually is an answer that retells what other answers say. However, it's doing it for a very specific reason, and is completely up-front about that (going so far as to link to the other answers). Namely, that none of the existing answers tells the whole story. It also goes into further detail than many of them do. stackoverflow.com/a/65748081/1108305
    – M. Justin
    Commented Apr 25, 2022 at 15:20
11

I have received the trending answers test a couple of times now. Both times, the result of the trending sort was awful, with low-quality, low-voted, newer answers sorted above high-quality, highly-upvoted, older (but still valid) answers. I understand the objective here, but I am skeptical about finding an adequate solution along the present line of development.

The fact is that being old does not necessarily make an answer incorrect or even dated, and old votes for old, still-good answers are as reflective of the answer's quality now as they were when they were first cast.

Plus I'd rather see once-good, now outdated answers ranked ahead of poor new answers.

11
  • 1
    It will be interesting to see what the test results say. Maybe your experience was just extremely unlucky. People hopefully haven't just voted completely garbage in the last years. Or maybe they have. We will see. Commented Apr 24, 2022 at 8:01
  • 4
    This matches precisely with my experiences. I absolutely hate the "trending" sort. I don't care what is "trending" (i.e., what kind of noise that people have recently been upvoting); I want the best answers. I, also, do not understand why, despite the fact that I keep turning it off, my preference doesn't stick.
    – Cody Gray Mod
    Commented Apr 24, 2022 at 9:31
  • 1
    @CodyGray "I absolutely hate the "trending" sort. I don't care what is "trending" (i.e., what kind of noise that people have recently been upvoting);" Then you definitely need the "oldies" sort option, i.e. where newer votes are weighted less not more (if you really do not care about the noise of recent upvotings). Commented Apr 24, 2022 at 9:48
  • 3
    You mean the sort option that has been the default for SO for many years and which has contributed to making this the best site on the Internet for finding an answer to programming questions, @Trilarion? Yes, I think that is exactly what we need.
    – Cody Gray Mod
    Commented Apr 24, 2022 at 10:56
  • What kind of answers? "Try this" code-only answers without any explanation? Or something else? Commented Apr 24, 2022 at 11:01
  • 1
    @CodyGray No, I didn't mean that one. That is still there and should stay and even if it temporarily doesn't stick anymore then surely this is a fault that would neither speak for or against another sort option. I take it your experience was that trending sort is rubbish (the same way John Bollinger reports on it) and that this is because recent votes are just noise and therefore what you really need is the opposite of trending, i.e. an oldie sort were never votes are just discarded or only regarded with a very small weight compared to the good old votes. Commented Apr 24, 2022 at 15:02
  • 2
    @PeterMortensen for example, copycat answers, just retelling what older answers already said (without being an actual plagiarism). Or an answer, suggesting “you could try this instead”, not being outright wrong, deserving their three upvotes, but in no way, deserving to be placed above a long, detailed answer rightfully having hundred upvotes. At the moment, practically every “late answer” to an old question heavily benefits from the “trending” sorting when only having a few upvotes. If this persists, I have to counter by retracting all my old votes and reapplying them, to make them “new”.
    – Holger
    Commented Apr 25, 2022 at 9:22
  • 3
    A simple example, a late, code only answer, just one upvote, beats everything in the new sort order…
    – Holger
    Commented Apr 25, 2022 at 9:58
  • 2
    @Holger See my answer here about longer decays for answers with low number of votes because otherwise they become too noisy. This might be such an example. In general the chosen decay time might be too short. It seems the current trending algorithm basically only looks at 1 year or shorter (even with 50% decay, the lowest possible in the original post here the weight is only 50% after already one year). Maybe lower decays overall should be considered. Commented Apr 25, 2022 at 13:41
  • 3
    @Trilarion, this answer raises a deep problem inherent in the whole concept of a "trending" sort: a vote's age is not well correlated with its value for ranking answers if the objective is to rank by answer quality and relevance. It doesn't much matter how you tweak the parameters if the underlying concept is flawed. You might make the results less bad, but I'm not seeing how we end up with results from this that are actually good on average. Commented Apr 25, 2022 at 14:51
  • @JohnBollinger I'd say that a vote's age is at the very least somewhat correlated with content quality, i.e. votes lose relevance a bit with their age. Therefore some sort of weighting makes sense. However, they may overdo it and end up drowned in noise. But I don't see the underlying concept as flawed, rather as sound. If the total score is simply the accumulated score then trending score is something like the first derivative of it and tells you how much momentum is gained lately. Surely this is additional information, how useful it is or how to best use it nobody really knows currently. Commented Apr 26, 2022 at 8:13
11

The decay process makes the implicit assumption that the accuracy of an answer will decay over time. This may be true for some topics (six major Python libraries completely changed their APIs in the time it took to write this answer), but other answers are timeless or very nearly so.

None of these questions have answers that have become any less accurate since the day they were written, and while none of them have decayed to near zero yet, they eventually will:

An answer should not be expected to be more up-to-date just because it received a vote more recently. It's highly unlikely that a language (for example) will change in such a way that a previously written answer happens to become correct. Only answers written more recently than the top-voted answers might be expected to be more useful due to being more up-to-date.

Further, if you permit decaying to zero, then any question for which the answers are not continually receiving new votes will eventually have the value of each answer go to zero. Once this has happened, the modified score will depend only on recent votes, and even if we assume that the probability of an answer receiving a vote is proportional to its quality, there won't be enough votes for old questions for this sampling process to converge to a reliable representation of an answer's quality.

Suppose you have a question with two answers. One has received 3,000 votes. The other has received 2. But it's been 2 years since the question was active, and so both have decayed to effectively zero (we'll use the 97% rate for this example). Since then, someone occasionally stumbles upon the question and votes. The good answer got 20 votes a year ago and 2 votes 6 months ago, for a modified score of 0.96. The bad answer got a vote yesterday, which means it's at the top, and will remain so until the next times someone votes. This is clearly the Wrong Thing.

The correct way to bias towards answers that are more current is to estimate the expected value of the number of votes an answer would have received if it had existed since the question was posted, given the rate at which it has accumulated votes since it was written. (There are a few ways you might do this: multiply out the average rate of change, or fit a curve to the rate of change and assume the trend will continue, or perhaps do one of those and then scale by the number of total votes received over that period, to take a few examples. Arguments can be made for any of those, but that's out of scope for this answer.) An answer that was written two days ago and received 10 votes probably should be ranked highly. An answer that was written at the same time as all the others that just happened to receive a handful of votes slightly more recently than the other answers should not.

Addendum: When calculating the expected number of votes, we need to be careful to not update until enough time has passed that we have a reasonable sample period. This answer received a vote 5 minutes after being posted and 48 days after the question was asked. It would have been an error to scale that up to +13,800.

2
  • While being an important argument and I personally wouldn't set the weight to zero but always a positive non-zero value, I think it's highly unlikely that an answer that got 3000 upvotes in the past will get none anymore for some years. This might be a rather constructed case and most probably not representative of what really happens. The votes statistics is available on SEDE. Maybe someone could lookup how often such or similar cases really happen. Commented Apr 27, 2022 at 8:59
  • @Trilarion Absolutely true. I constructed the example specifically to show that it was possible for an answer to receive an absurd number of votes originally, continue to receive more votes each year, yet still fall below an answer that receives a single vote more recently. I expect that such examples would be uncommon when the better answer received that many votes. But if the better answer received 30 votes and the worse received 2? It's quite likely that either could go years without receiving new votes, and as soon as one of them gets a single vote, it's on top for the next few years.
    – Ray
    Commented Apr 27, 2022 at 18:04
9

This sort option awfully misrepresents locked answers

I just visited this popular question where this famous* answer used to be the top one because it has the highest score and is the accepted answer**. Now, that answer cannot even be found anywhere on the first page. Instead, it's listed near the end (only followed by 5 non-deleted answers).

One could argue that it's not "trending" anymore. But that would be an unfair and misleading argument. Since the answer is locked and can no longer be voted on, we can't really tell whether or not it's "trending". In fact, it has been locked for many years, and in all those years, no other answer has managed to surpass its score. That should tell us the algorithm something.


* ...or infamous, depending on who you ask.

** Yep, that no longer counts.

5
  • 1
    Not to dismiss this as a data point, but this feels like a knock-on effect. While these sorts of answers are very rare (this was locked to prevent people messing up the formatting), this is a side effect of the lock not allowing votes, which is sometimes a misfeature but we have no choice as locking prevents voting by default.
    – Catija Staff
    Commented May 2, 2022 at 14:15
  • 2
    @Catija that seems more like locking is a misfeature in design. Maybe it should just be scrapped,
    – Braiam
    Commented May 2, 2022 at 15:04
  • @Braiam Um. No. Locks are extremely useful. In most cases they work exactly as they should. But there are some places where they don't work optimally.
    – Catija Staff
    Commented May 2, 2022 at 15:05
  • 6
    Rather than abolishing locks, I think this phenomena is instead more of a reason to push for more granular locks, e.g. preventing a post from being edited but not from being voted on.
    – zcoop98
    Commented May 2, 2022 at 15:25
  • 1
    So I poked around a bit... There's only 10 answers with a score of 10 or more that are locked and most of them have comment locks, which don't impact voting... so this is only an issue on maybe 5 posts currently on the site... which... feels like it probably doesn't need a solution?
    – Catija Staff
    Commented May 2, 2022 at 18:13
5

Overall, I agree that older votes can likely be considered less relevant than newer votes. I welcome the efforts to raise the visibility of more relevant posts.

Just a couple of notes:

  • A post with a mix of older and newer votes may benefit from giving their older votes greater weight than a post that only has older votes. However, leaving older votes with some net weight probably mimics this effect (e.g., the 97% decay algorithm).

  • Since vote weights decay over time, it may be worth considering offering a mechanism for a user to touch their original vote to refresh it. One idea would be to make the up-vote icon clickable for votes that have been decayed. The result of the "revote" would of course only refresh the original vote, not grant another vote.

10
  • 2
    "...a mechanism for a user to touch their original vote to refresh it..." Voting is already so much effort. I'm not sure I really want that. If I need to renew my votes every five years or so it would become too much work and I would probably stop doing it. Trending sort orders will always be much more volatile than the aggregate voting. I fear there is nothing we can do really short of constantly asking: "Is this answer still good" .. "and now" ... "and now" ... which wouldn't work. Commented Mar 11, 2022 at 8:29
  • 4
    @Trilarion It is not something I think would be used all the time. I imagine it would just be for occasional posts that get rediscovered.
    – jxh
    Commented Mar 11, 2022 at 16:38
  • 2
    Still it would be additional effort. Maybe, if I revisit a Q&A that I have voted on in the past and I scroll to an answer I voted on and I do not change the vote, maybe that could be seen as partly resetting the timestamp of my vote. It's surely a possibility to renew votes, but I just wanted to point out that it means that additional effort must be done. Commented Mar 11, 2022 at 17:04
  • 3
    Now I have a better idea. We could maybe notify people that are active (have visited the site in the last X days) and upvoted an answer that is trending very badly or downvoted an answer that is trending very favorably right now. Kind of "maybe recheck your vote". If done sparsely (for example only on popular questions or where it could make a difference), that could help. Commented Mar 11, 2022 at 17:07
  • 1
    @Trilarion That's a good idea too. Again, I don't expect users to be presented with a queue of questions with decayed votes. I only think a user may want to refresh an old vote for a post they still find relevant if they happen upon it again.
    – jxh
    Commented Mar 11, 2022 at 19:55
  • 6
    @Trilarion I find the “auto-revote when rediscovered” idea better than the “notify users of old answers” one. If I randomly come across an answer I upvoted a long time ago, it means I actually care about this answer right now. I don’t necessarily care about some answer the system tells me to look at. This even currently happens when I follow some poorly asked question to wait for the OP to provide an MRE or whatever’s missing. When they finally (rarely) do provide the missing details after several days, I already lost interest; I’ll undo my downvote but I’m not reading everything again. Commented Mar 14, 2022 at 8:02
  • ‘A post with a mix of older and newer votes may benefit from giving their older votes greater weight than a post that only has older votes.’ — doesn’t that problem tautologically solve itself? If a vote has a mix of older and newer votes, then the weight of newer votes is going to compensate for the loss of weight of the older votes.
    – dumbass
    Commented Mar 17, 2022 at 14:47
  • @user3840170 It depends. When comparing two posts, where one has many many older votes, and one has not as many older votes but has been tracking newer votes, it could make a difference as to which one gets highlighted as better.
    – jxh
    Commented Mar 17, 2022 at 15:15
  • @user3840170 from what I see in practice, no, it doesn’t work that way. There’s a simple reason for that: I can’t vote twice on the same answer. So if a new answer has been posted to an old question, I may consider it “also useful” and vote on it, so I have voted on two answers for this question, but the “trending” order considers my newer vote to be stronger, against my intention. Even worse, this implication is not recognizable beforehand. All you see, is an answer with 100 upvotes and another with 2 upvotes and not that giving the latter a 3rd upvote could move it atop the former…
    – Holger
    Commented Apr 25, 2022 at 9:33
  • @SebastianSimon The "auto-revote" mechanism I had in mind was the first suggestion. That a post with newer votes in a particular direction will sort of automatically raise the weight of older votes in the same direction. In the absence of an "auto-revote", I suggested a manual method to refresh a vote. But, the "auto-revote" on visit is a good idea none-the-less.
    – jxh
    Commented May 16, 2022 at 18:05
4

I'd like to say this is one of the best things Stack Overflow has ever done for me. I frequently stumble upon questions with 10+ answers and try to make sense of which one is the best by reading comments, last updated time, upvotes count, etc.

Now, I've found "Recent votes" to be a much better heuristic (if not the best) for determining the best answer. Thank you for saving my time.

4

I am confused after receiving the survey. I mean, I don't know which sorting I am supposed to comment on. The question does not tell me. The sorting for me was the new one (Trending):

enter image description here

In this case, does the survey question ask about the "Trending" sort order?

enter image description here

2
  • 1
    Yes, it should be the new sorting. Survey question asks abot "Based on the question you were just viewing" - if that question was using the Trending sort, then that's what its asking about.
    – VLAZ
    Commented Apr 14, 2022 at 7:03
  • 1
    It should be whatever sort you saw. So if it was the old way, say whether that was good. If it was the new way, same for that.
    – Ryan M Mod
    Commented Apr 14, 2022 at 7:21
4

The survey could be phrased better. Specifically this:

enter image description here

When discussing programming in general and when discussing how programming languages are supposed to work specifically, there is no "seems to work". We are supposedly dealing with engineering here, and as such it is based on science. Not on aimless trial & error or guessing. Either it works or it doesn't.

Particularly not in languages such as C and C++ that come with a lot of poorly-defined behavior.

In fact this scenario is common on SO:

  • Someone asks a question about strange behavior they can't explain.
  • Someone answers with a quote from a standard or documentation explaining why the behavior is such and how to deal with it (the correct answer).
  • Some John Doe shows up and posts a late answer (ignoring the correct one already present) "try this it worked for me". That is, it happened to work at a very specific compiler for a very specific target. Pretty much a school book example of a bad answer which will get lots of down votes.
3
  • 2
    I disagree. Yes, programming either "works or it doesn't", but if I find an answer that doesn't exactly match the "technology versions I am using", then "seems to work" might be an extremely appropriate description. This is especially true when the answer is written in a similar but distinct language from what I'm using, like MySQL to MSSQL. General concepts can be applied, and could very well "seem" to work, but then later fail for some other reason that I missed at first. I feel the current wording is adequate.
    – zcoop98
    Commented Apr 19, 2022 at 15:40
  • 1
    @zcoop98 What you are describing is a process called "guessing". It is one way to solve problems. Another way is to study the technology being used through sources that are actually reliable. One of the core purposes of SO is to become a collection of technically accurate knowledge. Those who are looking for "seems to work" solutions are better off using the various tutorial sites of diverse quality.
    – Lundin
    Commented Apr 20, 2022 at 6:19
  • Same, I just skipped this question because none of the answers made sense in context to what I was seeing and I could not be bothered to "describe", especially after I was already bothered by seeing answers in a strange, unhelpful order.
    – briantist
    Commented Apr 21, 2022 at 13:45
4

It's generally confusing UX to sort a list according to a property invisible to the user. This is also important to gauge the relative relevance of a question compared to another.

If traditionally I have read an answer with 100 upvotes, and the next one has 85, I would maybe read the next one, too. If the next one after 100 has only 3 upvotes, I might not invest the time. With the current vote values shown this makes no sense anymore, and I am doomed to scroll to the bottom to check if there is a great 1000 upvotes answer that got punished.

It's similar to making recent downvotes more visible. Just knowing the algorithm had its reasons to put a -2 voted answer above a +13 voted answer like in How useful is Turing completeness? are neural nets turing complete? is not very helpful. I would like to know if the higher voted answer was recently downvoted even more, or just had not much activity.

So the votes column should show eventually something like a "recent score" or "normalized" score when sorting by "trending".

Generally I am not sure if the algorithm performs as well on low-activity items as on high-activity items. Those seem quite different beasts.

I agree to another answer here that downvoted are more meaningful than upvotes.

I would also like to see some more initiative to hide duplicate answers in questions with 10+ answers. Currently it seems all I can do is add a comment or call for a moderator, and the closing of duplicate questions seems much better organized.

An alternative approach might be to be able to explicitly mark questions as outdated (similar to marking as duplicate).

1
  • Basically what you want is a displayed trending score? That should not be a problem. For determining the order, the company should already computed this. They just need to display it and give it a nice name. Maybe "trending votes". Commented May 2, 2022 at 11:02
2

This experiment is now live

Since this is a experiment, there should be test goals. How are those goals measured?

We're going to measure how copy and voting behavior changes between the default Score sort and the four Trending sorts

The copying can be misleading. Yes, I may copy the top answer, but only to test it out. Or I may not copy it at all, because I couldn't figure out how it applies to my problem.

We're also watching the rate at which users change between the different Sort options

Apparently, I'm not savvy enough to see those options.

enter image description here

We want to ship one of these four algorithms

What if there isn't any statistical difference between trending and score based sorting? Is that part of the test? Intervention vs no intervention should also be evaluated, not only the several kinds of interventions against themselves. Control groups are required.

Also, I would like to see some chaos ordering. Totally random. It could not be a default option, but a option. And no, it's not a joke option either. If the trending vs score vs random makes no difference to certain posts, that's something that "experienced (with the topic) users" may want to evaluate.

2
  • 5
    To be honest, I personally don't have any appetite for a random ordering option. Commented Apr 19, 2022 at 1:28
  • 1
    @mickmackusa it's the chaos route and the best way to actually measure doing something vs doing nothing. If you don't beat random (or do so by almost nothing while investing effort) then that method is probably not useful. A scientific driven approach will recognize the value of that.
    – Braiam
    Commented Apr 19, 2022 at 11:18
1

I would suggest that any recent changes in accepted answer should also be counted towards trending.

1
  • 2
    I kind of agree, but not enough to upvote your answer. We actually seem to be moving away from trusting the asker's opinion when trying to determine if an answer is "best". At most, I think I could support allowing the green tick to be the tie breaker ...but that's about it. Commented Apr 19, 2022 at 1:30
-1

Maybe it is a bug. When I open this post in Google Chrome (version 100.0.4896.75 (Official Build) (64-bit)),

Using a pre-defined color palette in ggplot

I get an empty box between the question and the top answer:

enter image description here

But I can see there is some text when I open the same question in Edge (Version 100.0.1185.39 (Official build) (64-bit)):

Enter image description here

5
  • 7
    Good odds that you have an ad blocker or privacy protector in Chrome that is blocking this survey banner, since it tracks you.
    – Cody Gray Mod
    Commented Apr 14, 2022 at 12:35
  • @CodyGray I see, thanks, when I go into incognito the survey banner shows up. Yes, I have blockers, but why is that banner treated as an ad?
    – zx8754
    Commented Apr 14, 2022 at 12:45
  • 4
    @zx8754 depends on what filter lists you're subscribed to. My adblcker doesn't block it, for example.
    – VLAZ
    Commented Apr 14, 2022 at 12:48
  • 5
    This is indeed an ad blocker. The cause is EasyList Cookie and Fanboy's Annoyance, and possibly any other that go after div[data-cookie]. You' want #@#div[data-cookie] in your filters to fix it. If that doesn't work, consult the HTML, and logger for your ad blocker to find the cause, and create a whitelist entry for it. Not SO's fault anyway Commented Apr 14, 2022 at 18:22
  • 1
    Some of the screenshots could benefit from having frames added to them. It is confusing when included here. Commented Apr 15, 2022 at 17:31
-3

As we’ve been moving forward on the Outdated Answers project

I still don't get why isn't editing part of the toolset. This change presumes there's an up-to-date answer that simply doesn't gather the upvotes necessary to rank higher. This will not help in the most egregious cases, while editing will solve the "problem" instantly everywhere. I don't see how sorting differently will make sure that up-to-date information surfaces to the top. Not even google knows how to algorithmically order freshness of information.

8
  • 11
    I’m not seeing what the problem or feature request is here. You can edit. What specifically do you desire to be done more with editing? Commented Apr 1, 2022 at 0:57
  • @SebastianSimon that this is presented as a solution to a problem, when the process doesn't address the issue at all (no amount of ordering will make post suddenly be up to date).
    – Braiam
    Commented Apr 1, 2022 at 14:54
  • 5
    Ordering can still bring a more up-to-date answer to the top, over an outdated answer. That’s the problem being solved here. The fact that the other answer remains outdated is an orthogonal problem that can be solved by editing, deleting, or providing a new answer. You can’t expect Stack Exchange to solve all the problems in one fell swoop. Commented Apr 2, 2022 at 1:12
  • 4
    @SebastianSimon how this guarantee this? It doesn't. Heck, I've seen +10 years old answers that are still up-to-date (and actually are the best answers) while new ones are just janky solutions of things that may or may not work with specific versions.
    – Braiam
    Commented Apr 2, 2022 at 12:17
  • 2
    if the old edited up-to-date answers are indeed better, they would presumably also be getting recent upvotes, equal or more compared to any new answers / janky solutions, so they will rank and show up higher. Therefore, these changes should be compatible with edited answers.
    – Arthur Yip
    Commented Apr 16, 2022 at 12:36
  • 2
    @ArthurYip "they would presumably also be getting recent upvotes" and I would presumably presume nothing. I want guarantees that the top answer is the best™ answer. Changing the ways chairs are moved doesn't guarantee that.
    – Braiam
    Commented Apr 16, 2022 at 12:45
  • Editing doesn't guarantee that either. Any idiot with 2k+ score can edit. So if you presume nothing, editing should reset the score. Because with no presumptions, there is no reason to think that an edit has made an answer better rather than worse. And yes, I've seen some edits of top ranked answers that IMO clearly made the answer worse.
    – mdfst13
    Commented Apr 19, 2022 at 9:11
  • @mdfst13 considering that a any edit will bring up the post to the top of the active, is less likely that an edit that made the post objectively worse stands as is. This has had 50 revisions, by 40 users and only two inconsequential rollbacks. Incremental improvements work and has been proven to work. This will not work and has been proven that it's a hard problem to solve (see Google ordering algorithm).
    – Braiam
    Commented Apr 19, 2022 at 10:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .