-17

Update

Our experiment incorporating machine learning (ML) to improve the relevance of related questions has concluded. Thank you to everyone who provided valuable feedback on that experiment. It is vital data that help us ensure we are on the right track with this initiative.

Results

This is the newest data that we received from the last experiment incorporating machine learning (ML) to improve the relevance of related questions:

  • Related questions recommended from the Machine Learning (ML) model generated a whopping 155% increase in clickthrough rate (CTR) in the aggregate. Please note, the variant group includes both questions from the model and Elasticsearch because newer/recent questions are not factored in, thus they fall back to being recommended by Elasticsearch. However, the majority of the clicks in the variant were from questions recommended by the model.

What about other metrics?

  1. Are users voting more?

    • Yes. Of the users who are able to vote, there was a 23% lift in overall votes combined (upvotes and downvotes).
  2. Are users commenting more?

    • Yes, we saw a similar trend as well with a 26% lift.
  3. What about voting or commenting attempts? (We defined attempts as users who aren’t able to vote due to being anonymous or lack reputation. This is used to measure intent)

    • We saw a 68-70% lift in users who attempted to engage (vote/comment) on related question pages in the variant group.

What happened to our postponed plan for the second experiment?

We are graduating the "Related questions using Machine Learning" experiment (without doing a second experiment)

What happened to our original plan for the second experiment?

We have decided to postpone the next experiment.  When we updated the model we found something that was not acting as expected and did not want to move forward until it was resolved. We will update the community soon about when we plan to reschedule.

What was our original plan after the first experiment?

Our next experiment will involve updating the sort order of related questions recommended by utilizing the machine learning (ML) model.

How it works is that questions recommended by the ML model will be sorted by similarity score first, which means the questions with the highest relevance will appear first in the list. Just like the last experiment, we calculate similarity or relevance by leveraging an embedding model using titles and tags which converts text into a numerical vector. We then measure of distance between the two to populate the results. In the previous experiment, the recommended questions appeared in a randomized order, so we hope that this will improve the reliability of the list of recommended questions.

We agree that clickthrough rate (CTR) is not enough of a metric to judge relevancy. In this experiment in addition to measuring CTR, we will also be measuring a number of new funnels including:

  • Whether or not a user copied (i.e. code snippet) from the page
  • Whether engagement attempts were made (i.e. anonymous users or registered users with or without enough reputation attempted to vote or comment)
  • The amount of time spent on pages

We heard your feedback regarding including inline feedback during these experiments. While we will not be collecting that data with this experiment, we are working incorporating a prompt to collect qualitative feedback when users interact with related questions.

Just like the last experiment, we will be asking that you share your feedback on a linked question (link to be added soon) that we will title with the experiment name that we are currently running.

Thanks to your invaluable contributions, we have already started talks on ways to incorporate some of the ideas on how better to achieve relevancy, so please continue to offer your suggestions as much as possible.


Background

As part of our Content Discovery initiative, we've undertaken several experiments on Stack Overflow, and have several more planned. For each of those experiments, we’ve created a post here on meta that discusses the experiment and links back to the initiative.

While this simplifies things in the sense that each topic has a dedicated post, it also leaves feedback dispersed. Additionally, the central initiative post isn't serving as a single source for updates in the way that we had hoped.

In order to centralize feedback and provide a single source for sharing experiments and updates — we're rolling out this post as the new format for informing the community on this initiative.

Overview and Objective

Site satisfaction surveys have made it clear that discovering helpful content is a pain point for many visitors to Stack Overflow. Out of the 21,595 responses to the survey year to date, only 11% of participants mentioned that discoverability was one of the most valuable aspects of Stack Overflow. However, when asked “What would you most like to improve about using Stack Overflow?”, discoverability ranked third in responses, indicating that it’s important to prioritize.

The objective of these experiments, then, is to make finding content easier/more reliable/more successful for more users. Not only is "find an answer" the core service but finding quality related content means that we can drive learning and exploration.

Moving Forward

These experiments are largely focused on small changes over time. The current focus of our attention is bettering the Related Questions module, from moving where it appears on the page to improving the content shown there. We may also investigate other ways to recommend content to users, such as using browsing history to point users to pages that could be relevant to their searches.

To summarize the current focus of Content Discovery, we are:

  1. Continuing to iterate on related questions (i.e. improve relevance)
  2. Introducing user-to-content recommendation modules

Related Questions will go through several iterations, especially once we've launched the first machine learning experiment in the near future. Internal content recommendations (i.e. recommended for you because we think you like "python") will be another area of focus for experimentation that will be prioritized next quarter.

Moving forward, we'll announce new experiment releases as an update on this post and the table below, instead of a standalone Meta post for each experiment.

The Experiments

We’ll be updating the below chart when experiments have been released or have concluded. That way you’ll be able to easily see updates and next steps all in one location. Note that the experiments listed on this table are listed from newest to oldest experiment.

Summary Start Date End Date Description Measure of Success Status
Updating the sort order of related questions utilizing the machine learning (ML) model Using machine learning to improve the reliability of the list of recommended questions Clickthrough rate (CTR), and other engagement data Postponed
Related questions using a Machine Learning model March 29, 2023 April 12, 2023 Using machine learning to recommend more relevant related questions using the question title and tags Clickthrough rate (CTR) Concluded
“Most asked in [Tag]” sidebar module Mar 8, 2023 Mar 16, 2023 Displaying frequently visited/asked questions above Hot Network Questions Clickthrough rate (CTR) Concluded
Related questions within answers list Feb 14, 2023 Feb 24, 2023 Displaying related questions on pages without answers Clickthrough rate (CTR) for Related Questions Graduated
Separating Overflow Blog and Community Bulletin Jan 30, 2023 Feb 9, 2023 Two experiments were conducted for creating a dedicated space for blog content Variant does not perform worse than control, CTR for Community Bulletin Concluded
Moving Related Questions higher on question pages Oct 26, 2022 Nov 22, 2022 Two experiments were conducted to increase the prominence of related questions in the sidebar Clickthrough rate (CTR) for Related Questions and Community Bulletin Concluded

Feedback

When new experiments are released or concluded we’ll update this central post with a call to action for the current community feedback we’re looking for. We ask that you share your feedback on a linked question (link to be added soon) that we will title with the experiment name that we are currently running. We are doing it this way so that we can more easily compartmentalize the feedback for each experiment and take action on the information you’re providing us with.

Our goal in shifting communication on this project is to keep you better informed and make it easier to surface your valuable feedback to Product and Engineering.

33
  • 56
    I mean, the largest problem with the Related A/B test (which I assume prompted this) wasn't that people didn't know where to find it. Or the updates were hard to follow. Both were issues, but the biggest one is rolling it out with no announcement. A pattern that is not at all new to SE and has been repeatedly criticised by the community. Unless you commit to making the proper announcements before releases and stick to this I really don't see what the point of this is. Do note that we've been repeatedly promised better information in the past. Only to see repeat lack of it.
    – VLAZ
    Commented Mar 23, 2023 at 16:59
  • 16
    OK, so should I take this as no commitment for better announcement of upcoming changes? Because being informed about initiatives doesn't suffice. I saw the announcement about the A/B test on Related section. It read that it will run for anonymous users only. And thus I assumed it might at best be released for anonymous users. It wasn't. Nor was there any information what the result was after it concluded and before releasing. The very reasonable point about "Related" not really being related was brought up but dismissed. I didn't see reason to leave feedback. Apparently I should have
    – VLAZ
    Commented Mar 23, 2023 at 17:25
  • 6
    Remember the MSE "roadmap" we were supposed to get, that looked like these?
    – Travis J
    Commented Mar 23, 2023 at 17:59
  • 30
    "Out of the 21,595 responses to the survey year to date, only 11% of participants mentioned that discoverability was one of the most valuable aspects of Stack Overflow." I'm not sure those respondents meant "new feature" discoverability. I think they meant "the answer I'm looking for" discoverability.
    – TylerH
    Commented Mar 23, 2023 at 20:18
  • 9
    I appreciate this effort. I think the most important change that could be implemented, process-wise, is to make a new featured announcement post on Meta (Meta.SO if it is SO-specific, or Meta.SE if multiple sites), at least 24 hours before any new change (like this) is pushed live. Inherent in that change is not using any pre-existing A/B test thread as the announcement that it is live.
    – TylerH
    Commented Mar 23, 2023 at 20:21
  • 6
    @TylerH is 24 hours enough time to prevent a release based on community feedback? Because I don't just want to be told something is happening. I'd rather have a say if and how it does. Although, for the record - do not have an issue with the plagiarism flag. That already involved discussion with mods before it was announced for release. But the Related thing was a mistake based on Related failing at its job. Feedback which I feel would have been crucial and should have prevented rolling out the feature. I feel if SE already made the decision, such feedback wouldn't have mattered.
    – VLAZ
    Commented Mar 23, 2023 at 20:29
  • 6
    My impression is that the useless shoved-in-your-face "[un]related questions" mess is no longer an experiment, but it's here to stay. I'd argue that for non-experiments it would be especially critical to announce substantial changes like this. Commented Mar 23, 2023 at 22:42
  • 8
    I don't know about you, but when I'm looking for an answer (which is what this place is for) and find it (no thanks to your search), I don't go clicking on things, I get back to work. If I don't, I go back to Google search and either try a different link, or tweak my search term. So if content was "more discoverable", if anything I'd expect your apparently universal CTR metric to go down, not up.
    – Dan Mašek
    Commented Mar 24, 2023 at 0:20
  • 26
    The "Measure of Success" listed in that table is not a very good representation of a feature's usefulness... Of course you're going to have a higher click through rate if you're going to be placing links more prominently! That doesn't mean the links are suddenly more useful...
    – Cerbrus
    Commented Mar 24, 2023 at 6:58
  • 4
    Not sure what exactly the problem is and how the solution helps solving the problem. As an overview this Q&A is good, as a place to give feedback terrible. Please keep the feedback localized. An overview announcement is nice, but also a misuse of Q&A. Maybe a new type of content needs to be invented. But in any case, please don't ask for feedback on multiple initiatives in a single place. This will result in chaos. Commented Mar 24, 2023 at 8:51
  • 4
    The response we got to that feedback lead me to believe you were just testing the theory, not actually planning to release it.
    – Kevin B
    Commented Mar 24, 2023 at 15:00
  • 9
    This is now a list of new elements to add to my Brave filter to hide. It is convenient to have them listed in one place rather than having to hunt through the Meta junk to find announcements. Also perhaps change your 'success metric' to Ad Revenue for transparencies sake.
    – pilchard
    Commented Mar 27, 2023 at 13:20
  • 11
    Could you please write dates in a manner that will be unambiguous to the majority of the world?
    – Dan Mašek
    Commented Apr 13, 2023 at 16:20
  • 6
    "calculate similarity or relevance by leveraging an embedding model using titles and tags which converts text into a numerical vector" Unless you demonstrate that the result correlates with actual "similarity or relevance" this is GIGO. I realize you are asking for feedback but what is the justification for what correlation? SO Inc posts & comments have still not given more than this quoted unhelpful vague fragment of an implementation ... of what behaviour/goal? (If you trained to approximate output of the present (bad) Elasticsearch results, you haven't said so, or why that's worthwhile.)
    – philipxy
    Commented Apr 14, 2023 at 6:42
  • 4
    Now I understand the problem. Who is going to read updates to posts? If only there was a way to keep track of these important updates. I just come by to see an announcement that in one week this Q&A will be updated with important information but will I really come back in one week or will I have forgotten then? Who knows. I don't want to re-read the same post over and over again and scan for updates. If only there was a way to highlight the differences to when I was here last time. Commented Apr 14, 2023 at 15:05

9 Answers 9

48

I understand that this initiative is for collecting feedback (from a comment by Rosie):

We want to make it easier for everyone to have awareness of what changes are happening and a way to give actionable feedback for our teams to review.

And more specifically, the focus seems to be to collect feedback from the community.

While a noble goal, I would like to propose two-way communication here. Not only do we need to be informed about the experiments run, but also of what the results are.

Most recently related questions within the answers list were A/B tested. There was no feedback to Meta until after it was released where the community did have a lot of feedback. Like "why is this thing here" or "who thought it was a good idea". This feedback prompted staff into action to update the post with the rather laconic

Update Mar 21: TL;DR: Today, we graduated the experiment for Stack Overflow question pages with zero answers, specifically the variant where three related questions are shown by default, with a link to view more.

During the experiment, we observed a statistically significant 900%+ increase in clickthrough rates (CTR) in both experiment groups. Users in the experiment were engaging with related questions at an exponential rate compared to those in the control group where related questions were shown in the sidebar.

Now we got this announcement which aims to be informative. However, let's examine the list of experiments:

Summary Start Date End Date Description Measure of Success Status
“Most asked in [Tag]” sidebar module Mar 8, 2023 Mar 16, 2023 Displaying frequently visited/asked questions above Hot Network Questions Clickthrough rate (CTR) Concluded

This apparently already concluded a week ago. There was no announcement of it that I can find. The only thing I can see is revision #15 (March 13, 2023) of the original product discovery announcement which adds a single line to the "Currently testing" section:

A/B testing new "Most Asked in Topic" module for anonymous users.

But no topic was made for it. Which is probably why the same revision removed the following text:

Each piece of the initiative will get its own detailed Meta post which will be a venue for feedback and discussion of that specific experiment.

Then revision #16 (March 22, 2023) moved the previously added line to the "Testing concluded" section.


Summary Start Date End Date Description Measure of Success Status
Related questions within answers list Feb 14, 2023 Feb 24, 2023 Displaying related questions on pages without answers Clickthrough rate (CTR) for Related Questions Graduated

As discussed above there was no sharing of results here until after releasing it.


Summary Start Date End Date Description Measure of Success Status
Separating Overflow Blog and Community Bulletin Jan 30, 2023 Feb 9, 2023 Two experiments were conducted for creating a dedicated space for blog content Variant does not perform worse than control, CTR for Community Bulletin Concluded

The last update of the post is:

Update Feb 9: Testing has concluded.

Which means we are still waiting on:

Once the experiment concludes, we will analyze the results and share with Meta.


Summary Start Date End Date Description Measure of Success Status
Moving Related Questions higher on question pages Oct 26, 2022 Nov 22, 2022 Two experiments were conducted to increase the prominence of related questions in the sidebar Clickthrough rate (CTR) for Related Questions and Community Bulletin Concluded

Basically the same as above. The last update is:

Update Nov 22: V2 experiment has concluded. We will share results and next steps soon.


To be clear, if it seems a bit harsh I am criticising lack of feedback here for experiments that probably did not show results in need of changes I think it is important to keep in mind that the Stack Exchange company has repeatedly told us they would try to listen more to the community. And repeatedly broken this promise.

I am trying to hold you accountable for the latest bout of these. Where are the results of these experiments? Why were they not published? It was not one that was missed out but each one.

The latest unannounced release is not even the worst example of this. However, it is one that shows Stack Exchange keeps doing the exact same mistakes we were led to believe are a priority to not make any more.

Perhaps one can point out that there was indeed very little feedback to those announcements. Perhaps the community failed to get engaged. The reception was so weak on the Related questions one (before it the change was released) that it was eventually automatically deleted by failing the criteria to be left alone by the Roomba. For the record the bar is rather low: positive score or at receiving at least one answer would have prevented it.

Let me be the first to heroically defeat my own strawman by point out that without and feedback from the company any feedback the community leaves seems hollow. Perhaps you do indeed want us to participate more. I do actually believe that you hope the community gets more involved in these initiatives. I cannot really speak for all the community but I personally often end up thinking "why should I even bother?". Looking through the list of these experiments I see they all ended with a very consistent lack of final information share. Seemingly enough eventually Stack Exchange gave up even trying and the last experiment is just a single line of text.

6
  • 6
    Thinking about this, I'm coming to the realization that I find the inertia to engage and leave feedback on initiatives simply too high. If we engage, it's a fair sentiment to presume we'd have a dialog, but I don't recall the last engaging dialog we had on a feature or design they wanted to roll out since the profile page (and even then, we had to kinda drag it out of 'em).
    – Makoto
    Commented Mar 23, 2023 at 22:26
  • 2
    I guess it keeps us in this unwinnable position, and I don't really believe that showing these results now would "fix" it. It speaks to a deeper issue with how we communicate with Stack Overflow, in all frankness. It's something we were hoping would have been addressed with the coming of Philippe + 12 months + α, but I don't think we're there yet.
    – Makoto
    Commented Mar 23, 2023 at 22:28
  • 3
    I did get a direct response, but it was to say the quality concerns of related questions, the thing we're moving around and trying to get people to click, was out of scope. ¯\_(ツ)_/¯
    – Kevin B
    Commented Mar 24, 2023 at 15:28
  • 3
    @KevinB "quality concerns"?? "Measure of Success [...] Clickthrough rate"
    – philipxy
    Commented Mar 27, 2023 at 10:01
  • 2
    I should also point out that "feedback sink" is a widely deployed strategy to get rid of social unrest without compromising anything. It's used everywhere from state politics to PR. When feedback is completely and repeatedly ignored, I don't see how it can be unintentional. Commented Apr 16, 2023 at 4:03
  • @polko see the Spotify Community, Microsoft Feedback Hub, and then some.
    – CodeCaster
    Commented Apr 30, 2023 at 17:13
45

In all honesty, your progression seems backwards. You are promoting content which you already know doesn't answer people's questions (it has been one of the major pain points for the almost 12 years I have been active here) instead of fixing that first.

If you can magically make your machine learning suggest properly relevant links, that will basically invalidate all the experiments you have performed with "relevant" links so far.

Secondly, as pointed out in comments, click-through rate is a horrible measurement for what you say your objective is. How about measuring the number of upvoted answers instead? That's a reasonably confident indicator that somebody actually found something useful at the end of their clicking (and then the fewer the clicks they needed, the better, if you can measure that too).

1
  • 5
    Thank you for your feedback. We are also in the process of looking at other metrics by which to measure success of an experiment. CTR was only the first data point we measured, but we plan on tracking other metrics in future experiments as we agree we want a fuller picture of engagement.
    – Bella_Blue StaffMod
    Commented Mar 29, 2023 at 15:42
32

For each of those experiments, we’ve created a post here on meta that discusses the experiment and links back to the initiative. While this simplifies things in the sense that each topic has a dedicated post, it also leaves feedback dispersed.

I didn't see that as a problem and I'll try to explain why I think that.

In order to centralize feedback [...] — we're rolling out this post as the new format [...].

Isn't that one of the very definitions of a question lacking focus? I can't see how it would be better for feedback for multiple experiments to be written as answer posts to a single question about all the experiments. (or have I misunderstood what you are saying here?)

I'd prefer for updates and their corresponding solicitation for feedback to each be their own Q&A- each one tagged and given some time for the community to discuss.

  • One proposing the experiment asking for feedback on the way it will be conducted
  • Once for announcing the beginning of the experiment and soliciting feedback on it
  • One for sharing results and discussion on the results
  • One for proposing upcoming changes based on the results and discussing those changes
  • One for announcing the full release of those changes and soliciting feedback on the full release

I don't see fragmentation as being a problem with the above general flow of discussion because each one is seeking feedback from the community on a specific step. If anything, I see it as less helpful by virtue of being less organized if they were to be mixed together. It would be very annoying for me to scroll through interleaved answer posts for multiple stages of multiple experiments to read and engage with community feedback. I don't find the "dispersal" of these staged discussions problematic. Just do what the recent staging grounds and collectives posts did (and that our FAQ does): have a main post indexing each stage of the discussion for each experiment.

If anything, the problem with the way things went recently is that you didn't separate Q&A for each stage into its own Q&A and opted to edit-hijack Q&A for an existing stage for a later stage: you used the experiment results report Q&A to announce the feature's full release, (and didn't feature the experiment results question post when you first posted it), and ended up getting a bunch of feedback about the release there. Quoting @Travis J from the comments there:

Please, please, please stop just editing questions with "updates" that change the entire demeanor of the post. This is not a forum. The change to turn this on is drastically different than the original discussion, and there is no way to answer the now live version of this with issues or other problems observed in the wild. Releasing this as a feature should have its own post, which describes the results of the A/B test, gives some sort of nod to community feedback (or any feedback), explains expected outcomes, and allows for a discussion on the widespread integration.

19

Moving forward, we'll announce new experiment releases as an update on this post and the table below, instead of a standalone Meta post for each experiment.

This is a terrible idea. It will hinder progress, hide feedback, and create stale content. It is absolutely not how any of this works.

Paired with individual posts, the tag wiki is best suited for this type of approach, it was literally designed for this type of information. It is visible, on each post, and easy to access.

If you look at you will find this:

There is no tag wiki for this tag … yet!

Tag wikis help introduce newcomers to the tag. They contain an overview of the topic defined by the tag, along with guidelines on its usage.

All registered users may propose new tag wikis.

(Note that if you have less than 20000 reputation, your tag wiki will be peer reviewed before it is published.)

Specifically, this is the best place for overall information relevant to the tag itself such as the table and explanation of the project.

  • Explain the project scope in the tag wiki
  • Place the table, with "The Experiments" in the tag wiki
  • Place a list of iterations in the tag wiki

We’ll be updating the below chart when experiments have been released or have concluded.

These updates should be to the tag wiki table as described above. The details of the updates should be in posts which the tag wiki should reference.

In addition to making sure that a relevant overview of the project is available and can be directly referenced, posts should be created for new features, discussions, updates, and changes. Posts do not need to be massive in scope, and can be relatively mundane.

11
  • 10
    (But tag wikis get like, no traffic. They're pretty useless when it comes to this. No one really pays attention to them. Getting people to pay attention to them is a lot harder than posting a new question for each thing.)
    – Makoto
    Commented Mar 23, 2023 at 19:44
  • 2
    And note, I'm just playing Devil's Advocate. Special cases aren't special enough to break the rules, but practicality beats purity.
    – Makoto
    Commented Mar 23, 2023 at 19:45
  • 10
    I'd rather not have this information stuck in a tag wiki. I'd prefer an actual post I can follow to get updates.
    – VLAZ
    Commented Mar 23, 2023 at 20:03
  • I can see how from this description it came across as having the information "stuck" in a tag wiki. However, that is not the only place for this, but that is the place for a stale table of what happened when which seems to be the topic here. The table should be kept in the wiki, the updates should be posts for discussion or whatnot, each as an individual post, and the table should link to the posts.
    – Travis J
    Commented Mar 23, 2023 at 21:47
  • @Makoto - I wasn't really trying to infer that no individual posts be made, so much as the home for a table with constant updates is in a tag wiki. I tried to edit the clarification into this answer.
    – Travis J
    Commented Mar 23, 2023 at 21:56
  • @TravisJ: No no, I'm fine with the solution in all actuality, I'm just playing Devin's Advocate here. I think that tag wikis are the right choice here, but I can see the practical decision as well.
    – Makoto
    Commented Mar 23, 2023 at 21:58
  • 1
    Tag wikis are such a neglected area, but if you look at some of the old school well maintained ones (JavaScript, C#), there is a whole wealth of information there.
    – Travis J
    Commented Mar 23, 2023 at 22:07
  • 2
    @TravisJ I regularly use the JS wiki. However, I don't really expect to have to keep checking it to keep myself up to date. The information there is well-established, so I can have quick access to it. A table that will keep being updated I will keep having to go look up every...what, few days? Weeks? An uncertain amount of time, thus the only way for me to be up to date is to keep checking all the time. Or we can have a notification going out to all interested. Consider meta.stackexchange.com/q/59445 or meta.stackexchange.com/q/386360 as examples of maintained tables.
    – VLAZ
    Commented Mar 23, 2023 at 22:44
  • 7
    "It will hinder progress, hide feedback, and create stale content". Sounds like [by-design] to me. Decrease the amount of feedback (by virtue of people not finding these future "announcements"), and let the little feedback that goes through stay tucked away in some dump of a Q&A thread. "Need to know" basis and all that. Commented Mar 23, 2023 at 22:45
  • 4
    Upvoted for the first sentence, but I agree with the other comments that putting info into the tag wiki would be close to useless as nobody reads those. Given that there is already a lack of communication from SE in general, advising them to put parts of their communication in a place nobody is looking at doesn't seem like a great idea.
    – l4mpi
    Commented Mar 24, 2023 at 9:33
  • 8
    I have created the tag wiki for this project :)
    – Bella_Blue StaffMod
    Commented Mar 24, 2023 at 16:35
17

Using machine learning to recommend more relevant related questions using the question title and tags

This is a big change that should be its own featured post & not an unpublicized unexplained edit in a post full of many other (related) issues with many unrelated answers.

Clickthrough rate (CTR)

Still a bad "Measure of Success".

1
  • 9
    When the machine learning experiment goes live tomorrow we are going to link to a post for feedback on that experiment, and ask the moderators to move the featured tag from this post to that one.
    – Sasha StaffMod
    Commented Mar 28, 2023 at 17:04
14

We plan on sharing the results of that experiment on or around April 19, as an edit to this post

PLEASE make a dedicated post for it!

Make a dedicated post for it, and then just link to that dedicated post in the question post here. That way we can have discussion on the results and plans for moving forward in that dedicated post, without it getting jumbled up with everything everywhere all at once here. Your users really want to engage with you in a way that is focused and not cluttered.

Will there be a way to opt-in to the next ML experiment?

I think your long-time contributors can give really valuable feedback on the ML-recommended posts, so it's really a shame that they're all given regular odds of being in the control or test group. And I think it was quite a silly state of affairs that because long-time contributors who made up the majority of people giving you feedback in the first ML experiment couldn't tell what group they were in, you were getting a lot of feedback saying that things were as bad as they had always been (which I'm guessing/hoping is because they were in the control group).

If what you care about is having enough percentage of users in the control group, why not allow users above a certain rep threshold to toggle back and forth between being in the control or test group, or toggle seeing both simultaneously?

Looking at the reputation league, the vast majority of even just registered users are below 200 rep. I.e. if you let users above 200 rep toggle their group, it will still probably have negligible effect on the percentage of users in the control and test groups.

I'm feeling a little underwhelmed and a tiny bit played-with looking at the changelist

Just like the last experiment, we calculate similarity or relevance by leveraging an embedding model using titles and tags which converts text into a numerical vector.

My brain: "Oh. So it'll still have all the problems with the limitations of only using the title and tags. They're not going to try anything else?"

In the previous experiment, the recommended questions appeared in a randomized order, so we hope that this will improve the reliability of the list of recommended questions.

Reading this this, my brain thought: "Wait- Why didn't you just do this from the start? Were you trying to do that thing people do with surveys to prevent bias from ordering? But... isn't the whole point of this feature to put the most-probably-useful/related thing at the top of the list to serve your users? Why care about 'bias'?"

There's a joke in the web development world that if you want to make your clients love you, start by adding arbitrary slowdown to their site experience. Insert some setTimeouts, etc. Then slowly reduce those arbitrary delays and watch as they praise you for making the website faster. The part of me that wants to assume the best of people thinks "Nah they wouldn't do that". The other part of me is confused and a little bit suspicious.

What does "Reliability" mean with respect to the Related questions list?

Using machine learning to improve the reliability of the list of recommended questions

Reliability is a measure of how well something continues to do its job. But it's not clear to me whether we even pinned down what that job is, and even if we did, whether we were ever any good at it (Okay to be fair, there's also a pretty highly upvoted post from 2014 that praises the Related Questions listing as actually being very useful, but I don't think that hold true today).. So what exactly do you mean here when you say "reliability"?

5
  • 2
    "a way to opt-in" -- especially after they moved that section to a more obnoxious place and gave some of us very good motivation to just hide it with an adblocker. A 30% chance to see something different (along with the dubious chance that it might actually be any better) is hardly worth the pain of making it visible again.
    – Dan Mašek
    Commented Apr 14, 2023 at 11:04
  • 1
    We are not able to do a separate post for the results of each experiment, as it distributes feedback too widely. But we will continue monitoring this central post as well as the linked posts for each experiment, for questions and feedback from the community. In terms of the experiment groups, it is a best practice in this kind of research to randomize the participants in the experiment and control groups. We deeply value the perspective of long time contributors, but we are not able to provide an option to self select into the experiment.
    – Sasha StaffMod
    Commented Apr 14, 2023 at 21:26
  • 2
    The previous experiment took the same type of inputs (title and tags) as the control experience, so we could evaluate the differences in the ML model against it. This next experiment builds on that by exploring the impact of changing the sort order, while keeping everything else the same. We do have plans to evaluate other inputs/metadata in the model in the future, but we don’t want to introduce too many variables at once and lose track of the effect of each new change. We are taking an iterative approach, and there are many more experiments to come.
    – Sasha StaffMod
    Commented Apr 14, 2023 at 21:27
  • @Sasha just updated with another question.
    – starball
    Commented Apr 15, 2023 at 7:03
  • 5
    @Sasha "But we will continue monitoring this central post as well as the linked posts for each experiment" You might be able to monitor that, but we may not. We have problems seeing the updated information because we do not look that often at it. Maybe this Q&A system is not so suitable for project management. If you cannot do separate posts that well and we cannot follow updates of single posts that well, then that's already the best we all can do. Commented Apr 17, 2023 at 7:21
12

I've updated the chart to reflect that there is a new experiment launching on Wednesday. When it goes live, we'll link to that post for feedback on that experiment, and ask the moderators to move the featured tag from this post to that post, per the community's request for the current experiments to be the featured post.

We've heard your feedback that you'd like as much heads-up as possible. Moving forward, we're going to try to give you at least several days' heads-up when a new experiment is coming up.

2
  • 2
    Excellent! Visibility is very important :D
    – Cerbrus
    Commented Mar 27, 2023 at 22:11
  • 11
    You're talking about giving heads-up for experiments - how about for the actual rollout of the features? And more importantly, will that amount to a simple notification "we're going to do this" or will you actually address comunity feedback then? For example, if community reaction for your heads-up is "don't roll out this garbage", are you going to ignore it like the last n times?
    – l4mpi
    Commented Mar 28, 2023 at 10:45
10

I didn't see the update that was made to this post on the 19th until just now (and this being the only new/edited answer post since, I don't think I'm the only one). I'm still not a fan of this format for discussion. Q&A here was not meant to work like this.

Are users commenting more?

Yes, we saw a similar trend as well with a 26% lift.

More commenting often means something is wrong. As a refresher, the purposes of comments are:

  • Request clarification from the author; (something's wrong: clarity)
  • Leave constructive criticism that guides the author in improving the post; (something's wrong: Ex. correctness / something else)
  • Add relevant but minor or transient information to a post (neutral)

The fourth option (the list below that list) is that the user doing something else (Ex. saying thank you), in which case something is also wrong: the user doesn't know what comments are for and are using them improperly.

I suggest that you stop using more comments as a measure of success.

We have decided to postpone the next experiment. When we updated the model we found something that was not acting as expected and did not want to move forward until it was resolved.

What was/is it?

Our next experiment will involve updating the sort order of related questions recommended by utilizing the machine learning (ML) model.

This is how it should be in my view. Why would you waste your users' time by ordering the list by anything other than what seems to be the most related?

How it works is that [...]

The transparency of mechanics this time is appreciated :)

Whether engagement attempts were made (i.e. anonymous users or registered users with or without enough reputation attempted to vote or comment)

Again, I suggest you don't use more comments as a measure of success.

The amount of time spent on pages

I'm confused about how this correlates with success. Do you want more time spent on pages? And what about questions where the answers are short, simple, correct, and few? (good), or questions where the answer are many (neutral), long (neutral), some outdated (not good), and some incorrect or poorly explained (bad)?

We heard your feedback regarding including inline feedback during these experiments. While we will not be collecting that data with this experiment, we are working incorporating a prompt to collect qualitative feedback when users interact with related questions.

Glad to hear it. Again, feel free to use what I have proposed.

Just like the last experiment, we will be asking that you share your feedback on a linked question (link to be added soon) that we will title with the experiment name that we are currently running.

Glad to hear at least that will have its own post.

Thanks to your invaluable contributions, we have already started talks on ways to incorporate some of the ideas on how better to achieve relevancy, [...]

Can we get a more up to date response to this then?: For the purposes of the "Related" section of the Q&A UI, what definition of "related" aligns closely to what users of Stack Overflow find value in?

0
4

I think a few clarifications to the update could make the post more understandable, imo.

Related questions recommended from the Machine Learning (ML) model generated a whopping 155% increase in clickthrough rate (CTR) in the aggregate.

How do you identify the effects of the ML model? The related questions were moved from the side to be directly underneath the question. How much of the 155% increase be attributed to such design decision (or is the increase so even after controlling for it)? If you used a specific statistical model, it might be useful to mention it.

Yes. Of the users who are able to vote, there was a 23% lift in overall votes combined (upvotes and downvotes).

It might be useful to include the magnitude of the difference. A bare percentage says very little. Does 23% translate to 23 votes or 2300 votes? Is it statistically significant etc. Again, how much of it can be attributed directly to the ML model?

When we updated the model we found something that was not acting as expected and did not want to move forward until it was resolved.

What was the issue? Will you share it in the future? Or would disclosing it compromise future experiments?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .