7

Please check out this post for more context on the Content Discovery initiative where we go into detail about the background, overview, and objective of this project as well as information on past experiments and information on experiments going forward.

The current experiment we’ve released is incorporating machine learning (ML) to improve the relevance of related questions.

During this round of experiments, we are going to be incorporating machine learning (ML) to improve the relevance of related questions. We saw all of your feedback on the A/B testing related questions within the answers list post and feel that this will be an improvement in the suggested content.

This experiment will launch today and will run for at least two weeks. During this time, the questions recommended by the ML model will be available on Stack Overflow question pages for both registered and anonymous users.

We will also be measuring clickthrough rates (CTR) of questions from the control group (powered by Elasticsearch) and the experiment group (powered by ML). The participants are chosen at random and we do not have control over who gets the test or who doesn’t at this time. You could try a different browser but we cannot guarantee that you'll get bucketed. However, as we have plans in the future to iterate on the model for recently posted questions, they will be exempt from the ML model.

Feedback

We are looking to find out how this change has affected the questions that are being suggested, and if they are more relevant. Please let us know if you notice increased relevancy. We are monitoring this closely and will make adjustments to the model based on your feedback. Our goal is to make the related questions as relevant as possible to users, so we really value your feedback which can help us understand if we are moving in the right direction.

35
  • 5
    Quick clarification: does this apply to all questions or only to unanswered questions? Or that's something that's also tested and the user shouldn't know in the first place? :)
    – Andrew T.
    Commented Mar 29, 2023 at 16:40
  • 6
    @AndrewT. Thanks for asking! This applies to all questions :)
    – Bella_Blue StaffMod
    Commented Mar 29, 2023 at 17:04
  • 42
    The stated objective is to "improve the relevance of related questions" why then is the metric measured click through rate? How is the CTR related to the objective? Commented Mar 29, 2023 at 17:04
  • 24
    @AbdulAzizBarkat While in the long term a better metric is needed (discussed more in VLAZ's answer and the comments there), right now the related suggestions are often so clearly unrelated that getting suggestions plausible enough to click would be a huge improvement.
    – Ryan M Mod
    Commented Mar 29, 2023 at 18:26
  • 13
    How do you tell the difference between "none of the links look useful" and "I got my answer on this page, so no need to find related questions"? Commented Mar 29, 2023 at 19:02
  • 16
    you've gotten four answer posts so far and none of them are really focused on the kind of feedback you've solicited here. It's because you "skipped" the step that all these answer posts are hoping you would have taken. "One proposing the experiment asking for feedback on the way it will be conducted"
    – starball
    Commented Mar 30, 2023 at 2:23
  • 6
    @AnonCoward The proportion of users who got their answer on the page and thus have no interest in the related questions should be the same between the control and test group, so any difference in click-through rate should be attributable to the change in the link selection algorithm.
    – Ryan M Mod
    Commented Mar 30, 2023 at 4:27
  • 5
    Fair enough, I guess I don't see the point still. The only time I've ever clicked on of these links is to answer "what the heck, how did that link end up related to this question". Maybe Noise A is better than Noise B, but good luck proving it in any sense if you can't know if people are getting the answers they seek out of the links.. Commented Mar 30, 2023 at 4:37
  • 3
    Is this supposed to be live right now, for all users? Or is it A/B tested, and if so, how do I know I see the ML suggestions? Asking because the related questions look just as trash for me as they always did, or maybe even worse now. Just clicked a random python question on main, and not only is the "related" list full of the usual suspects of unrelated high-score python questions, there's also an unrelated JS question in there...
    – l4mpi
    Commented Mar 30, 2023 at 7:53
  • 7
    Related questions was never stellar, but usually at least few of the top linked questions were actually related to the asked question. I browsed around today, and my first impression is that results are horrible. On some questions not even single one linked question was actually related, on some questions related picks are very poor (there are better suggestions). Most notable problem is irrelevant technology (wrong tag), but also picking questions that only have technology (tag) in common, but otherwise are completely random.
    – Dalija Prasnikar Mod
    Commented Mar 30, 2023 at 9:11
  • 6
    Nice initiative. Could you use ML also for possible duplicate question detection then? There should be a huge training set available. Commented Mar 31, 2023 at 7:47
  • 9
    @MarkSetchell To be bucketed means that you will be involved in the experiment. There is no pain, but there will be some mild pressure, then a light popping sensation, followed by an overwhelming sense of euphoria. If you begin to exhibit symptoms such as an intense desire to learn macrame, or an insatiable hunger for thin mints please contact us immediately.
    – Bella_Blue StaffMod
    Commented Mar 31, 2023 at 13:13
  • 4
    so.... to determine whether or not a given question is relevant i'd need to click through to it in any case where the question title isn't clearly unrelated... if you're measuring success based on ctr...
    – Kevin B
    Commented Mar 31, 2023 at 15:00
  • 4
    What i'm gathering is it's not really considered important whether or not SME's see the results as relevant to the question asked, rather, we're just gonna look at data and make some correlations.
    – Kevin B
    Commented Mar 31, 2023 at 15:08
  • 6
    In JavaScript tagged questions I have yet to see a single related question that matches even when I know there are duplicates and there should be one. Commented Mar 31, 2023 at 20:04

15 Answers 15

75

Can we get some transparency into how this is working?

  • What kind of model is being used?

  • More importantly (to me), how was the model trained?

    • What inputs is it given for a post? What parts of the question? Do answers get taken into consideration? Do manually linked posts get taken into consideration in any way?

    • What criteria was used to qualify positive or negative feedback (to tell it how useful / related something it proposed as "being related" actually is)? Were linked questions used at all in the training? (it seems to me like they would be a good place to start)

    • It seems like post score is still being used as a significant contributor to whether something is considered "relevant". I'm very curious to see what would happen if it was given almost zero consideration (aside from a hard binary separation of favouring non-negatively scored questions over negatively-scored questions).

  • In relation to Dalija's post, what influence do tags have on the ML model? I would hope that popular, non-meta tags (meta tags are tags like , ) are used as hard filters to what is counted as related or not at least for some percentage of the "related questions" (I suppose using it for 100% of them could possibly(?) be disadvantageous in preventing "escaping" from less-relevant tags).

Will data about what the model recommended be released later?

You mention that you will still be measuring click-through on the control group. I can understand why you might not want to show regular users which "related posts" come from the control group and which come from the ML model (to avoid tainting user behaviour for the experiment), but will any data about which links were recommended by the ML model for which posts be made available after the experiment?

Why not give users the ability to give explicit, inline feedback?

I agree with what VLAZ said. Clickthrough is not a generally good way to judge whether something really is related or not, because it's hard or impossible just to know from reading the title. You often have to actually read the question body and/or answer posts.

Have you considered giving users a way to explicitly indicate whether a "related post" is useful as a "related" post to them? Ex. giving them vote buttons (separate from actual votes on the post), or putting a feedback banner at the top of the post when clicked-to from the related posts sidebar that asks something like "how relevant was this related question given the post it was recommended from? {not relevant, relevant, very relevant}"- or whatever adjective actually represents the goal.

Do we have a formal definition of what "relevant" means?

  • Does it mean "I bet you were or will later wonder X given that you are wondering Y"?

  • Does it mean "I bet you'll find question X fun and interesting"?

    If this is the case, then I'd suggest that this should be given its very own UI separate from the Q&A UI. Make a full blown "explore interesting, loosely related posts" UI.

  • Does it mean "X has already been asked here"?

    In that case we should be focusing on the suggested duplicates in the Ask-Question UI- preventing duplicate questions from being asked in the first place, unless the asker really thinks a new signpost is needed. (and even that's a bit dubious (see Is it encouraged to intentionally post duplicates for searchability?))

  • Does it mean "these questions have some same words, similar tags, and are scored very highly"? (I'd have to wonder how useful this could be to anyone)

I'm still kind of confused by the goal. I don't really understand why we would be honing in on any problem other than suggesting possible duplicates. I have never thought to myself "hm. I'm really bored. I'll just go on a link-clicking spree to explore random Q&A pages on Stack Overflow", or "clicking these suggested links is going to be so much more useful to me in finding the answer to my question than using google search."

People come to Stack Overflow to get answers to a question. I've never heard of someone who uses Stack Overflow as something fun to browse. I can see why someone else would enjoy seeing Hot Network Questions, and I can see people enjoying random-link-exploring sprees on other network sites like our gaming, anime, movies, etc. sites, but not why they'd care about "related questions" on Stack Overflow unless they're really designed to be possible duplicates, or predictions of future questions one might have in the near future based on the current question being looked at.


I've ended up elaborating on my thoughts here in dedicated Q&A posts. Feel free to chime in:

20
  • 19
    I think this is a very important point. Any machine learning model is only as good as the training data that it is given, ideally that would be at least ten thousand or so question-question pairs that are manually verified to be relevant to eachother. I however doubt that SE has such a dataset. Moreover, there are dozens of different ML algorithms and literally millions of hyperparameter combinations, and some setups will work while others don't. So I'm afraid SE might conclude that "ML doesn't work" while it is just a matter of finding better data and trying different algorithms.
    – Marijn
    Commented Mar 30, 2023 at 8:07
  • 14
    @Marijn: They have something pretty close to such a data set, though: dupe-closed questions and their dupe targets. It's not a perfect fit, but it might be good enough to train a model that improves on the current state of affairs. Commented Mar 30, 2023 at 11:06
  • 12
    In this experiment, we are leveraging an embedding model which converts text into a numerical vector representing the semantic information. An embedding is generated for every Stack Overflow question (over 20M) using the question titles and their corresponding tags. Using semantic search, a list of similar questions is returned. Similarity or relevance is defined as the measure of distance between the vectors.
    – Bella_Blue StaffMod
    Commented Mar 31, 2023 at 17:15
  • 4
    This first version of the model incorporates the same features that Elasticsearch uses: question title and tags. Based on the results of this experiment, we will determine additional features and data points to include in the model such as the question body. This will require re-training the model to generate a new set of vectors to determine the new set of related questions.
    – Bella_Blue StaffMod
    Commented Mar 31, 2023 at 17:16
  • 5
    With this first iteration, we are not looking for inline feedback, but it is something we may consider in the future. And yes we do plan on sharing the results on Meta once the experiment concludes, and we analyze the results. :)
    – Bella_Blue StaffMod
    Commented Mar 31, 2023 at 17:16
  • 9
    @Bella_Blue (forgive me for my ignorance- I know next to nothing about ML) where's the "learning" element? what I'm hearing just sounds like... a single round of math. Also, I'm a bit surprised to hear that both the ML model and elastic search only use the title and tags. It made me think "no wonder they're so bad" (not intended to be offensive). Title and tags are very little to go off of. And titles are often lacking in descriptiveness. Is there any particular reason that highly-scored questions seem to be favoured?
    – starball
    Commented Mar 31, 2023 at 20:33
  • 4
    @Bella_Blue How does the model know, in the training stage, if the questions are in fact similar for purposes of the vector space? Is it leveraging existing links/duplicates?
    – Ryan M Mod
    Commented Mar 31, 2023 at 21:02
  • 1
    I've never heard of someone who uses Stack Overflow as something fun to browse. What about question answerers? Admittedly, I almost never browse related links however.
    – QHarr
    Commented Apr 1, 2023 at 7:22
  • 2
    @QHarr when I said "fun to browse", I was thinking of "fun to browse (without a purpose / end goal in mind other than to "just browse")". I treat answering itself as a (separate) end goal from the act of browsing to find something to answer.
    – starball
    Commented Apr 1, 2023 at 7:35
  • 5
    @Bella_Blue - "the model incorporates the same features that Elasticsearch uses"... has there been any progress moving away from Elasticsearch? It has been acknowledged as not being functional by Stack Overflow numerous times in the past, and plays a very large role in hindering discoverability of content on the site.
    – Travis J
    Commented Apr 2, 2023 at 3:26
  • 3
    I assume I'm in quite the minority on this, but personally I do sometimes browse stack overflow purely for entertainment. Commented Apr 3, 2023 at 20:22
  • 5
    Negative-score questions sometimes have useful answers. Like when someone writes an answer that makes a good canonical for that specific problem, but posts it under yet another duplicate or near-duplicate question. Our modern standards towards "research effort" and so on often lead to downvotes on such questions, even if it's not a huge mess that will distract future readers badly. In such cases the answer alone is all future readers need, regardless of a confused question, so a decently upvoted and linked-to answer shouldn't get rejected based on a negative question score, I think. Commented Apr 3, 2023 at 22:41
  • 1
    "fun to browse" to me, it's more like, "fun to moderate" considering it's not rare for questions with a bad title to be shown...
    – Andrew T.
    Commented Apr 6, 2023 at 4:57
  • 1
    @AndrewT. nice use-case. but for the average user, probably not even remotely on their radar of things they want to do here :P
    – starball
    Commented Apr 6, 2023 at 4:59
  • 1
    @Bella_Blue You still have said nothing useful about what the ML is trained on.
    – philipxy
    Commented Apr 12, 2023 at 10:59
60

The way I can tell if a question is related is to click on it and examine the content. The title alone is often not enough to rule a question completely irrelevant or totally relevant.

Would these clicks be accounted for when measuring the success of the experiment? Would my act of verifying the link be counted as if it was more relevant even though I was trying to find out at the time?

Conversely, some questions I know very well and do not need to click on them. Would they be deemed less relevant even if I think they are?


Pre-emptively: I am very well aware I can visit a link without clicking it, thus not disturbing the statistics. But what about everybody else - I doubt all users would just change their entire workflow to make sure the experiment results are not tainted by their behaviour.

14
  • I think people aware of this experiment would generally abstain from clicking these links for the duration. I think the idea is to gauge click rate from people unfamiliar with these canonicals.
    – Dharman Mod
    Commented Mar 29, 2023 at 16:24
  • 24
    @Dharman But that completely missed the point made by VLAZ. If the idea is to improve relevance, then the CTR is completely irrelevant in determining if you've actually improved the relevance, and thus they should stop using CTR as a measure for success.
    – mason
    Commented Mar 29, 2023 at 17:26
  • 15
    I think a good solution to this problem would be looking at things like upvotes cast as well. Although. right now, CTR is probably a decent proxy metric just because the current recommendations are so bad that you definitely don't need to visit the page to know if they're relevant or not. Getting to plausibly related questions would be a huge improvement over the current state of clearly unrelated questions.
    – Ryan M Mod
    Commented Mar 29, 2023 at 18:24
  • 3
    On the other hand, if this initiative is successful, and links are noticeably more related and useful, then I would absolutely expect an increased CTR. Clearly the CTR alone doesn't adequately account for "perceived usefulness", but I don't think the metric is quite as flawed as you hold; it just needs to be a useful proxy for the outcome here, and it seems like one to me. It seems very reasonable to assert that if the outcome with ML turned on is actively worse, then the CTR won't actively increase in the experiment.
    – zcoop98
    Commented Mar 29, 2023 at 19:15
  • 9
    I do not expect staff to actually care about this answer, or anything else about how the metric is poorly chosen. I only expect them to care about what they explicitly solicited - answers telling them whether the results subjectively actually are more relevant, rolling in over the next two weeks - and even then, only minimally. CTR is a) something they can actually measure objectively and b) aligned with their incentives, so of course the company can be expected to optimize for it. Commented Mar 30, 2023 at 6:26
  • 1
    Agree in principle, but: The related questions I see are still so bad that it is easy to see they are unrelated from a glance at the title (e.g. opened random python question about pylance, first 3 "related" are about random selection from a list, creating a singleton, and using pip/requirements.txt). If you are seeing related questions where you would have to click on the question to tell if it is related or not, that would IMO already be a huge improvement over the current suggestions ("good enough" would be a different question).
    – l4mpi
    Commented Mar 30, 2023 at 8:19
  • 1
    @l4mpi for some questions, yes, the title is enough. But not all of them. Some times the title describes the same thing but the body quickly reveals it to be unrelated due to different context. Or vice versa - the title seems unrelated but the body shows it is the exact same as the current one. So title alone could work but also could not work.
    – VLAZ
    Commented Mar 30, 2023 at 8:22
  • @VLAZ unless that only applies to a very small subset of related questions (in which case I would question its importance), it already sounds superior to the "related" stuff I see when I click on a random python question. There might be one or two questions in the list where I'm unsure if they're entirely unrelated or just very distantly related, but ones where the title seems to describe something similar are extremely rare. It's even visible just by score alone - the vast majority of "related" questions look like a high-score list of the main tag and often the lowest score is around 3k.
    – l4mpi
    Commented Mar 30, 2023 at 9:42
  • 1
    @l4mpi sorry, I was talking more in general about titles. My experience with current related system is that the true positives are there more as a fluke. Usually because it's just a high-score question which is linked to other unrelated questions all the time. But happens to work from time to time. However, with the ML suggestions maybe (hopefully) this changes. Yet I would not be able to verify using the title alone. At least not always.
    – VLAZ
    Commented Mar 30, 2023 at 9:46
  • 3
    "Would these clicks be accounted for when measuring the success of the experiment?" Yes. But a few clicks from Meta won't sway the results. We have lots of users already in the experiment.
    – Bella_Blue StaffMod
    Commented Mar 30, 2023 at 12:45
  • 3
    "Would my act of verifying the link be counted as if it was more relevant even though I was trying to find out at the time?" Yes. But again, it shouldn't affect the results because the number of users who come across this Meta post and are aware of it being an experiment is significantly small compared to the 30% of users who are in the experiment.
    – Bella_Blue StaffMod
    Commented Mar 30, 2023 at 12:46
  • 3
    "Conversely, some questions I know very well and do not need to click on them. Would they be deemed less relevant even if I think they are?" If something doesn't happen (i.e. click) we don't have any information to go off of. A click lets us correlate other activities that occurred on a related question page.
    – Bella_Blue StaffMod
    Commented Mar 30, 2023 at 12:47
  • 3
    @Bella_Blue "If something doesn't happen (i.e. click) we don't have any information to go off of." OK, so scenario - I see a canonical duplicate in the Related section. I right-click it -> copy the link -> vote to close the current question as a duplicate of the related one. That is, not something taken into account unless I actually open the question. Yet, it seems to me it would be information to go off of.
    – VLAZ
    Commented Mar 31, 2023 at 15:09
  • 7
    @Bella_Blue, your comment seems to say that only a subset of users are included in the experiment, which I take to mean that only that subset are seeing related answers lists generated with the help of the ML model. In that case, why are you asking everyone for feedback? How is it worth my while to spend time writing an answer explaining that related questions lists still exhibit poor relevance if the likelihood is that that's because nothing has changed for me? Commented Apr 2, 2023 at 16:25
41

I am not sure if this is what leads to what I am seeing if I want to answer this new question:

Question title

Shown related questions

Anyhow, that's the worst related questions I have seen so far. Normally, when I ask a question, there is a lot of examples of questions, which might even be duplicates, so first I thought this is a new UI to make it easier to find duplicates. But none of the related questions seems to have anything to do with the question... I mean they are probably all Python, but I think none of them has the Plotly tag.

10
  • 16
    Maybe they finally incorporated the feedback that related questions should have the same main language tag... but indeed this is really bad. Note that all these questions have very high scores and are very old, and each of them has 90+ questions linked to it. It almost seems like the ML algorithm is trained to return questions that are in general related to other questions, which would be dumb. However, for this comment I clicked almost all of them, so the CTR stats will show that the list was very good :)
    – Marijn
    Commented Mar 30, 2023 at 12:31
  • 1
    "so first I thought this is a new UI to make it easier to find duplicates" related: my answer post asking "Do we have a formal definition of what "relevant" means?", and also suggesting making all non-negatively scored questions have the same weight to the model.
    – starball
    Commented Mar 30, 2023 at 17:23
  • 7
    Thank you very much for this feedback. This is exactly what we are looking for as we run this experiment.
    – Bella_Blue StaffMod
    Commented Mar 30, 2023 at 18:54
  • 2
    But all the titles have letters in them, surely they're related?
    – pilchard
    Commented Mar 31, 2023 at 23:27
  • 1
    That all the questions on the list are (probably) about Python is an improvement (supposing that Python is the subject of the question to which they're purported to be related). Commented Apr 2, 2023 at 16:45
  • Amen! For bash and other *nix cmd-line tools as the key element in the question, the new related questions at the bottom are complete nonsequitors to their parent question above. Ironically, the old "Related Questions" that used to appear below "Hot Topics" and other sections on the right were much better. I could often find a duplicate of the asked question there. Finally, I find the new layout, with the Related Qs below the main question to be (very?) intrusive to my reading flow. If I've arrived late to a question I want to start reading real answers without having to parse intrusive text
    – shellter
    Commented Apr 3, 2023 at 15:21
  • 2
    @shellter there's a lot of issues with your statement. one, the old "Related Questions" that used to appear below "Hot Topics" is the same list you're seeing in the new section, if you're in the control group. If they were useful on the right, there's no reason they wouldn't be useful on the left. The second part of your comment complains about the placement being intrusive of reading real answers when this new widget only appears when there are no "real answers" to be read.
    – Kevin B
    Commented Apr 3, 2023 at 15:25
  • @KevinB : Thanks for your corrections. But "... is the same list you're seeing in the new section" is not my experience. I stand by that (-: ... "this new widget only appears" .. yes you're right. .... If my Related Questions are appearing at the bottom does that mean I'm getting the ML generated items? I guess we're not supposed to know. Also, I have been getting Rel Qs on the bottom for at least a week now. .... I like the idea elsewhere to add a flag to the URL. /q/123456/?related_questions=1
    – shellter
    Commented Apr 3, 2023 at 17:55
  • 2
    No, all users get related questions below the question on questions with no answers
    – Kevin B
    Commented Apr 3, 2023 at 18:05
  • @shellter Maybe you're conflating the widget placement and widget algorithm? These were independent features. The widget placement has already been tested+deployed for all users. The recommender algorithm is currently being A/B tested. If you're seeing improvements, you might be in the testing group (but that's unrelated to the widget placement).
    – tdy
    Commented Apr 7, 2023 at 20:24
37

One of the things you should keep in mind when judging question relevance is that quite often "Related questions" show questions for completely different technology stacks that are of zero use for the asked questions.

For instance, for question about finding memory leaks in Delphi, related question that covers C# is completely useless even though its title will make it look like extremely relevant to the asked question. But without seeing the tags that are not visible on the list of related questions, the only way to discover that is by opening the question which will skew your CTR measuring.

The problem is two fold, you would have to include tags in the ML model to judge relevance and to avoid showing irrelevant questions, and also it might be useful that you show at least few tags from the suggested questions, so that people can have better idea about suggested question before they try opening it.

4
  • 12
    Just speaking from personal experience: I know I've sometimes had the same issue on Role-playing Games, where the same or similar question could be asked about different editions of the same system, and potentially have very different answers. So I think it may be worth exploring solutions to this issue in general (e.g. "show at least few tags from the suggested questions") – but I'm guessing that part's probably outside the scope of this particular experiment.
    – V2Blast Staff
    Commented Mar 29, 2023 at 18:43
  • 8
    In this first version of the ML model, we are incorporating title and tags to better help with relevancy. As far as including tags into a future UI change, that is an interesting idea that we will keep in mind moving forward. Thank you : )
    – Bella_Blue StaffMod
    Commented Mar 30, 2023 at 18:53
  • @V2Blast Ultimately I think for 'related questions' lists to be truly useful, it will need to use almost entirely unique parameters/criteria for each site.
    – TylerH
    Commented Mar 30, 2023 at 21:14
  • @Bella_Blue see also: Include the primary tag in the title of "Related" questions
    – starball
    Commented Mar 31, 2023 at 23:13
19

I don't know if I'm in the ML group, but for me the RQs are still missing the key topic of a question.

Example: Sum of numbers "around" a number with NumPy

Putting aside the convolution answers (which wouldn't be obvious only from the title/tags), the question is at least about summing, but somehow none of the RQs are even remotely about summing:

  • How do I change the size of figures drawn with Matplotlib?
  • How do I check if a string represents a number (float or int)?
  • Convert a number range to another range, maintaining ratio
  • How can the Euclidean distance be calculated with NumPy?
  • How do I get the number of elements in a list (length of a list) in Python?
  • How do I print the full NumPy array, without truncation?
  • How to upgrade all Python packages with pip
  • Rank items in an array using Python/NumPy, without sorting array twice

Instead it's just a hodgepodge of totally unrelated topics: plotting, strings, lists, printing, upgrading, sorting. The only "related" aspect is Python (in fact 5/8 are not even NumPy).

17

One concern I have with this - people don't always know what they're looking for.

Take this question for example. Absent any context, you'd assume that this person is looking at a way to encrypt a string somehow.

What the person is really asking is how they can keep track of their SMTP password, even if they commit their code to GitHub. Spring offers a lot of ways to do this, but the OP wouldn't know that from jump.

Machine learning can adapt to this sort of thing over time, but it'd take a very long time to do so. My concern is that if it's not immediately apparent to another human what's being asked, a machine learning algorithm isn't going to fare much better (for further context, XY Problem describes exactly this).

The measure of success - clickthrough rates - only indicates that someone looked at a similar question. It doesn't indicate:

  • if the question has answers
  • if the answers were useful
  • if the OP understood that the question was similar (a-la dupe closure angst)

...so seeing lots of clickthroughs but no noticeable satisfaction from the OP that their question was answered seems to be missing the point for me.

3
  • 10
    Moreover almost all posters are terrible at phrasing their questions & their titles when they do "know what they are looking for"--whether they have only an inkling or have a clear idea. They will not compose a clear concise generic phrasing by which they could title, ask or even have searched. Or by which they could debug or reason.
    – philipxy
    Commented Mar 30, 2023 at 3:13
  • 1
    One very common problem which revolves around "people don't always know what they're looking for." is that users ask questions which should have been de-composed more. "Why does X not work in ABC" where A, B, and C are three circumstances. But A and C probably have no relevance. This might be something like "summing digits inside an if" where the if is a red herring. Then from that lack of decomposition, some directly ask about X in Y where the Y is irrelevant. Example being "summing digits inside React" where the problem isn't with React but with the summing.
    – VLAZ
    Commented Mar 30, 2023 at 7:49
  • 3
    You are correct, CTR won’t be enough. But it is only the first data point we are tracking in these experiments. In future iterations, we plan to also analyze the activity and user behaviors once users land on the related question pages. Things we will be tracking are subsequent actions taken on the page; including voting and commenting (attempts and successes), and time spent on page before closing. We believe building out from CTR and including these metrics will give us better signals as to whether something might be relevant or not.
    – Bella_Blue StaffMod
    Commented Mar 30, 2023 at 18:53
17

Here are a few examples of the related question list not working.

First we have why is my comparator function not working? and the top three suggestions are

None of which are at all related to the OP's problem.

Another example: C++ polymorphism and cast has

which are all unrelated. The 5th item in the list of When should static_cast, dynamic_cast, const_cast, and reinterpret_cast be used? is close, and it has good advice on when to use the casts, but it still won’t answer the OP's question.

I can get more posts if you need them, but even the top C++ questions have pretty dubious related posts. For instance, in Why is "using namespace std;" considered bad practice? which is 100% a C++-only question we get as a related question What does if __name__ == "__main__": do? which is a 100% Python question.

1
  • 7
    Knowing the C++ tag followers are fanatic taggers, showing them related questions with a Python tag must be an April Fools joke. What a coincidence ....
    – rene
    Commented Mar 31, 2023 at 8:00
16

From a comment of Abdul Aziz Barkat

The stated objective is to "improve the relevance of related questions" why then is the metric measured click through rate? How is the CTR related to the objective?

I agree. Click through rates only measure how interesting the title is, not how relevant the content is. Consider asking about the relevance (Was this helpful?) on the target page or manually (by humans) estimate the relevance of linked questions.

This way you may end up optimizing for click-bait titles only.

7
  • 2
    note: I also make this suggestion in my answer post.
    – starball
    Commented Mar 31, 2023 at 7:54
  • 8
    optimizing for click-bait titles ... but ... that is the business model, right?
    – rene
    Commented Mar 31, 2023 at 8:02
  • 4
    Shouldn't the title be improved if it is a click-bait title? So if you click on a related question because of a click bait and you see "Oh the title doesn't fit" -> edit that question title? Others than that: The was it helpful? is a good idea. That button could also be added to the list of related questions. Commented Mar 31, 2023 at 10:30
  • @OcasoProtal You suggest that click baity titles be edited, but that would reduce the click through rate. The company won't like that. That's how it is measuring its success. What you maybe want to do is making titles of questions even more click baity, so the question is seen more often. Commented Mar 31, 2023 at 11:17
  • 3
    @Trilarion "3 errors when using Foo library. You WON'T BELIEVE number 2!" or "Take a quiz to find which Bar library exception are you!" or "Fool even Jon Skeet with THIS SIMPLE LINE OF CODE!"
    – VLAZ
    Commented Mar 31, 2023 at 19:53
  • 3
    @VLAZ "Nullpointerexception - here is how our experts tackled it", "Four secret programming tricks nobody wants you to know." Commented Mar 31, 2023 at 21:33
  • 1
    Great suggestion! We are in the works to incorporate a feedback mechanism to solicit qualitative responses to be used in conjunction with quantitative metrics. This could be used as a way to incorporate human-in-the-loop to give direct feedback back to the model.
    – Bella_Blue StaffMod
    Commented Apr 12, 2023 at 16:58
13

I have noticed a new list "related questions" below unanswered questions. I had no idea there was an experiment going on, but as it happens I've been on the verge to bug reporting this one for a while.

The posts listed are very irrelevant and almost always completely unrelated to the question. To the point where the site would be much better if this list was removed entirely.

It doesn't take a lot of domain/programming knowledge to tell that the feature is broken, so how this experiment made it as far as live testing on the main site, I have no idea...


To make an example, I just now grab a random, unanswered question below the tag. How can I order string linked list in c? This question is about sorting a linked list containing strings. Relevant, related posts thus have to be about linked lists and sorting, or perhaps more far-fetched about string handling or maybe pointers.

This is the result we get:

enter image description here

  • How to determine the size of an array in C is completely unrelated, useless information.
  • How to set/clear/toggle a bit is completely unrelated, useless information.
  • How to get a list of files in a directory using C or C++ is completely unrelated, useless information.
  • If I expand the 7 other so-called "related", it is more such nonsense.

The only thing the algorithm managed to get right was the programming language. It noticed that the question was tagged and then picked 10 random, high traffic questions about C programming and formed a list out of them.

1
  • 1
    Let's just hope that the "How can I" title resulting in a list with the most popular questions with the same tag starting with (more or less) the same words is the old recommendation algorithm and that we're in the control group (see also my answer).
    – CodeCaster
    Commented Apr 5, 2023 at 9:22
11

All I see now are high traffic questions with little to no relevance beyond perhaps a common tag. The number of times I've seen "Is Java "pass-by-reference" or "pass-by-value"?" in the related section makes me believe the feature is just entirely broken right now.

If it is the case that a question is genuinely too niche to have relevant related questions, then just tell me. The current method seems to prioritize filling some "result quota" and less concerned about the relevance of those results. Today's related section is 99% distracting, misleading noise that reads like a pool of high-level interview questions. Not only is it unhelpful, it's become downright antagonistic.


To throw another data point in here, this is essentially what the related questions look like for any Java or Spring question now. For example, here's what I get for this question: Spring Configuration phase for Custom Condition
Very general and high-scoring questions that are mostly all unrelated.
All but one of these is extremely unrelated to the question at hand - most barely have a keyword in common. Note how all the (terrible) suggestions have exceptionally high scores, while the one that actually is sort of relevant has 1 point and is buried at the bottom.

2
  • 1
    How do these compare? i.sstatic.net/D4C5x.png they do look at least more "relevant", to my "i know nothing about this language" point of view, than the list you received
    – Kevin B
    Commented Apr 5, 2023 at 19:06
  • 3
    @Kevin B At a glance those seem like pretty good suggestions. While they may or may not be perfectly related (hard to say without reading), they at least seem to have identified a "core theme" of the question and established relationships between others that touch on that theme. I'd definitely click some of those if that's what I saw, but not so much the ones in my screenshot above.
    – ParkerM
    Commented Apr 5, 2023 at 19:17
9

It looks like a question title starting with "How to" or "Is there any way to" or the likes just triggers a list of popular "How to" questions.

Or maybe I'm just in the control group. But the related questions being all in your face and that, this is what stand out to me (on mobile, where I don't know how to hide the list from the center of the screen).

8

From this Comment by @Bella_Blue:

[...] the number of users who come across this Meta post and are aware of it being an experiment is significantly small compared to the 30% of users who are in the experiment.

I know for sure I don't belong to the "select" group of those 30% lucky/poor(?) ML-Testers.
Is there a way I can force myself into/out of that group or to toggle looking at a question with the "classic" or "ML" 'Related Questions' results at will?
(That would be handy to compare the results.)

For example by manually adding a Param in the URL, like /q/123456/?related_questions=1 for "classic", and /q/123456/?related_questions=2 for "ML"?

5
  • 2
    The participants are chosen at random and we do not have control over who gets the test or who doesn’t at this time. You could try a different browser but we cannot guarantee that you'll get bucketed.
    – tanj92 Staff
    Commented Mar 30, 2023 at 17:32
  • 6
    @tanj92 so when people give feedback here- not knowing whether they were chosen, will you (and other staff) know whether they were chosen or not?
    – starball
    Commented Mar 30, 2023 at 22:39
  • @tanj92, yep a bit like #user, then I don't really see the use of the Exp if everything is random and you don't have any control... And nope, I'm not going "to try" with (an)other Browser(s), I already only use "this Browser" (FF111) because all my other (Prod) Browsers don't work (anymore) with 'SO'... (And I consider 'Chrome' and 'Brave' as Malware, so I refuse to use those... (And the Versions I have for those 2 probably only work half with 'SO'...)) // But Users asking for Transparency and Control are probably the ones the most "motivated" to give Feedback over the Exp, I would think...
    – chivracq
    Commented Mar 31, 2023 at 0:25
  • 1
    @chivracq The reason user's are not given an option to select what group they are in is to probably prevent bias being introduced in the captured results. Commented Mar 31, 2023 at 8:30
  • 2
    @Abdul, yeah "biased" is not the Problem, I would think... From 20M Users, 10 Answers on this Thread, 4 give "real" Feedback with concrete Examples, and nobody knows if the Feedback is really about ML or "classic"... => SE will be Happy they still have CTR to measure the "Success" of the Experiment, ah-ah...!
    – chivracq
    Commented Apr 2, 2023 at 6:36
7

measuring clickthrough rates (CTR) of questions from the control group (powered by Elasticsearch) and the experiment group (powered by ML)

As others have mentioned, this is a terrible target variable, imo. When I have a question, I usually click questions with similar titles. The fact that I clicked doesn't mean that question was relevant, it indicates that I wanted to check if it was relevant. To determine relevancy, I would have to spend time on the post, read the question and perhaps read a few answers. So perhaps, relevancy should be determined by how long a user spends on a page they clicked from Related Questions? Or maybe track if a user positively votes or copies code from a question they clicked?

1
  • Thanks for the feedback. We will definitely be monitoring additional signals from here on out, regardless of whether or not we’re conducting experiments. We’ve been working at collecting more data for how users are interacting with related questions. To name a few: 1.) Time spent on related question page 2.) Engagement measured through votes and comments. Users who do not have enough reputation or anonymous users who cannot vote will be captured as “attempts” given the majority of users who visit are anonymous. 3.) Whether something (i.e. code snippet) was copied
    – Bella_Blue StaffMod
    Commented Apr 12, 2023 at 17:00
6

EDIT

As explained in

We are graduating the "Related questions using Machine Learning" experiment

This example is not relevant.

Users had reported that they had not seen an increase in relevancy, but rather a worsening of the results.

However, we were able to check the data and determined that those who had reported seeing less relevancy in the suggested related questions were not part of the experiment group, meaning they were not seeing related questions generated from the ML model.

That seems to be my case, considering the following

To know whether you’re seeing related questions from the machine learning model or Elasticsearch, you can inspect the related question’s URL.

  • URLs containing rq=1 or rq=2 indicate the related question is being recommended by Elasticsearch.
  • URLs containing rq=3 or rq=4 indicate the related question is being recommended by the machine learning model.

ORIGINAL

Sorry, no, I haven't noticed any increased relevancy.

Consider this C++ tagged question:

enter image description here

Those are the "related" question that were shown to me:

enter image description here

None of these have any relevance other than maybe the same language tag and the keyword "class".

In this first version of the ML model, we are incorporating title and tags to better help with relevancy.

It makes me wonder if the actual bodies of the questions are even planned to be considered in the training data. Unfortunately, the titles (and the tags) are often misleading or just not informative.

As others suggested, please let the users vote the relevance of the suggestions.

5

Why not add vote buttons to related questions? It could also serve as training...

It works perfectly for answers. Why not for related questions? Maybe too much many-to-many relationship?

1
  • 1
    see also my answer post's "Why not give users the ability to give explicit, inline feedback?" section, where one of my suggestions is "Ex. giving them vote buttons (separate from actual votes on the post)"
    – starball
    Commented Apr 7, 2023 at 6:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .