8

I feel like a step was skipped before the experiment that is ongoing at the time of this writing got launched (Do you observe increased relevance of Related Questions with our Machine Learning experiment?). Actually- make that two steps (Step 1: see For the purposes of the "Related" section of the Q&A UI, what definition of "related" aligns closely to what users of Stack Overflow find value in?). Step 2: We never got asked for input on how the experiment could be conducted, or how the machine learning (ML) model could be trained. I know it's "too late", since the experiment has already started, but I'm feeling idealistic. Let's talk about it anyway.

What would you like (/have liked) to see SE Inc. try with respect to:

  1. The training of the machine learning model?

    • Inputs (question title? question body? tags? score? answers? manually linked questions? duplicate links? inline code / code-blocks / Stack Snippets? external URLs in the post bodies and/or comments? inbound links from external sites found on the same external page? etc.)
    • How the inputs are used (impact of score? impact of tags? etc.)
    • Anything else you think is relevant
  2. The experiment and its execution?

    • How the degree of success is qualified
    • How feedback on the effectiveness of the ML-recommended links is gathered (discussion on meta? voting/rating mechanism? (positioned where and "when" in the UI?))
    • What data to release from the experiment
    • Anything else you think is relevant

No pressure to give an idea you think is the-perfect-idea™. Let's brainstorm.

3 Answers 3

6

I'd rather for it to have been tested using the duplicate suggestions feature of the ask question wizard and/or the duplicate question dialog rather than related questions. Areas of the site that actually need improvement... rather than "related questions" which has always just been throwaway content similar to the "recent badges" list.

1
  • OTOH, if it's producing even-more-useless lists of related questions, an unimportant piece of the site is maybe a better place for initial testing. Some answers on the feedback request report lists of related questions that are worse than ever. (Of course, maybe they were in the control group, and paying some attention to the related list merely revealed how bad it's always been for some questions.) Commented Apr 3, 2023 at 22:53
6

For the training of the ML model

  • I'd like to see what the machine gives when told to not take score into account, except to favour non-negatively scored posts over negatively-scored ones (/ when all positive scores are flattened to be as if they had a score of zero).

    Or perhaps to take the total score of the question plus the score of its positively-scored answers, since sometimes there are downvoted questions with fairly upvoted answers, which may be insightful.

  • I'd like to see tags be used as hard filters for some percentage of the entries, or at least have a high degree of influence. Meta-tags (like and ) should probably have their influence diminished though.

  • Taking URLs in questions, answer, and comments into account could be useful. For example, two posts with answers linking to the same documentation would be "related" in that sense. It might be good to take the hash/fragment, and query parts of the URL into account (weighting matches that also match those parts of the URL higher), since those can actually be very significant for the content of pages or what the link is referring to (Ex. a particular search query, or a particular section of a page of documentation).

  • I'd like to see some part of the training take into account popular manually linked questions. Hopefully we could get the machine to suggest those where appropriate. I mean questions like What is a NullPointerException, and how do I fix it?, What is an undefined reference/unresolved external symbol error and how do I fix it?, What is a debugger and how can it help me diagnose problems?, etc.

  • Taking into account inbound links from external sites found on the same external page sounds interesting, but also sounds a lot more complicated to implement, and I'm not sure whether it'd actually end up being useful. Perhaps for a separate experiment.

For the experiment

How the degree of success is qualified

Some form of manual feedback, supplemented with click-through rate. We've given our feedback on the weaknesses of using CTR alone already (cannot distinguish false-positives, etc.).

How feedback on the effectiveness of the ML-recommended links is gathered

I'd suggest a banner-like section in a Q&A page that appears when clicking to that Q&A from the "Related" section. The banner would ask something like "Was this possibly related question valuable to you?". The response options would be:

  • "yes. It contained information pertinent to answers for the question I came here from."
  • "yes. I have had this question before around a time when I had the question I came here from."
  • "yes. I think I might have had this question naturally soon later considering the question I came here from."
  • "yes. It's not a question I've wondered naturally, but I found this Q&A fun and interesting to read."
  • "no. It was not valuable to me in any way."
  • "null. I abstain from giving this feedback" (default selection, with option to "not ask me again")

The positioning of the banner could either be at the top of the page for better visibility, or under the question post for more logical procedural flow.

The user can respond multiple times, but only the latest response will be retained (not sure if this is possible to implement for logged out users. If it's not, I'd still suggest taking that feedback, and just flipping a bit in the DB row that says the user wasn't a logged-in one).

(also, yeah. those response descriptions are a mouthful. I'm not sure how to shorten them. Suggestions welcome.)

What data to release from the experiment

I'd like to get a full dump of anonymized responses to the above-described feedback prompt (what the source question was, what the question the machine said was "related" that the user click on, and what response option the user submitted). It could be potentially useful have associated info about whether the user who gave each feedback entry was a logged in user, and if so, roughly how much rep they have (order of magnitude?).

Ideally for me, the data would be published on something like SEDE so I could play with it myself. Or the next best thing being something like the regular data dumps we do.

0
2

Thank you so much for the time and attention that you have put into this. We’re doing similar analysis and finding that we largely match your thoughts, which is helpful in guiding our path forward.

For the first experiment, we used the question title and question tags as inputs to the model. The reasoning for considering these as inputs is that both inputs are currently being used in the Elasticsearch index.

There are countless data points we could measure when recommending related questions but happily, we are on the same page, as all of the ones you suggested are ones that we have discussed internally.

Moving forward, we plan on collecting additional data points to reinforce our understanding of whether someone found value with related questions, some of which has been covered in your answer already.

  • Whether someone attempted to engage via voting or commenting
  • The amount of time spent on the page
  • Create a mechanism to collect qualitative feedback in the UI
  • Whether something was copied from the page (i.e. code snippet or any text)
1
  • nice! for "Create a mechanism to collect qualitative feedback in the UI", feel free to take inspiration from my answer post.
    – starball
    Commented Apr 12, 2023 at 17:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .