For the training of the ML model
I'd like to see what the machine gives when told to not take score into account, except to favour non-negatively scored posts over negatively-scored ones (/ when all positive scores are flattened to be as if they had a score of zero).
Or perhaps to take the total score of the question plus the score of its positively-scored answers, since sometimes there are downvoted questions with fairly upvoted answers, which may be insightful.
I'd like to see tags be used as hard filters for some percentage of the entries, or at least have a high degree of influence. Meta-tags (like array and string) should probably have their influence diminished though.
Taking URLs in questions, answer, and comments into account could be useful. For example, two posts with answers linking to the same documentation would be "related" in that sense. It might be good to take the hash/fragment, and query parts of the URL into account (weighting matches that also match those parts of the URL higher), since those can actually be very significant for the content of pages or what the link is referring to (Ex. a particular search query, or a particular section of a page of documentation).
I'd like to see some part of the training take into account popular manually linked questions. Hopefully we could get the machine to suggest those where appropriate. I mean questions like What is a NullPointerException, and how do I fix it?, What is an undefined reference/unresolved external symbol error and how do I fix it?, What is a debugger and how can it help me diagnose problems?, etc.
Taking into account inbound links from external sites found on the same external page sounds interesting, but also sounds a lot more complicated to implement, and I'm not sure whether it'd actually end up being useful. Perhaps for a separate experiment.
For the experiment
How the degree of success is qualified
Some form of manual feedback, supplemented with click-through rate. We've given our feedback on the weaknesses of using CTR alone already (cannot distinguish false-positives, etc.).
How feedback on the effectiveness of the ML-recommended links is gathered
I'd suggest a banner-like section in a Q&A page that appears when clicking to that Q&A from the "Related" section. The banner would ask something like "Was this possibly related question valuable to you?". The response options would be:
- "yes. It contained information pertinent to answers for the question I came here from."
- "yes. I have had this question before around a time when I had the question I came here from."
- "yes. I think I might have had this question naturally soon later considering the question I came here from."
- "yes. It's not a question I've wondered naturally, but I found this Q&A fun and interesting to read."
- "no. It was not valuable to me in any way."
- "null. I abstain from giving this feedback" (default selection, with option to "not ask me again")
The positioning of the banner could either be at the top of the page for better visibility, or under the question post for more logical procedural flow.
The user can respond multiple times, but only the latest response will be retained (not sure if this is possible to implement for logged out users. If it's not, I'd still suggest taking that feedback, and just flipping a bit in the DB row that says the user wasn't a logged-in one).
(also, yeah. those response descriptions are a mouthful. I'm not sure how to shorten them. Suggestions welcome.)
What data to release from the experiment
I'd like to get a full dump of anonymized responses to the above-described feedback prompt (what the source question was, what the question the machine said was "related" that the user click on, and what response option the user submitted). It could be potentially useful have associated info about whether the user who gave each feedback entry was a logged in user, and if so, roughly how much rep they have (order of magnitude?).
Ideally for me, the data would be published on something like SEDE so I could play with it myself. Or the next best thing being something like the regular data dumps we do.