Skip to main content
Add #6 from Ian Campbell in a comment: https://meta.stackoverflow.com/questions/423143/a-b-testing-related-questions-within-the-answers-list-experiment-has-graduated/423760#comment951167_423743
Source Link
Makyen Mod
  • 33.2k
  • 14
  • 114
  • 119

and #5 provided by Donald Duck in a comment:

#6 provided by Ian Campbell in a comment:

  1. Percentage of users who land on a related question page and then copy a portion of that page (higher better, because they copied that code to use it). I think you keep up with copies, right?

and #5 provided by Donald Duck in a comment:

#5 provided by Donald Duck in a comment:

#6 provided by Ian Campbell in a comment:

  1. Percentage of users who land on a related question page and then copy a portion of that page (higher better, because they copied that code to use it). I think you keep up with copies, right?
Add #5, link to userscript & update to say I'm using the code.
Source Link
Makyen Mod
  • 33.2k
  • 14
  • 114
  • 119

Gah!!!Gah!!!

Metric suggestions:

  1. % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question).
  2. % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help).
  3. % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
  4. A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.

and #5 provided by Donald Duck in a comment:

  1. % of users who, after landing on a page with a related questions module and clicking on one of the related questions, upvoted the related question or one of its answers (good: higher, showing that more users found the related question helpful).

Note: I don't actually like the new position for this section, so will, eventually, have a userscript changewhich changes it back, but without improving the selection of "related questions"thanks to work by Samuel Liew and released under a MIT license (code in commit 1 and 2), having this be "in your face"which is bad for all your users andincluded in his ReduceClutter userscript (see his answer below for Stack Overflowmore details on that userscript).

Gah!!!

Metric suggestions:

  1. % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question).
  2. % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help).
  3. % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
  4. A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.

Note: I don't actually like the new position for this section, so will, eventually, have a userscript change it back, but without improving the selection of "related questions", having this be "in your face" is bad for all your users and for Stack Overflow.

Gah!!!

Metric suggestions:

  1. % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question).
  2. % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help).
  3. % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
  4. A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.

and #5 provided by Donald Duck in a comment:

  1. % of users who, after landing on a page with a related questions module and clicking on one of the related questions, upvoted the related question or one of its answers (good: higher, showing that more users found the related question helpful).

Note: I don't like the new position for this section, so have a userscript which changes it back, thanks to work by Samuel Liew and released under a MIT license (code in commit 1 and 2), which is included in his ReduceClutter userscript (see his answer below for more details on that userscript).

Reorganize, to an extent. Add metric suggestions provided by Ryan M in comments.
Source Link
Makyen Mod
  • 33.2k
  • 14
  • 114
  • 119

Gah!!!

Gah!!!

Before making the "related posts" section be in your face, the selection of "related" questions needs to be substantially improved to at least take the question's tags into account. There are multiple places where the entire site experience would be far better if the list of automatically shown "related" questions took the current question's tags into account. Fixing that overall is what should have time allocated to it, not making very poor selections of "related" questions more "in your face".

Please improve the selection of "related questions" before making this change.

Note: I don't actually like the new position for this section, so will, eventually, have a userscript change it back, but without improving the selection of "related questions", having this be "in your face" is bad for all your users and for Stack Overflow.

It appears you graduated this "feature" primarily because of the dramatic increase in click-through rates. That the click-through rates are dramatically increased is whollyabsolutely expected. If you make something more obvious, then people are going to click on it more. The real question is if those clicks actually helped your users. Given the near-universally poor relevance of the links that are shown, due to poor "related question" selection, I'd argue that a large percentage of the people that did click through ended up wasting their time looking at questions which were not, in fact, related and weren't relevant to solving their issue.

The problem with evaluating it on that basis is that I'm unsure howyou'd have to measure itsome secondary factors as surrogates for the links being helpful, given that a notable number of people will just give up looking on SO, or at least with SO search/"related" links, once they realize that what they are being presented with has a low probability of being useful.

Ryan M has some good suggestions for metrics in comments (1, 2, with some reformatting):

Metric suggestions:

  1. % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question).
  2. % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help).
  3. % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
  4. A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.

Please improve the selection of "related questions" before making this change.


Note: I don't actually like the new position for this section, so will, eventually, have a userscript change it back, but without improving the selection of "related questions", having this be "in your face" is bad for all your users and for Stack Overflow.

Gah!!!

Before making the "related posts" section be in your face, the selection of "related" questions needs to be substantially improved to at least take the question's tags into account. There are multiple places where the entire site experience would be far better if the list of automatically shown "related" questions took the current question's tags into account. Fixing that overall is what should have time allocated to it, not making very poor selections of "related" questions more "in your face".

Please improve the selection of "related questions" before making this change.

Note: I don't actually like the new position for this section, so will, eventually, have a userscript change it back, but without improving the selection of "related questions", having this be "in your face" is bad for all your users and for Stack Overflow.

It appears you graduated this "feature" primarily because of the dramatic increase in click-through rates. That the click-through rates are dramatically increased is wholly expected. If you make something more obvious, then people are going to click on it more. The real question is if those clicks actually helped your users. Given the near-universally poor relevance of the links that are shown, due to poor "related question" selection, I'd argue that a large percentage of the people that did click through ended up wasting their time looking at questions which were not, in fact, related and weren't relevant to solving their issue.

The problem with evaluating it on that basis is that I'm unsure how to measure it, given that a notable number of people will just give up looking on SO, or at least with SO search/"related" links, once they realize that what they are being presented with has a low probability of being useful.

Gah!!!

Before making the "related posts" section be in your face, the selection of "related" questions needs to be substantially improved to at least take the question's tags into account. There are multiple places where the entire site experience would be far better if the list of automatically shown "related" questions took the current question's tags into account. Fixing that overall is what should have time allocated to it, not making very poor selections of "related" questions more "in your face".

It appears you graduated this "feature" primarily because of the dramatic increase in click-through rates. That the click-through rates are dramatically increased is absolutely expected. If you make something more obvious, then people are going to click on it more. The real question is if those clicks actually helped your users. Given the near-universally poor relevance of the links that are shown, due to poor "related question" selection, I'd argue that a large percentage of the people that did click through ended up wasting their time looking at questions which were not, in fact, related and weren't relevant to solving their issue.

The problem with evaluating it on that basis is that you'd have to measure some secondary factors as surrogates for the links being helpful, given that a notable number of people will just give up looking on SO, or at least with SO search/"related" links, once they realize that what they are being presented with has a low probability of being useful.

Ryan M has some good suggestions for metrics in comments (1, 2, with some reformatting):

Metric suggestions:

  1. % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question).
  2. % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help).
  3. % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources).
  4. A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed.

Please improve the selection of "related questions" before making this change.


Note: I don't actually like the new position for this section, so will, eventually, have a userscript change it back, but without improving the selection of "related questions", having this be "in your face" is bad for all your users and for Stack Overflow.

talk about 900% increase in click-through
Source Link
Makyen Mod
  • 33.2k
  • 14
  • 114
  • 119
Loading
more on "this is bad"
Source Link
Makyen Mod
  • 33.2k
  • 14
  • 114
  • 119
Loading
Source Link
Makyen Mod
  • 33.2k
  • 14
  • 114
  • 119
Loading