**Gah!!!** ## This change is counterproductive, unless the selection of "related" posts is *substantially* improved. **Before** making the "related posts" section be *in your face*, the selection of "related" questions needs to be *substantially* improved to *at least* take the question's tags into account. There are multiple places where the *entire site experience* would be far better if the list of automatically shown "related" questions took the current question's tags into account. Fixing *that* overall is what should have time allocated to it, not making *very poor* selections of "related" questions more "in your face". Changing the location and visibility of these is *counterproductive* when the selection of "related posts" is *often* completely useless. The reason people, at least experienced users, don't click on them when they are in the sidebar is because we usually just ignore that section, given that the "related posts" are commonly *not related at all* to the current question. Once the selection of "related" questions is fixed, *then* it would be reasonable to look at changing where they are displayed. Changing these to being more "in your face" will increase users' frustration in a lot of circumstances (e.g. "the site can't even show me questions that are actually related"). Not having good selections will also result in user's just learning to ignore the new in-your-face "related questions" section, which is *counterproductive* to, eventually, having an actually good selection of "related questions". If the selection isn't improved **first**, then when/if they are, you have to work against all the training you've done for your users to ignore that section. ### You had a 900%+ increase in click-through rates, but is that relevant? It appears you graduated this "feature" primarily because of the dramatic increase in click-through rates. That the click-through rates are dramatically increased is absolutely expected. If you make something more obvious, then people are going to click on it more. The real question is if those clicks *actually helped* your users. Given the near-universally poor relevance of the links that are shown, due to poor "related question" selection, I'd argue that a large percentage of the people that did click through ended up wasting their time looking at questions which were not, in fact, related and weren't relevant to solving their issue. #### The primary factor which this should be evaluated is: Did the users' experience improve (i.e. did they find a solution to the problem they are having)? The problem with evaluating it on that basis is that you'd have to measure some secondary factors as surrogates for the links being helpful, given that a notable number of people will just give up looking on SO, or at least with SO search/"related" links, once they realize that what they are being presented with has a low probability of being useful. [Ryan M](https://meta.stackoverflow.com/users/208273/ryan-m) has some good suggestions for metrics in comments ([1](https://meta.stackoverflow.com/questions/423143/a-b-testing-related-questions-within-the-answers-list-experiment-has-graduated/423743?noredirect=1#comment950973_423743), [2](https://meta.stackoverflow.com/questions/423143/a-b-testing-related-questions-within-the-answers-list-experiment-has-graduated/423743?noredirect=1#comment950974_423743), with some reformatting): > Metric suggestions: > 1) % of users who, after landing on a page with a related questions module, went on to ask a question (good: lower, showing that more users found the solution they were seeking without needing to ask a question). > 2) % of users who, after being shown the related questions module on their own question, deleted the question (good: higher, showing that more people found a solution without needing another person's help). > 3) % of questions where the module is displayed that are closed as duplicates (good: higher, showing that users are being pointed to canonical resources). > 4) A fourth interesting metric would be the % of questions with the module shown that receive answers. An increase in this metric would probably be bad (suggesting that people are using the related questions as a source to copy answers from), but that would require investigation to confirm. Similarly, a decrease would probably be good, but only if it's resulting from users being pointed to canonical resources instead of getting answers on their own question (which are likely to be lower-quality than those on the canonical question). Again, investigation into the root cause would be needed. #5 provided by [Donald Duck](https://meta.stackoverflow.com/users/4284627/donald-duck) in [a comment](https://meta.stackoverflow.com/questions/423143/a-b-testing-related-questions-within-the-answers-list-experiment-has-graduated#comment951096_423743): > 5. % of users who, after landing on a page with a related questions module and clicking on one of the related questions, upvoted the related question or one of its answers (good: higher, showing that more users found the related question helpful). #6 provided by [Ian Campbell](https://meta.stackoverflow.com/users/13095326/ian-campbell) in [a comment](https://meta.stackoverflow.com/questions/423143/a-b-testing-related-questions-within-the-answers-list-experiment-has-graduated/423760#comment951167_423743): > 6. Percentage of users who land on a related question page and then copy a portion of that page (higher better, because they copied that code to use it). I think you keep up with copies, right? <!-- #6 provided by []() in [a comment](): --> #### *Please* improve the selection of "related questions" *before* making this change. --- Note: I don't like the new position for this section, so have a userscript which changes it back, thanks to work by [Samuel Liew](https://meta.stackoverflow.com/users/584192/samuel-liew) and released under a MIT license (code in commit [1](https://github.com/samliew/SO-mod-userscripts/commit/04a3bda8577ca122cc343714f11932cb2644f783) and [2](https://github.com/samliew/SO-mod-userscripts/commit/9d24e1e5a09da29df6df7cc7ceebb32eccb5da6c)), which is included in his [ReduceClutter](https://github.com/samliew/SO-mod-userscripts/blob/master/ReduceClutter.user.js) userscript (see his [answer below](https://meta.stackoverflow.com/a/423760/3773011) for more details on that userscript).