Update — January 19, 2024
We are concluding this experiment effective January 19 and the module will no longer appear for users.
The objective was to increase traffic to the Collectives-specific content types, and the module did not drive any significant increase.
Additionally, we saw that traffic from Stack Overflow to Meta Stack Overflow and the Overflow Blog did not have any significant change in either direction.
We have no immediate plans to revisit this specific experiment. The idea of highlighting content from Collectives and/or Discussions in a right sidebar module may be something we return to in the future.
Update — November 15, 2023
This temporary (90-day) experiment has been launched. A notable change from the initially stated matching logic is that we’re no longer matching on title, because it negatively impacted performance on the site globally.
As previously stated, the aim is to increase awareness of and engagement with the collective-specific content types.
Based on early testing, we expect:
- 15-20% of all questions in a collective may have the module displayed
- Around 5% of all questions on Stack Overflow may have the module displayed
If you would like to provide feedback about the module’s content on Question pages that you visit, please do so on this post.
We’re working on better discovery and awareness for Collectives™ on Stack Overflow and will be experimenting with a change to question pages that have a tag associated with a collective.
Background
Research has shown that even when users find and join a collective, they are not always aware of the new features or content coming from that space. We’ve seen interest in the new content types (collections, articles, and now discussions) but there is a clear lack of awareness that these features even exist. We believe users should be delivered content relevant to their needs and this includes Collectives. We won’t truly know how valuable these new features are unless we provide improved pathways and discovery for those features.
The current discovery paths focus on the overall awareness of the subcommunity only. One of the higher trafficked pathways is through the collective badge (currently found at the top of the right sidebar on questions that have a collective tag).
However, we’ve also heard feedback that some users gloss over this badge for varying reasons; the right sidebar is ignored altogether or they assume it’s an ad. Additionally, we’ve seen there’s a preference and increased traffic for the Related questions section.
Hypothesis
We believe by boosting discovery and awareness of Collectives, we will see more Members join and participate, which in turn will help these subcommunities continue to contribute high quality content.
A community with increasing membership and activity is an indication of healthy growth. This growing population of contributors is critical to ensuring the body of knowledge evolves & stays relevant. Stack Overflow has a responsibility to raise awareness about and highlight the products we're building.
We know that most users come to our site through Google search results and try to find their answer by sifting through individual Q&A pages. Our best chance at raising that awareness is to give these users direct access to key features that are normally hidden (i.e. articles, bulletins, collections) from Q&A pages. Through better discovery of these features, we’ll be able to evaluate and iterate through real user experiences.
Experiment
So what’s actually changing for this experiment? We’re going to be replacing the community bulletin block within the right sidebar (yellow box) with related collective content. This related collective widget will be shown directly below the current collective badge and will only appear on question pages with an associated collective tag. All other question pages will remain unchanged and still continue to display the community bulletin widget. This will impact approximately 0.44% of traffic coming from the community bulletin block on a question page belonging to a collective. We’ll review the findings of the experiment after 90 days to assess the impact for collectives and evaluate the degree to which traffic to The Overflow and Meta are impacted, if at all.
Risks and constraints
Removing the current community bulletin content from that space means a potential reduction in awareness for the Overflow Blog and Meta. We are aware this is a significant change. However, we believe the upside to Collectives Discovery is worth testing and will give us additional insight into The Overflow and Meta traffic impacts. Here are the reasons for moving forward with this decision:
- For this experiment, we needed to keep the position of the ads that currently appear in the right sidebar relative to its placement on the page (approximately within reason).
- Based on the data, we know that the top position of the right sidebar is seen by the most users. The more a user has to scroll, the less likely it will be seen on that question page.
- The goal of this experiment is to test discovery for collective content only and in order to test that as accurately as possible, we want to avoid making too many changes to the right sidebar (like re-ordering everything in that space) so that we can isolate any differences as much as possible.
If we see a negative impact on The Overflow blog and Meta traffic and no significant increase in collectives traffic, we would not graduate these changes and revert back to showing the community bulletin on all pages, and explore other ways to drive discovery for Collectives.
On the other hand, if we see both an increase in collectives traffic but also a significant negative impact to the blog & meta, we will experiment with further refinements (e.g. more complex rules that govern the content in the space so it can be better distributed across blog, meta, and collectives), or offsetting the impact through other discovery channels for meta and the blog (e.g. via the site navigation).
Current question page
New question page
Our goal for this content is to prioritize relevancy over other criteria.
- Only up to 4 posts will be shown in this widget. These could be a combination of bulletins, articles, collections and/or discussions — tag relevancy will determine which has priority. Question posts will not appear in this widget because there are already several other areas within the page that show relevant questions.
- We’ll start by using title and tag matching to determine how relevant a post may be to a question. For example, we would prioritize showing a post that matches 4 out of 4 tags over another that shows only 3 out of 4.
- A minimum amount of matched tags will be required to show any related post. If there isn’t a strong enough match (0 related collective posts), the community bulletin will remain visible.
- If a question page has tags from more than one collective, we would still prioritize relevancy to the question — there could be posts from both collectives shown in the widget.
Design thinking
As part of this initiative, we’re making minor visual changes to the badge and chose a slightly different layout for these content types compared to the other question links within the right sidebar.
Rationale
- We chose not to display the post score for these content types and use icons instead (similar to Hot Network Questions). Our assumption is that the title will be the primary indicator for determining whether a piece of content is of interest to a user. We also wanted to keep all post link titles left aligned down the entire sidebar to help with legibility and scannability of the page.
- For the collective content, we opted to use icons to help with visually identifying the different types of posts that could appear here. We chose not to use a collective logo (like on Hot Network Questions) since this would be the same logo appearing up to 4 times in most cases. Even in the rarer case of a multi collective question page, knowing which collective the post was from didn’t seem as valuable.
- The different content types are new and less known. In order to help build recognition for the icons and meet better accessibility standards, we decided to add the content type within the metadata line as well. The other piece of metadata shown will be different for each content type — based on what is most important per type.
A designer’s easter egg
A response from starball in an answer to visual changes to the current community bulletin’s yellow box really resonated with us.
It always felt like yellow notepad paper to me in my mind
We decided to lean into this idea as a source of inspiration and look at visual representations of notebooks, post-it and the original legal yellow pad of paper. We wanted to give a visual nod to the original design. This type of thing is less common for the site but it would be fun to have a few more of these moments in the future.
- The top left corner of the badge is not rounded on purpose to represent the typically harder header piece of a legal pad where individual pages can be ripped from.
- This sharper corner also points towards the title of the question creating a stronger relationship between the widget and the question.
- There’s extra left padding within the list so that the items feel like they are ‘under’ and part of the collective badge. This is similar to how you would write on a legal pad of yellow paper, not directly on the left edge but with some spacing.
- The informational sentence at the bottom uses a divider line instead of its current boxed border to mimic the vertical lines typically found on a yellow notepad.
What does success look like?
We’ll be looking measuring the effectiveness of this widget by tracking the following metrics and comparing them to our current sidebar badge data:
Primary
- Increase in the clickthrough rate for the right sidebar widget (total of all content links)
Secondary
- Increase in new membership join rate (incoming from right sidebar widget)
- Increase in monthly Collectives pageviews
- Increase in monthly unique visitors to collectives
- Clicks on specific content types (articles, discussions, bulletins, collections)
Our ask
We may improve the logic for which posts are shown in this widget and are willing to add more complexity once we see the initial results. Our goal is to show content that is relevant and potentially valuable to the individual user viewing a question page. Please feel free to share any feedback by September 22nd on how we can improve the relevancy (e.g. targeting title matching or user based criteria based on rep levels or privileges) or information shown with each post (e.g. what single piece of metadata is most important for you for each type).
Additionally, if you have any feedback about the design of the widget (e.g. consistency with other elements and the visual nod to notepads), please don’t hesitate to share your thoughts.
We plan to review the results after 90 days and share an update to this post with the findings shortly thereafter.