As you might have noticed, we launched an experiment on Stack Overflow called Discussions (currently only available within the NLP Collective). This post is the origin story behind this feature, diving into all the research, activities, and milestone decisions that led us to launch Discussions. Settle in—this post’s a long one!
TL;DR:
Our core purpose at Stack Overflow is to enable technologists to better and more easily make informed, technical decisions. While Q&A has served this purpose well for decisions that have definitive, objective solutions, Discussions aims to expand upon this by allowing technologists to source diverse sets of perspectives that may be broader and more subjective than what Q&A allows for.
We know from prior research that many users want more diverse ways to engage, learn, and share knowledge on Stack Overflow. In multiple rounds of discovery research and concept testing, Discussions successfully provided users with more opportunities to interact with other knowledgeable individuals and engage in ongoing conversations around specific questions, broader technical strategies and approaches, and general comprehensive knowledge-sharing.
We recognize that not all users seek the increased sense of community we believe Discussions can help build. Launching Discussions within our Collectives subcommunity space is how we can experiment with this while minimizing any unintended impact to the core site experience.
Discussions will not sacrifice the high-quality content that Stack Overflow is known for. Research shows that quality Discussions should be highly focused and technical within the field/area of practice the larger discussion space centers around. This is not an “anything goes” space, and our community guidelines and moderation approaches will work to uphold this.
We want to iterate and build with you. The Discussions feature launched this week is a pared down version of what we believe this feature can ultimately be. Our goal is to see how the larger community uses it and what additional features are needed. We will continue researching user experiences in Discussions as well as gathering feedback from the community.
Background and Context
Prior Research
During the discovery and research phase that led to the next iteration of Collectives, one key issue kept surfacing: the desire for users to connect more with one another, bringing "humans to the forefront” but still remaining highly technical. Over the last two years, this is a sentiment that has surfaced across a lot of research activities in a variety of ways — surveys, co-creation sessions, and traditional interviews. Across them all, we’ve learned that — done right — a supportive community that is focused on knowledge sharing, interactive collaboration, and made up of smart, technically-minded peers would be highly valuable to many of our users.
In some ways, Stack Overflow today has some of these elements, but it’s lacking in others. Internally we’ve come to think of this as the “Big City” vs. “Small Town” dilemma. Across two years of research, we’ve come to understand that the most valuable and sticky online communities convey the feeling of a small town, regardless of the size of their user base. The majority of its members understand its norms, feel that it’s approachable and welcoming, can actively ‘connect’ with others around them (we’ll get into what high-quality connection looks like later — but spoiler, it isn’t social media-esque), and can take some time away but easily jump back in when they’re ready.
To the majority of our users, Stack Overflow today feels like a Big City, which isn’t necessarily a bad thing. (As a company headquartered in NYC for a large part of our history, we love a big city!) But there are very special corners of Stack Overflow that have emanated the feeling of a small town, which its members speak of with great fondness — those certain chat rooms you keep going back to, the tags of more niche technologies where you start to recognize and appreciate other active users, and some of our Stack Exchange sites.
We want to intentionally create the feeling of a small town on Stack Overflow through Discussions. We hypothesize this will help foster engagement and knowledge sharing by a wider base of our users who might be averse or find it challenging to start off in the big city of Q&A. This directly ties back to our core purpose: enabling technologists to make informed decisions. In order to do this we need a curated, maintained repository of knowledge that’s accurate, credible and up to date. Discussions aims to offer new, diverse ways to engage on Stack Overflow while simultaneously providing technologists the opportunity to share technical knowledge that leans a little more subjectively than our main Q&A function.
Emerging Needs within Collectives
Meanwhile, within the Collectives product, we realized we needed a space for Recognized Members to collaborate and organize efforts. Additionally, through research activities but also general participant feedback, the need for a space that allowed expanded conversation about the collective’s content and area of practice began to emerge. Yet these valuable conversations would fall outside of site guidelines, typically for being too broad to be a good question. We summarized these issues into a mission statement to be explored and validated: How might we provide a space for Members of the collective (including Recognized Members) to engage with one another and build subcommunity in ways that are not currently possible on Stack Overflow?
We knew this would be a large paradigm shift for the site overall and didn’t want to introduce something that could impact the current Q&A experience in a negative way. If we changed the core of Q&A, we could risk damaging the knowledge base itself and causing it to become untrustworthy and unreliable. This kicked off an initiative to allow Admins and Recognized Members a space to communicate within the site. We started by creating a more restricted space to allow us to explore having subjective content on Stack Overflow in a focused and controlled environment — small steps.
Additionally, we ran a test of what this could look like on a slightly larger scale by adding a Discord server for the CI/CD and R Collectives. Even though this received very little engagement (largely due to a lack of discovery and driving people off-site, things we knew would be an issue up front), we started to see some use cases and appetite for such a space. We also learned about some potential issues fairly quickly (e.g. somebody posting a link to their Stack Overflow question in the chat in hopes of getting an answer faster) that we would need to mitigate and solve for in a future iteration.
Other Factors at Play
We had additional long overdue issues that were getting resurfaced as some of these conversations were happening.
Large scale research on top user issues on our site found strict rules around objective content have blocked users from being able to get information that would be helpful for decision making around more complex technical problems.
The restrictive nature of Q&A has created boundaries and limitations on the type of knowledge that is preserved on the site.
There’s a significant number of questions that are posted and then closed due to being subjective despite having significant upvotes for both the question and answers provided (examples: #1, #2, #3).
Key Findings from Recent Research
The new Discussions feature was born out of a series of recent research, design efforts, and user feedback - all iteratively informing one another. Our goals were what has been described above: providing our large user base engagement opportunities that can drive a stronger sense of community, while also positively contributing to the knowledge base of Stack Overflow. All of this ultimately supports the sustainability and value of the site.
First, we led multiple cross-functional design sprints. These sprints led to a number of concepts being proposed and then finessed. We then tested two of the most promising concepts in over 8 hours of in-depth user feedback sessions, with representation across a large spectrum of user types. These sessions led to important learnings shared below. Please note that neither of these designs were the exact Discussions feature being launched today, but both included elements of users engaging in online tech-focused dialogue with one another that would likely be closed for being off-topic in Q&A and became the backbone for today’s feature.
Key Finding: Diverse Ways to Engage, Foster Community, and Gain Knowledge
A major finding is, regardless of how users felt about the specific features, our participants all expressed a desire for Stack Overflow to innovate and felt that more diverse engagement options furthered this goal. Discussions - or the ability to be in technically-focused but subjective dialogue with other users - revealed to be very interesting to users participating in our various research activities. Specifically, participants felt Discussions would positively contribute to both their knowledge sharing experience on Stack Overflow and their sense of community on Stack Overflow.
Knowledge Sharing: We found that not only does subjective dialogue like Discussions offer new engagement opportunities to users but it also has the potential to contribute to and increase the rich knowledge maintained on the site. Stack Overflow is a trusted platform that our research participants felt could host high-quality discussions driven by the knowledgeable users already on the site. Done well, Discussions on Stack Overflow could enhance how we deliver on the site’s ultimate value: enabling knowledge-sharing to support technologists in making strong, well-informed decisions. The site is already filled with technical content for problem-solving and learning, thanks to you all. Discussions adds the opportunity for our users to interact with other knowledgeable individuals and engage in on-going conversations around specific questions, broader technical strategies and approaches, and general comprehensive knowledge-sharing.
Sense of Community: We know some of you might be thinking to yourselves, “I come here for information, not for a sense of community.” To that point, yes, this research also highlighted that different users have different expectations regarding the sense of community they want to experience on Stack Overflow. This is precisely why we are currently testing Discussions within the Collectives space. We don’t want to negatively impact the Q&A experience for those users who are currently getting the site experience they desire (although we know improvements can be made to our Q&A experience too!) But it’s important to recognize how large and diverse our Stack Overflow community is. There are many users, all of varying experience levels who do want a stronger sense of community on Stack Overflow, and who could see themselves engaging more and sharing knowledge more within that stronger community space. Our approach is to create a ‘choose your own adventure’ experience, where users are able to plug into the level of engagement and community they individually desire. Launching Discussions within our Collectives subcommunity space is how we can start building towards this without disrupting the core site experience.
Key Finding: An Active and Streamlined Experience
If our Discussions feature is to enhance Stack Overflow’s value as a trustworthy knowledge repository that supports technologists in making informed technical decisions, it needs to be a streamlined experience. Users flagged the importance of being able to find valuable information efficiently. A “feed” of active discussions helped strike the balance between allowing users to explore interesting conversations vs quickly finding what they need based on their use cases.
Unsurprisingly, we also found that users want to stay onsite when in conversation with other Stack Overflow users. This is a learning that was also validated through evaluative research regarding the launch of Topic Collectives, particularly when it came to the Discord experiment. Not many of us prefer to be taken off site when engaging with a site feature!
Finally, we know from many hours of research that to create a valuable experience for our users, this space needs to be active. And that means it has to be discoverable! As mentioned earlier, Discussions is meant to be an extension of Stack Overflow’s knowledge base - meaning the content within it should be organized, relatively easy to find, and easy to jump back into if a user has taken time away. All of this is also why Discussions is more than a simply reworked chat feature, but if you have further questions along those lines, we’ve shared a more detailed explanation in this MSE post.
Key Finding: Quality
Across all these research sessions, a critical caveat was introduced by our users. Discussions would be a valuable addition to Stack Overflow, so long as the content was high-quality. Users continuously emphasized that the space would need a set of standards and guidelines to maintain quality discussions, otherwise spam and irrelevant content would overwhelm the space. While we intended to keep the same site standards around obviously low-quality contributions like spamming (see our community guidelines on how not to be a spammer), we also wanted to investigate what specifically marks a low vs high quality discussion in a tech-focused space.
To hone in on what makes discussions high or low quality, we conducted an unmoderated card sort activity with over 50 users, whose years of professional experience ranged from >1 year all the way to 40+ years.
What we learned is that high quality discussions are highly focused and technical within the field/area of practice the larger discussion space centers around. For example, discussion posts that centered around topics like “advice on best practices and approaches to a technical problem”, “theoretical/conceptual discussions related to the field”, “providing or receiving feedback on a project”, and others in similar veins had the majority of our surveyed users indicating they’d like to both read and actively participate in these online conversations. It’s for this reason that we are firmly emphasizing in our Discussions guidelines and curation/moderation practices that discussion posts be technical and focused within their relevant field/area of practice.
This research also helped us understand what is less interesting and lower value to our community. Discussion posts centering on career advice and course/certification recommendations captured the least amount of engagement interest from surveyed users. These posts are also likely to be less technically focused compared to the topics mentioned above. Findings like this helped directly inform our guidelines for the Discussions space.
But of course, this is just the beginning of our understanding. As with many of our newly launched features, we will continue utilizing quantitative data and alongside rigorous qualitative research to evaluate the Discussions feature and better learn what our community members find valuable in the space and what needs to change. We expect that the guidelines and best practices around discussions to ensure that all the content on our site is higher quality will evolve as we learn more.
Fast Experimentation and a Willingness to Fail
Foundational research is essential, but ultimately we need to test our assumptions by releasing something and observing what happens. This led to the decision to experiment and test this idea quickly by releasing a pared-down version of Discussions in a relatively small, self-selecting community: the NLP Collective. We’ll be transparent about this—we had to cut a lot of great ideas for features that would enrich the experience. The goal here was to get something out quickly so we could see how people use it and which additional features would be most beneficial. We want to involve the community sooner rather than later and gather feedback from people using the feature as we continue to build out the experience. If this experiment goes well and people are finding Discussions valuable, we’ll iterate and develop it into a more complete experience.
Final Thoughts
If you’ve made it this far, thank you for your time and interest in this space. We have a long way to go still and lots more to learn and validate with this feature. We’re extremely thankful for all the time and energy spent by users to share their opinions and thoughts on these ideas. We’re happy to answer any questions you have about the steps and research outlined in this post. We’ll be closely monitoring engagement and usage data, as well as talking to more users directly and looking into any feedback we receive. Please feel free to share your thoughts and the product team will aim to respond.