Note for context: this post is part of a series about Collectives on Stack Overflow. To read the full series, begin at this primary post.
This is a post that’s certainly overdue. As we look ahead to the evolution of Collectives, taking a candid look back is important.
The desire for a level of subcommunity on Stack Overflow is a finding based on a great deal of research. We decided to first explore the idea of subcommunities on Stack Overflow with the Collectives concept, and the first iteration of that was a commercial offering of provider-specific collectives. The commercial offering was definitely the more challenging path, for a variety of reasons. But this initial iteration resulted in a number of learnings that have been beneficial and helped hone the idea, just as a beta release should.
What did we learn?
We learned that there is a post-join increase in user engagement. For users who joined a collective, their activity in the collective’s tags increased by about 30% afterwards, compared to before joining. This was extremely encouraging from early on, validating that there is merit in creating a focused space.
However, we also learned that this increase is a fairly abstract measurement and it doesn’t really manifest in the day-to-day for users; there was not a big feeling of added value for the community. People were helping out more in the specific content areas, and more people answering and voting is what led to increased content quality. Some community members have found value in tag-specific leaderboards or the opportunity to engage directly with the creators of the technologies they work with, but there was still no broadly noticeable feeling of added benefit.
We learned that sponsoring organizations are hesitant to designate members of the community as Recognized Members, since that role comes with the ability to give the organization’s stamp of approval/endorsement on answers and articles. These mechanisms to engage with and recognize contributions by the community are key interactions that tech providers can have with collective members within collectives. We’re working closely with customers on this subject and have explored a number of approaches to help.
We learned that the success of a sponsored collective means having a good connection to the organization’s Developer Relations team – working with them on what’s going to benefit their developer community the most and what’s going to help them show that value to their leaders (and budget holders). DevRel teams often struggle to quantify the ROI for their work, and showcasing that value is key. With a beta product where we’re still experimenting with implementation, there are bound to be some misses, but every instance of a collective has resulted in an improved approach.
What could have been better?
Trust was damaged because the specifics of the Collectives product were initially crafted without much transparency or community input (and the community members that were engaged were placed under an NDA, so they could not share the details publicly). Starting with a commercial offering forced that in some ways, but we absolutely could have taken a better approach even with that limitation.
Trust was also damaged by the rollout of Articles without sufficient community discussion ahead of time – specifically without deeper consideration of how a new form of knowledge content would affect norms and policies, as well as the ripple effects on the reputation system. Research showed a broad interest in long-form content on Stack Overflow, but as always, the devil is in the details, and a slower and more thoughtful rollout would have benefited everyone.
Lastly, the perception of Collectives being solely a revenue generator is something we should have expected and planned better for. Trust was damaged there as well. The long-term vision of this product being a mutually beneficial addition to Stack Overflow could have been articulated more strongly. We are very much invested in being a product-led company with the community at the center, and we've been extremely deliberate about coaching customers how to engage with the community and pushing them to invest meaningfully in the space. But many in the community perceived Collectives as another form of advertising. Exploring new types of commercial opportunities like Collectives enables us to continue investing in our core experiences, with improvements like Staging Ground and Moderator tooling. Sponsorships are a key component of Collectives, but the main purpose of Collectives is to enable focused sub-communities within Stack Overflow to learn, share, and grow together within a dedicated space.
What has been gained?
Opinions on this question are sure to be varied. As a community manager who has been deeply involved in Collectives for almost a year now, I’ll give my take on where we currently are.
We’ve managed to put a beta product through its paces and arrive at something that does (some of) what we’d hoped – notably, increasing engagement in the subject area and teaching us about what works and what needs more iteration. This all happened without negatively impacting the core community in a big way, though certainly, we’d hoped the positive impacts would be more apparent to the community. Now, as we look ahead to bringing Collectives into a new stage and additional method of implementation, we’ve got a product that’s been iterated on and improved upon without disrupting what works well.
This “iterating on the sidelines” approach was part of the initial vision for Collectives. To quote Teresa, our Chief Product Officer:
Collectives give us the opportunity to experiment in ways that won’t leak features out. This enables new features and capabilities to be tested without endangering the underlying knowledge base, and - to meet our first priority - they fill an actual market need.
We believe they also allow us to experiment and iterate more rapidly on features in a smaller, contained way.
It’s been a journey getting to where we are now, and we are grateful for the candor, ideation and collaboration from those who’ve taken part in the Meta discussions, taken part in research and participated in the collectives launched thus far. We look forward to all of that continuing as we look ahead to working with the community and continuing to iterate on Collectives. You can read more about that on this post and the others linked from it.