5

Note for context: this post is part of a series about Collectives on Stack Overflow.  To read the full series, begin at this primary post.

This is a post that’s certainly overdue. As we look ahead to the evolution of Collectives, taking a candid look back is important.

The desire for a level of subcommunity on Stack Overflow is a finding based on a great deal of research. We decided to first explore the idea of subcommunities on Stack Overflow with the Collectives concept, and the first iteration of that was a commercial offering of provider-specific collectives. The commercial offering was definitely the more challenging path, for a variety of reasons. But this initial iteration resulted in a number of learnings that have been beneficial and helped hone the idea, just as a beta release should.

What did we learn?

  • We learned that there is a post-join increase in user engagement. For users who joined a collective, their activity in the collective’s tags increased by about 30% afterwards, compared to before joining. This was extremely encouraging from early on, validating that there is merit in creating a focused space.

  • However, we also learned that this increase is a fairly abstract measurement and it doesn’t really manifest in the day-to-day for users; there was not a big feeling of added value for the community. People were helping out more in the specific content areas, and more people answering and voting is what led to increased content quality. Some community members have found value in tag-specific leaderboards or the opportunity to engage directly with the creators of the technologies they work with, but there was still no broadly noticeable feeling of added benefit.

  • We learned that sponsoring organizations are hesitant to designate members of the community as Recognized Members, since that role comes with the ability to give the organization’s stamp of approval/endorsement on answers and articles. These mechanisms to engage with and recognize contributions by the community are key interactions that tech providers can have with collective members within collectives. We’re working closely with customers on this subject and have explored a number of approaches to help.

  • We learned that the success of a sponsored collective means having a good connection to the organization’s Developer Relations team – working with them on what’s going to benefit their developer community the most and what’s going to help them show that value to their leaders (and budget holders). DevRel teams often struggle to quantify the ROI for their work, and showcasing that value is key. With a beta product where we’re still experimenting with implementation, there are bound to be some misses, but every instance of a collective has resulted in an improved approach.

What could have been better?

Trust was damaged because the specifics of the Collectives product were initially crafted without much transparency or community input (and the community members that were engaged were placed under an NDA, so they could not share the details publicly). Starting with a commercial offering forced that in some ways, but we absolutely could have taken a better approach even with that limitation.

Trust was also damaged by the rollout of Articles without sufficient community discussion ahead of time – specifically without deeper consideration of how a new form of knowledge content would affect norms and policies, as well as the ripple effects on the reputation system. Research showed a broad interest in long-form content on Stack Overflow, but as always, the devil is in the details, and a slower and more thoughtful rollout would have benefited everyone.

Lastly, the perception of Collectives being solely a revenue generator is something we should have expected and planned better for. Trust was damaged there as well. The long-term vision of this product being a mutually beneficial addition to Stack Overflow could have been articulated more strongly. We are very much invested in being a product-led company with the community at the center, and we've been extremely deliberate about coaching customers how to engage with the community and pushing them to invest meaningfully in the space. But many in the community perceived Collectives as another form of advertising. Exploring new types of commercial opportunities like Collectives enables us to continue investing in our core experiences, with improvements like Staging Ground and Moderator tooling. Sponsorships are a key component of Collectives, but the main purpose of Collectives is to enable focused sub-communities within Stack Overflow to learn, share, and grow together within a dedicated space.

What has been gained?

Opinions on this question are sure to be varied. As a community manager who has been deeply involved in Collectives for almost a year now, I’ll give my take on where we currently are.

We’ve managed to put a beta product through its paces and arrive at something that does (some of) what we’d hoped – notably, increasing engagement in the subject area and teaching us about what works and what needs more iteration. This all happened without negatively impacting the core community in a big way, though certainly, we’d hoped the positive impacts would be more apparent to the community. Now, as we look ahead to bringing Collectives into a new stage and additional method of implementation, we’ve got a product that’s been iterated on and improved upon without disrupting what works well.

This “iterating on the sidelines” approach was part of the initial vision for Collectives. To quote Teresa, our Chief Product Officer:

Collectives give us the opportunity to experiment in ways that won’t leak features out. This enables new features and capabilities to be tested without endangering the underlying knowledge base, and - to meet our first priority - they fill an actual market need.

We believe they also allow us to experiment and iterate more rapidly on features in a smaller, contained way.

It’s been a journey getting to where we are now, and we are grateful for the candor, ideation and collaboration from those who’ve taken part in the Meta discussions, taken part in research and participated in the collectives launched thus far. We look forward to all of that continuing as we look ahead to working with the community and continuing to iterate on Collectives. You can read more about that on this post and the others linked from it.

6
  • 7
  • 1
    @SecurityHound We'll continue to work with the provider collective customers on how to adhere to citation guidelines and do proper attribution. And we'll do the same with the new collectives, working with the Recognized Members who will be doing article review.
    – Berthold StaffMod
    Commented Feb 1, 2023 at 20:22
  • 5
    Do we have any data that would indicate why there was increased engagement from those who joined a collective? Is it just a nested layer of further gamification on top of rep? Does it last?
    – Kevin B
    Commented Feb 1, 2023 at 21:02
  • 9
    30% alleged increase in activity but "it doesn’t really manifest in the day-to-day for users"? That seems a bit odd. A bit like a phantom. And "no broadly noticeable feeling of added benefit" sounds really worrying. Also not sure about "a slower and more thoughtful rollout would have benefited everyone" in relation with Articles. There were like 10 articles after half a year. It's almost not possible to rollout slower. Commented Feb 1, 2023 at 22:14
  • I remember a post that said it would introduce some 'automated processes' to identify recognized members but I doubt if they are introduced yet? I've usually been in top 3 in the GCP collective and related even before collectives were introduced, its sad to see people leaving a collective just because of this. This doesn't stop any of us from contributing but I've had cases where OP wants a recognized person to answer even though our answer is correct.
    – Dharmaraj
    Commented Feb 2, 2023 at 16:31
  • 2
    @Dharmaraj It's been interesting working with the different customers about how the RM role will work in practice, since each one has different ideas about how that ties into their larger developer relations strategy. (Ref my third bullet in the "learnings" section). That's part of why it's been slow-going to promote community members. The automated processes are in place and we (SO) are happily endorsing community members, but things are just moving slowly in some cases as the the respective plans are coming into place.
    – Berthold StaffMod
    Commented Feb 2, 2023 at 18:04

3 Answers 3

27

There don't seem to be many people at Stack Overflow who actually built Stack Overflow.

Perhaps a lesson to be learned from attempts at making features is that the community, who was here the entire time Stack Overflow was fleshed out and iterated on, should be included.

There was no real question asked here though, so it is hard to really give a strong response. It is good to post reflective discussions, however, recently it would appear that posts are coming across more as blogs, with the answer section available to merely comment on the work produced.

As a result, answers here are likely to appear like mine or Makoto's: topical to the post, but without strong direction.

I think the community would really appreciate, going forward, if it were slightly more involved with at least a dialogue. No one has to actually use any of the suggestions the community provides, but you might find a surprisingly large amount of value there which is not being realized.

3
  • 1
    Yeah but what is "the community?". Who would the company engage with? The easy answer would be "meta", but is that really representative at this point? Especially after that dark period where employees didn't dare to look at meta because it gave people anxiety issues?
    – Gimby
    Commented Feb 6, 2023 at 15:42
  • 1
    I mean... meta/chat is the best they have for interacting with the community. You can't force users to want to interact with staff, but not interacting with those who want to is certainly a missed opportunity, at the minimum. (That's not to say i think they aren't interacting, I think they are, i just wish it didn't feel so adversarial.)
    – Kevin B
    Commented Feb 6, 2023 at 15:45
  • 1
    @Gimby - Creating a dialogue that anyone in the community can take place in would be best suited for meta. Only using yes people from a mail group isn't representative of anything but a bunch of ego satiating. Regardless of the type of communication though, there isn't really outreach right now. These features are simply announced as something that the current new hires have thought of as interesting. There shouldn't be this much turnover of decision makers in a company like this, and it would more than likely improve if they actually took part in the community rather than talking over it.
    – Travis J
    Commented Feb 6, 2023 at 20:42
24

In this specific lessons-learned, one common theme sticks out more than any of them.

Trust was damaged.

By now y'all should realize that trust has an HP bar and thus is not finite, and given that you've cited...specifically three pointed things in which there were attacks that depleted it, I'm surprised that one of these takeaways isn't how to rebuild that trust or what can be done to soften the body blows to it.

As in...I think that this paragraph is a tad too optimistic for such a pragmatic realization.

We’ve managed to put a beta product through its paces and arrive at something that does (some of) what we’d hoped – notably, increasing engagement in the subject area and teaching us about what works and what needs more iteration. This all happened without negatively impacting the core community in a big way, though certainly, we’d hoped the positive impacts would be more apparent to the community. Now, as we look ahead to bringing Collectives into a new stage and additional method of implementation, we’ve got a product that’s been iterated on and improved upon without disrupting what works well.

If you're talking about damaging trust, then you've very much negatively impacted the core community in a big way, and you've very much disrupted what works well. It just may not be immediately obvious or apparent.

But it'll show up, one way or another, and this was your golden opportunity to assess what that damage was and how to start repairing it.

20
  • 4
    Trust (like HP!) is regained through actions and over time, not by words. This post offers candor and transparency and (even if perhaps too much for your taste) some optimism. I have no expectation that this was a magic wand to wave and then call it a day. We have lots of work ahead and the hope is that those who have been frustrated or disappointed can keep an open mind as we continue forward.
    – Berthold StaffMod
    Commented Feb 2, 2023 at 1:26
  • 10
    @Berthold: How long are you planning on making us wait, for real though? We tried to open the communication about this a long time ago and this just kept snowballing until Teresa said "this is the way, we're not questioning it anymore" and we just got to live with this. I get that you're just the messenger and you're not really the person who this message is meant for, but "we have lots of work ahead" is like...the most tone-deaf thing you could've said here.
    – Makoto
    Commented Feb 2, 2023 at 2:00
  • 3
    As in, yes, you do need optimism and yes there's a long road to get back trust. But you haven't even started this. You've just pretended for a year and change that while this has rolled out that things were alright or within parameter. Seriously?!
    – Makoto
    Commented Feb 2, 2023 at 2:00
  • 6
    I guess what I want to hear is what you're committing to do to rebuild that trust. I don't need magic wands or any of that, I'm a practicalist. How are y'all going to start fixing this relationship based on those three points you highlight? That is what I want to hear about.
    – Makoto
    Commented Feb 2, 2023 at 2:01
  • 5
    For long term users of the site, there was no trust to lose in the first place. It was in steady decline for a long time, then finally completely and seemingly permanently(?) erased 3 years ago. There hasn't been much in the way of actions to rebuild it even before this "Collectives" debacle.
    – Lundin
    Commented Feb 2, 2023 at 8:03
  • 2
    @Makoto this whole set of posts is laying out the vision for this next set of collectives that will be community-shaped and community-led. With a different sponsorship model that cleanly separates the commercial elements and community elements of collectives. That is the plan. It's all there. The community feedback, sentiment and input has been a huge factor in mapping out this vision. If you feel that something vital is missing, please be specific. We're working to be transparent and forthcoming, and it will take time, but please don't take the stance that we're doing and saying nothing.
    – Berthold StaffMod
    Commented Feb 2, 2023 at 18:21
  • 1
    @Berthold: I'm not naive when it comes to the work that needs to be done to rebuild trust. But I haven't seen anything that you've done to do that. I'm simply told that you're working on it. How much longer yet are we going to keep us waiting? Why wasn't this post seized as the opportunity to showcase - transparently and publicly - that you were serious about doing this? Why do I have to be placated with "we're working on it, please understand, thank you for your patience"? Y'know... again??
    – Makoto
    Commented Feb 2, 2023 at 18:28
  • 4
    To rebuild trust we are creating the next set of collectives which will be community-shaped and community-led, with a different sponsorship model that cleanly separates the commercial elements and community elements of collectives, incorporating the feedback and input from the community provided since the initial launch of Collectives, and working to do this in a transparent and collaborative way. It's a lot to cover in one post, that's why there's five.
    – Berthold StaffMod
    Commented Feb 2, 2023 at 18:55
  • 2
    @Makoto I'd say that is accurate. There is a general confidence, grounded in extensive research, that the core idea of "a subcommunity on Stack Overflow around a set of tags and/or area of practice" is worth exploring and has lots of potential benefit. And similar confidence that a long-form content format could exist effectively and beneficially alongside Q&A. Due to the choice to first explore these ideas with a commercially-focused implementation, that is the only way that the community was able to experience them.
    – Berthold StaffMod
    Commented Feb 2, 2023 at 21:47
  • 1
    Separate from the reception of the commercial aspects, the rollout could have been done better, and there have also been the expected ups and downs that are part of any beta product's launch. For some in the community, I think those core ideas may be irrevocably tainted by the first implementation, the difficult rollout, and the beta learning process. The core ideas are being brought forth again in another implementation. There will be learnings, as before. As I said, the hope is that those who have been frustrated or disappointed can keep an open mind as we continue forward.
    – Berthold StaffMod
    Commented Feb 2, 2023 at 21:47
  • 3
    @Makoto- Tangential to the efforts that Berthold discusses, there are other steps we are taking to rebuild trust that are not directly tied to Collectives. Those include things like our massive investment in the community team, in our community products team, and in the work that is ongoing around mod tools and the discovery work toward anti-plagiarism tooling that is happening now. These are directly in response to community requests. We've also requested moderator input on our rollout of the new collectives and adjusted plans based on their input. This is all trust-building, at the core.
    – Philippe StaffMod
    Commented Feb 3, 2023 at 6:19
  • 4
    @Philippe: None of that speaks to anything about what caused the trust to be lost to begin with, nor does it really confront the painful elephant in the room around how Meta users are perceived when giving feedback. But again, I'm hearing that you're doing "something" and I'm just disagreeing with the notion that this "something" is going to actually manifest in any trust rebuilt with the community at large.
    – Makoto
    Commented Feb 3, 2023 at 17:06
  • 4
    On the issue of trust... Sending what amounts to PR reps out to explain to the community exactly what is going to happen to them is a really poor read of the room. Stack Overflow is a community of developers. We all understand the actual nuts and bolts of these operations, and we do better discussing tangible aspects of implementation rather than fanciful descriptions of other people's work. Coming here and saying "I have a background managing communities so trust me, this is going to work" is a rather absurd way to build trust. It comes across as very disingenuous.
    – Travis J
    Commented Feb 3, 2023 at 18:45
  • 3
    It is also very hard to buy into the notion of a "massive investment in the community team" when we all witnessed what happened to the core group of community leadership that we all actually did trust. That group was also composed of people who actually made things, as opposed to just talking about which things were made.
    – Travis J
    Commented Feb 3, 2023 at 18:47
  • 5
    "We've also requested moderator input on our rollout of the new collectives and adjusted plans based on their input." Ehh…sorta. When the moderator team (several of us, including yours truly) noted that one of the proposed/planned Collectives was going to be focused on a subject that is borderline off-topic for SO, Berthold (and presumably the rest of the team involved, although he was the only one who spoke to us directly) was responsive enough to drop that particular Collective from the plans. However, mods weren't involved prior to that, which is at least part of the problem.
    – Cody Gray Mod
    Commented Feb 4, 2023 at 8:50
6

We learned that there is a post-join increase in user engagement. For users who joined a collective, their activity in the collective’s tags increased by about 30% afterwards, compared to before joining. This was extremely encouraging from early on, validating that there is merit in creating a focused space.

Was this randomized in the form of an A/B test, where half the users were offered the ability to join and the other half wasn't?

Do you really think that the act of clicking join makes people contribute more? Is it not likely that those who are in a phase of contributing more will join?

These sorts of stats sound nice but they should not be interpreted as causal.

Could this increase be because you served more of the tags questions to them? Or because the tags were marked up with colorful badges? Sounds like this could be a factor. And would not mean at all that there is "there is merit in creating a focused space."

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .