38

Collectives were presented (Beta release of Collectives and The user research behind collectives) as a way to increase participation, for example by reducing barriers to entry or by forming communities or by calculating and displaying leaderboards. And the number of members in the respective collectives (4-5k) looks impressive.

Is there already such an effect visible? Has the activity, participation or quality of recently generated content significantly improved in the tag sets defining the two existing collectives?

Quantity of created content could have improved or quality or both or maybe no change is visible yet.

It may depend on how quality of content or activity is measured. Some possibilities would be:

  • number of (positively scoring) questions asked
  • number of answers given (and upvoted)
  • fraction of questions answered
  • number of answers per question
  • number of unique answerers
  • number of votes given

See for example How is question quality measured in A/B tests for a possible definition of question quality.

To see if a change is significant one could take one month after the introduction and compare to a couple of months before possibly with a seasonal adjustment.

What makes gathering the statistics a bit difficult is that the tag sets are quite large and I'm not sure how to efficiently search within them.

12
  • 1
    I think with some SEDE quieres one should be able to come up with some numbers. Currently I do not have time to try myself on it. Commented Jul 24, 2021 at 16:06
  • 1
    Unless the company provides those numbers, we'd have to wait at least another 30 days to make sure that all questions asked in that timeframe had time to be roombad.
    – BDL
    Commented Jul 24, 2021 at 16:27
  • 6
    looks like not enough people downvote on collectives tag
    – nbk
    Commented Jul 24, 2021 at 17:50
  • 2
    @BDL You're right, it's better to wait 30 days more, otherwise statistics will be distorted. Commented Jul 24, 2021 at 19:14
  • @nbk Now or before? You probably mean that fewer downvotes not necessarily mean higher quality. Commented Jul 24, 2021 at 19:16
  • 1
    no i doubt you will find any difference, eitehr in the question or answer independent of collectives. It will not influence any of the quality. Company members answer often question of their product, only some like oracle doesn't do that, but i thing they take a look here.
    – nbk
    Commented Jul 24, 2021 at 19:22
  • 7
    I'm interested in more important curation metrics: meta.stackoverflow.com/a/408770/2943403 Commented Jul 25, 2021 at 12:09
  • 3
    Isn't it a bit too early to call for an evaluation.
    – 4386427
    Commented Jul 25, 2021 at 14:35
  • 1
    @4386427 Yes, we need to wait at least 30 days more to grow out of the roomba deletion period, but otherwise I think that period should be enough to get statistical certainty and be able to reliable detect even small changes should they have happened. If the positive effects need longer to manifest themselves, we could easily redo the analysis later on. Commented Jul 25, 2021 at 15:35
  • 1
    @mickmackusa Feel free to gather and display important curation statistics. Curation is part of the activity, participation and quality. Commented Jul 25, 2021 at 15:39
  • 1
    eh, simply looking at the one month after introduction would give you participation during the launch period, which is useful information, but isn't necessarily a good measure of how it will work when the spotlight isn't aimed at it.
    – Kevin B
    Commented Jul 26, 2021 at 15:08
  • 1
    @KevinB Sure, there might be a novelty effect, that wears off over time or it could be a slow build up effect that is not yet visible. If more times become available, it's not a big problem to redo the analysis to get a more complete picture. Commented Jul 27, 2021 at 7:17

0

You must log in to answer this question.