79

Can we statistically analyze the success of the Welcome Wagon, new Code of Conduct, and related UI changes?

I did some analysis I'll provide as a self-answer below, but new answers and edits are more than welcome!

19
  • 68
    Subjectively, I've noted that the tone is significantly more hostile everywhere now than it was before the changes, particularly on meta (such as this). Ever since the first blog post about these changes, the community has been significantly discontent with SO the company, judging by the numerous highly up-voted meta discussions where significant community consensus is opposing the changes. Quality concerned users don't feel particularly welcome on the site. ->
    – Lundin
    Aug 27, 2018 at 8:30
  • 4
    You can measure that reduced change in friendliness by the current score of +452(!) at the linked post, for example.
    – Lundin
    Aug 27, 2018 at 8:31
  • 5
    I would think that the most definitive answer will come, over time, from outside Stack Overflow. Just watch how many blos/forums/articles/whatever refer to SO as an "aggressive, elitist forum". Once we start seeing less of those we'll know that the changes were possitive.
    – Alejandro
    Aug 27, 2018 at 12:43
  • 9
    @MartinJames For someone who constantly complains about how bad SO is, you sure are here a lot.
    – Clint
    Aug 27, 2018 at 14:25
  • 2
    @Clint on meta, maybe. On SO main. not so much any longer:) Aug 27, 2018 at 14:33
  • 20
    @Alejandro no, that won't happen. The number of users who get really angry because 'their' homework was not done promptly by the SO drones will not change much. Likely, most of those external ranters are, well, external because they got banned/suspended from SO. That must be someone's fault and, since it could not possibly be because of the OP's bad questions, it must be the 'agressive, elitist' curators. Lashing out at the nearest target will not stop. Engaging with those people is like negotiating with terrorists - you just get more terrorists:( Aug 27, 2018 at 14:41
  • 21
    I spend a lot less hours helping people on StackOverflow in 2018 than I did in 2013 because the new users tend to be, overall, more incompetent, and less careful in asking questions that you can actually answer. If you want to make a warm fuzzy hand-holding site where the points are made up and feelings are all that matter, I suggest starting over. icanhaztehcodez.com may still be available.
    – Warren P
    Aug 27, 2018 at 14:53
  • 5
    @Lundin, I don't think the fact that "the tone is significantly more hostile everywhere now" necessarily supports the claim that "Quality concerned users don't feel particularly welcome on the site". I feel like a vocal minority of people have been acting out in protest, but I don't agree that it's a community-wide consensus. I personally find it perfectly viable to be both quality-concerned and also a nice, civil person.
    – Sam Hanley
    Aug 27, 2018 at 15:33
  • 4
    @SamHanley That's what I've seen: a handful of vocal posters driving the line that the sky is falling, that "try to be nicer to folks" translates as censorship and personal attacks. I haven't had any trouble telling people their questions don't belong on SO without the snarky quips (which, according to quality standards, shouldn't have been there under the old rules). Aug 27, 2018 at 18:21
  • 6
    @SamHanley +458 votes on meta is not a minority as meta goes, but exceptional community consensus. The main issue here is that SO the company repeatedly comes up with the wrong focus, implementing peripheral features that nobody asked for, instead of focusing on improving the quality of questions.
    – Lundin
    Aug 28, 2018 at 6:29
  • @Lundin the fact that that post got a high volume of upvotes is just one metric. If you look at the actual comments and answers left on it, it's clear that people have a wide variety of opinions. It's clearly not a case where a post was massively upvoted because people just unequivocally agree with its sentiments. I still firmly believe that while the community's feelings span a spectrum, the folks whose feelings are as extreme as the OP of that post are a very vocal minority.
    – Sam Hanley
    Aug 28, 2018 at 13:20
  • 2
    @SamHanley Up-votes on meta mean "I agree", down-votes mean "I disagree". Currently 580 people (83%) agree and 122 (17%) disagree with the post.
    – Lundin
    Aug 28, 2018 at 14:37
  • 1
    @Lundin, you can't make that assertion across the board. That's a way that votes can be used on Meta, but there's plenty of times I've upvoted a meta post where I don't fully agree with the poster, but I think their post raises something worth discussing.
    – Sam Hanley
    Aug 28, 2018 at 14:41
  • 1
    @Lundin what part of that do you think supports your claim? Here's what I read there, emphasis mine: "On posts tagged feature-request, voting indicates agreement or disagreement with the proposed change in addition to usual reasons based on the quality or usefulness of the post itself. Also, on discussion posts, votes may be used to indicate agreement/disagreement with the author's standpoint."
    – Sam Hanley
    Aug 28, 2018 at 15:26
  • 5
    One of the main criticisms of welcominggate was that the "observations" (and thus, the "conclusions" that have been drawn from them) have not been based on hard facts and data to begin with. Trying to do a statistical analysis now is certainly interesting, but raises crucial (and obvious) questions: How should "friendliness" me measured? And more importantly: If we find out that friendliness has decreased, will there be a rollback of the changes and the CoC? Certainly not. There is an ideology behind all that, permeating not only stackoverflow but our whole culture.
    – Marco13
    Aug 28, 2018 at 17:12

1 Answer 1

29

I pulled year over year (YoY) comparison treatment and control data sets using the Stack Overflow portal of SEDE and this SQL query.

The filters on the data were:

  1. Comments made on first-time questions
  2. Most recent complete data for treatment (August 7 - 19, 2018) - assumes most recent 1 week of data may be incomplete, thus I ignored it
  3. Same days of week and time of month 1 year prior (August 8 - 20, 2017)

I used the R language to read the data, clean the comments' text, and assign Sentiment Analysis scores from the Bing lexicon-sentiment database in the tidytext library.

The full code is available on GitHub.

The sample sizes were similar in size YoY, with a few more in 2018.

I have filtered the words in the list of comments on pre-defined and custom "stop words", which are inconsequential or misleading words (e.g. "and", "if", "stack", "exchange", "overflow", etc, etc). Update: I've now included > 2,000 tag names in this list.

When I do so these are some of the most common remaining words:

enter image description here

The differences between Treatment and Control unfortunately evaporate when I do this:

> # Compare Sentiment YoY
> yoy <- sample_sa %>% 
+   group_by(flag_treatment) %>% 
+   summarize(avg_sentiment_bing_scale = mean(score))
> yoy
# A tibble: 2 x 2
  flag_treatment avg_sentiment_bing_scale
           <int>                    <dbl>
1              0                    0.467
2              1                    0.464

Stay tuned though as I clean the text further and as more time goes by.

At the end I decided to check the correlation between commend upvotes and sentiment. This part was interesting.

It looks like more negative comments tend to receive more upvotes. This may make theoretical sense, given that negative comments are often calling attention to missing / mangled information in the question, and upvotes signify that other users have the same concern or issue mentioned in that comment. This trend significantly increased YoY (you'll see what I mean if you run the R code).

19
  • 1
    It may be statistically significant, but is it clinically significant? 0.03 difference doesn't appear to be much. Aug 26, 2018 at 21:01
  • 3
    Has it been a year yet?
    – user2100815
    Aug 26, 2018 at 22:08
  • 6
    @NeilButterworth Nope - year over year means that we compare last year (control group) vs this year (treatment group). Also known as a "pre/post" comparison.
    – Hack-R
    Aug 27, 2018 at 1:44
  • 25
    "It looks like more negative comments tend to receive more upvotes." I have pointed this out a number of times. Unfortunately scathing, negative, smartass comments are funny, so they tend to get upvotes. I fear that far outweighs the pollyannaish view that it is folks helpfully upvoting constructive criticism! :O For better or worse..
    – Fattie
    Aug 27, 2018 at 7:26
  • 7
    It seems like it would be better to do a monthly comparison, as the majority of this year was not under the influence of the welcoming push.
    – jpmc26
    Aug 27, 2018 at 15:40
  • 17
    What does as.factor(flag_treatment) mean? I don't understand how to read that graph or how it compares multiple years. Aug 27, 2018 at 15:55
  • @Draco18s Sorry, yea I should clean it up. My time was a bit constrained. The as.factor() is just where I had to do a last second change to the data type for data visualization purposes. It's a 1/0 indicator (aka flag, binary variable, dichotomous, etc) that indicates treatment (2018 - Welcome Wagon) vs control (2017 - no Welcome Wagon)
    – Hack-R
    Aug 28, 2018 at 14:33
  • 1
    Thanks, that helps, although I'm not sure what the graph is measuring. Post seems to indicate "number comments" with a filter, but I'm not sure if these are supposed to be "unhelpful" comments or "welcoming ones" or what. Aug 28, 2018 at 14:45
  • @Draco18s The first graph just shows the sample size year over year as some background info. The sentiment analysis are the numbers you see at the end. There would be more material there, except that what I found at this time is no difference between test and control, so I didn't go crazy on the visualization / tables. Once enough time has elapsed for us to see more interesting results, I'll definitely add more.
    – Hack-R
    Aug 28, 2018 at 14:57
  • 3
    @Hack-R I think the confusing part is your nice colorful graph is (no offence) a basically useless plot of participant numbers, while the important part is a grey table. Just make the numbers a single sentence...
    – mbrig
    Aug 28, 2018 at 21:06
  • 1
    @mbrig Done, thanks! You're right, it made more sense in the original version of the question when it was just 1 plot like that in a series. After I cleaned up the data the other plots went away because the gap in the outcome went away.
    – Hack-R
    Aug 29, 2018 at 0:34
  • 1
    This is important guys... when did we change SEDE to SODE? Wasn't it always Stack Exchange's?
    – Braiam
    Aug 29, 2018 at 1:10
  • 1
    It would be nice to make the comparison not only 2017 vs 2018 but for every year since 2010. Maybe there are natural large scale fluctuations. Aug 29, 2018 at 10:41
  • 1
    @Hack-R nice, looks better now :)
    – mbrig
    Aug 29, 2018 at 14:21
  • 1
    @Moody_Mudskipper You're absolutely correct. You know, I have been trying to blacklist misleading/technical words (~2K of them) but really maybe I should just white list a few positive and negative words? Or maybe I should try 2-word and 3-word ("n-gram") groups instead of just individual words, to get the context.
    – Hack-R
    Aug 29, 2018 at 15:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .