Can we statistically analyze the success of the Welcome Wagon, new Code of Conduct, and related UI changes?
I did some analysis I'll provide as a self-answer below, but new answers and edits are more than welcome!
Can we statistically analyze the success of the Welcome Wagon, new Code of Conduct, and related UI changes?
I did some analysis I'll provide as a self-answer below, but new answers and edits are more than welcome!
I pulled year over year (YoY) comparison treatment and control data sets using the Stack Overflow portal of SEDE and this SQL query.
The filters on the data were:
I used the R language to read the data, clean the comments' text, and assign Sentiment Analysis scores from the Bing lexicon-sentiment database in the tidytext
library.
The full code is available on GitHub.
The sample sizes were similar in size YoY, with a few more in 2018.
I have filtered the words in the list of comments on pre-defined and custom "stop words", which are inconsequential or misleading words (e.g. "and", "if", "stack", "exchange", "overflow", etc, etc). Update: I've now included > 2,000 tag names in this list.
When I do so these are some of the most common remaining words:
The differences between Treatment and Control unfortunately evaporate when I do this:
> # Compare Sentiment YoY
> yoy <- sample_sa %>%
+ group_by(flag_treatment) %>%
+ summarize(avg_sentiment_bing_scale = mean(score))
> yoy
# A tibble: 2 x 2
flag_treatment avg_sentiment_bing_scale
<int> <dbl>
1 0 0.467
2 1 0.464
Stay tuned though as I clean the text further and as more time goes by.
At the end I decided to check the correlation between commend upvotes and sentiment. This part was interesting.
It looks like more negative comments tend to receive more upvotes. This may make theoretical sense, given that negative comments are often calling attention to missing / mangled information in the question, and upvotes signify that other users have the same concern or issue mentioned in that comment. This trend significantly increased YoY (you'll see what I mean if you run the R code).
as.factor(flag_treatment)
mean? I don't understand how to read that graph or how it compares multiple years.
Commented
Aug 27, 2018 at 15:55
as.factor()
is just where I had to do a last second change to the data type for data visualization purposes. It's a 1/0 indicator (aka flag, binary variable, dichotomous, etc) that indicates treatment (2018 - Welcome Wagon) vs control (2017 - no Welcome Wagon)