19

I just read the new blog post about the comment rating experiment here: Welcome Wagon: Community and Comments on Stack Overflow

In the post, they separate raters out into categories of recruited by blog post, opt in research list, moderators, and employees.

I wonder, since I partook in the comment rating survey and responded both to the blog post and am in the research opt in list, which category do my ratings count towards? Both? One?

I'm curious about this one to better interpret the data.

3
  • 52
    Interesting how employees find not an insubstantial amount more comments unwelcoming/abusive than any other group 🤔 Commented Dec 4, 2018 at 16:19
  • Interesting how users with a rep going from 1k to about 12k are the most abusive. And start to chill out after getting past 12k leading to 100k - The most chilled out users seem to with reps over 750k
    – Piotr Kula
    Commented Dec 7, 2018 at 15:20
  • 2
    @ppumkin I'm interested in how you determined that, the scatter plot displays perceived unfriendliness during reviewing, i.e., the reputation shown is that of the reviewer rather than the author of the comment. Commented Dec 8, 2018 at 4:53

1 Answer 1

13

It's possible that you are in either one, but it is much more likely that you are in the "Welcome Project" group. When folks responded to that survey on the blog post in April, they had the option of giving a link to their Stack Overflow or Stack Exchange account page and/or their email address. If you gave a URL for your account or an email address that is associated with your account, then you were counted as part of the "Welcome Project" group.

Some folks who responded to that survey only gave text answers without account information (which is of course fine). If you were one of those, then you must have been invited as part of the research list and were counted in that group.

For the record, every person who responded to the survey on the blog post in April who said it was OK to contact them was invited to participate, all SE mods were invited to participate, and a few thousand folks from the research list were invited.

However you were invited, I want to extend my appreciation and thanks for your time in helping us understand these issues better. It really has been eye-opening and helpful for us internally to get a better handle on the situation and to understand where our own perception differs from others.

13
  • 3
    I want to throw my two cents out here and say that this research was quite fun, in my opinion, to take part in. Yeah, it was heavily opinion based, as these things will be, but it's for a good cause. Keep up the good work, y'all.
    – Kendra
    Commented Dec 4, 2018 at 15:45
  • 16
    Quote: "A typical "this is a big waste of my free time, bye" is going to bias the test outcome pretty heavily, hopefully they'll measure that as well". You didn't. Do consider to not "handle" anything based on biased test results and very poor correlations. Commented Dec 4, 2018 at 15:52
  • 11
    @HansPassant Users choosing not to participate will bias the results if there is a differential perception of unfriendliness/abuse with willingness to participate. Perhaps that is what you are hypothesizing here? We don't see evidence for that, i.e. no relationship between number of comments rated and rate of unfriendliness/abuse.
    – Julia Silge StaffMod
    Commented Dec 4, 2018 at 15:59
  • 16
    @Julia I dont know how you can measure the fact that people (like me) only ever saw comments that were fine and decided that you were wasting their time. You are very very much more likely to get reviews from people who can find anything at all offensive than you will get someone who correctly marks every comment they see as fine. That will inherently bias your results towards the opinions of people who can find anything at all offensive, and there's no way to measure that. Therefore your results are useless.
    – user4639281
    Commented Dec 4, 2018 at 21:20
  • 8
    @TinyGiant That reminds me of French government doing some "based on confidential secret state surveys for which we can't reveal the methodology, we decided to pass this law which contradicts the conclusions of the public open surveys from journalists"
    – Cœur
    Commented Dec 5, 2018 at 13:17
  • 2
    @TinyGiant It seems to me that Julia's comment above already addresses that; there's apparently no relationship in the data between how many comments someone rated and how soon they dropped out. Naturally, the data won't include people who decided to never even try participating, and that may introduce some unmeasurable bias, but people like you - who tried briefly and then gave up because the system wasn't feeding them enough offensive comments - are detectable in the data in principle, and apparently are too rare to show up.
    – Mark Amery
    Commented Dec 5, 2018 at 13:25
  • 4
    @TinyGiant We included folks who worked for about 15 minutes or more, so if you personally worked at least that long, then your data was included in the analysis. There are plenty of examples of users who saw only fine comments (in fact, the average unfriendliness percentage per user was zero) and like I mentioned before, we don't see a relationship between the number of comments rated and the rate of unfriendliness/abuse.
    – Julia Silge StaffMod
    Commented Dec 5, 2018 at 13:28
  • 3
    In my specific case, the reason why I felt that it was a waste of time, is because the lack of the "this comment is useless" option. I had to say that it was "Fine" when it didn't actually reflect my current evaluation.
    – Braiam
    Commented Dec 5, 2018 at 13:40
  • @Braiam It wasn't a survey of finding comments that weren't useful, so why would it have that option? Commented Dec 5, 2018 at 13:43
  • 2
    @GrumpyCrouton because it should have the opposite of a "Fine" option. If the objective was finding which comments where unwelcoming/abusive a likert scale would be the go-to in these kind of studies.
    – Braiam
    Commented Dec 5, 2018 at 13:50
  • @Braiam I gotcha, I think I misunderstood your initial comment. Commented Dec 5, 2018 at 13:51
  • 1
    @JuliaSilge Did you mean "the median unfriendliness percentage per user was zero"? Otherwise, the average (of non-negative numbers) being zero implies that every sample was zero... Commented Dec 7, 2018 at 12:08
  • @MaxLanghof No, the mean of the individual percentages (percentage of comments rated unwelcoming or abusive) is zero, while the median was higher. Most users saw no unproblematic comments but some raters saw more than none.
    – Julia Silge StaffMod
    Commented Dec 8, 2018 at 3:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .