36

So it turns out that my invite to the Comment Evaluator 5000 from two weeks ago ended up in my spam bucket. When I discovered it, I went to the evaluator and started answering questions. But in the few minutes I spent at this, every answer was Fine.

Seems like a waste of my time to spend an hour pushing the fine button. I have seen plenty of not fine comments in my time here. But not one in the comment evaluator 5000. Is my tolerance too high? Is there a point to this, or is it just a test of my patience to pushing the same button over and over again?

13
  • Looking for salt? Asking why don't they fight? Looking at all those socially functional human beeing make you sick. The comment evaluator based on [ insert your favorite online game] is what you need ! Commented Oct 10, 2018 at 6:50
  • 50
    'or is it just a test of my patience to pushing the same button over and over again?' - 'This is the story of a man named Stanley. Stanley worked for a company in a big building where he was Employee #427. Employee #427's job was simple: he sat at his desk in room 427 and he pushed buttons on a keyboard.'
    – Script47
    Commented Oct 10, 2018 at 8:00
  • 19
    That is not unusual. They first ran the test on a bunch of SO employees and a high number of them did not find anything objectionable about the comments. A typical "this is a big waste of my free time, bye" is going to bias the test outcome pretty heavily, hopefully they'll measure that as well. Commented Oct 10, 2018 at 10:19
  • 2
    I wasn't invited but found the link somewhere, analysed about 50 comments which all were fine in my opinion and got bored and closed the site, never to return.
    – CodeCaster
    Commented Oct 10, 2018 at 10:21
  • 28
    Oh, so "fine" doesn't mean 'give the commenter a financial penalty?
    – user5940189
    Commented Oct 11, 2018 at 10:18
  • 3
    I hope you at least had a little counter that increased.
    – the_lotus
    Commented Oct 11, 2018 at 15:28
  • 6
    Start with the Comment Evaluator Evaluator and take things from there. On a serious note, it would probably benefit the site more to participating in further development of the bad question/answer detection bots like Smokey. Those bots work pretty well and something similar should eventually be integrated in the site itself, IMO.
    – Lundin
    Commented Oct 11, 2018 at 15:51
  • 3
    I had the same experience. All of the comments I saw were fine so I just stopped doing it.
    – user4639281
    Commented Oct 11, 2018 at 16:00
  • 3
    This'll be a really interesting data dump to look at. Should be all kinds of vectors to draw between users, raters, text length, review time, etc. Each data point helps tell a more accurate story, even (especially!) if it's "these are actually all fine"
    – Undo Mod
    Commented Oct 11, 2018 at 16:41
  • Hey, I want the Comment Evaluator 5000! What is it? Where do I find it? Yes, I'm playing Chumlee ... ;) italiapokerclub.com/wp-content/uploads/2012/06/… Commented Oct 11, 2018 at 17:13
  • 4
    @YvetteColomb It's a very valid question. I've also asked variations of "is this really representative?", since everything feels so... civil. Remember that you and I see a very skewed set of comments in the flag queue. If CE5000 is using representative data, the site as a whole is far better off than we think from what we see.
    – Undo Mod
    Commented Oct 11, 2018 at 17:18
  • 11
    I see nothing wrong with the way it was asked. "Is this a waste of my time" and "is there a point to this" are natural feelings to have after classifying an apparently-homogenous set of data.
    – Undo Mod
    Commented Oct 11, 2018 at 17:20
  • 1
    I'd guess only a small percentage of comments on the site are not "fine", so in order to get a less homogeneous set for evaluation, it would probably make sense to artificially bump up the frequency of comments that have been identified as potentially "not fine". Commented Oct 12, 2018 at 20:35

3 Answers 3

39

I think it would be a large favor, and legitimate benefit, to all parties involved if you continued to hit that fine button, so long as that’s your honest assessment of the comment.

Remember that this is the quantitative data they’re collecting to make future decisions, both on broad policy and individual practical moderation decisions.

12
  • 8
    I can not disagree with the need to express one's opinion, but an hour of nothing but fine is likely to argggg....
    – Stephen Rauch Mod
    Commented Oct 10, 2018 at 5:18
  • 22
    @StephenRauch Oh, I totally understand it’s no fun at all for you. That’s what I meant by “a large favor”. You pay, we benefit. Right now, my concern is that comment moderation policy has been shaped by third parties based on small, anecdotal data. I think the costs I’m asking you to bear can help balance that with a perspective of an active participant to generate more statistically significant data. But it’s your choice —- as I say, this is a favor — and it’s completely fair if you don’t want to do it.
    – Dan Bron
    Commented Oct 10, 2018 at 5:21
  • 40
    @StephenRauch Just think, what if every time you hit "fine", it counted as a rebuttal against Jay Hanlon's blog post?
    – user773737
    Commented Oct 11, 2018 at 0:38
  • 2
    I agree with your answer. If the majority hit fine and a smaller proportion don't always hit fine, it's important to get the statistical significance.
    – user3956566
    Commented Oct 11, 2018 at 16:47
  • 3
    @Houseman I think it's gone beyond the blog post. Whether the blog post was mismanaged or not, the fact remains, the site was clearly intent on making changes. The blog post was the first public indicator of it. It's like throwing the baby out with the bath water to say "yeh we were always perfect - the blog post sucked!" We had issues. Every site does. Making snipey remarks doesn't help address the issues, what side of the fence you sit on. It just makes people think "give it a rest". If people object to how things are, address it intelligently.
    – user3956566
    Commented Oct 11, 2018 at 16:50
  • 1
    Maybe they should add the same measures to the comment evaluator that they do to the review queues to make sure you aren't robo-reviewing :P "oops, this comment wasn't actually fine." But they need our data first to determine that. Commented Oct 11, 2018 at 17:38
  • 8
    @YvetteColomb Was my comment really "snipey?" I was being sincere...
    – user773737
    Commented Oct 11, 2018 at 17:46
  • 3
    @Houseman it comes across that way. The blog post is not why we're seeing the survey. The blog post is an outward statement coming from the site. The survey is also coming from the site. It's not Jay's blog. He's representing the site. It went through approval processes. It's the entire impetus of change that we're dealing with, not an isolated blog. Rallying against the blog is meaningless. The site has made it's choice. What we can best do is try and negotiate with feedback, for e.g. when the CoC was featured on meta. with surveys like this. They're clearly not allowing us to dictate the...
    – user3956566
    Commented Oct 11, 2018 at 17:57
  • terms. We can give feedback as to some of the nuances. Does that make sense? It's time people accepted the network is not going back on this. We have no choice. So what are we going to do to work within this new framework?
    – user3956566
    Commented Oct 11, 2018 at 17:58
  • @YvetteColomb What are we going to do to work within this new framework? We can provide the owners with quantitative data so that they can make the right decisions by participating in the CE5K. It is my hope that the owners see this data and say "wow, we were completely off-base"
    – user773737
    Commented Oct 11, 2018 at 18:13
  • @Houseman I honestly don't know. It's stressful. It's nigh impossible not offending someone with all the personalities and cultural differences on a global site. My motto is, would a reasonable person think this is rude. then the debate starts "what is a reasonable person"
    – user3956566
    Commented Oct 11, 2018 at 18:23
  • 2
    @YvetteColomb You seem pretty reasonable to me, so I'd go with "what Yvette thinks".
    – wizzwizz4
    Commented Oct 12, 2018 at 20:02
4

Yes there would be a point to it, it's not a test of patience, but contributing on this site can be for all of us.

As for your tolerance being too high, no one can answer that within the frame of that question. Everyone is different.

I'm sure there's some people who are not finding all the comments as fine. So if the people who have a broader tolerance of what is fine quite the survey part way. They will skew the votes into a greater representation of "not fine" than there should be and also possibly miss bad comments that appear later in the survey.

As much as it seems a waste of time it's not. If you saw some of the comments that are flagged as not nice or abusive that are clearly ok (IMO), it's important to register your voice (IMO). Otherwise the site will be dictated to by a norm that is not truly representative of the norm. (using norm == bell curve norm)

3
  • 4
    Thanks for this, but the 5000 is evidently not showing many of the nasty ones. From an engagement point of view (IE: don't bore your participant to tears) it would be a good idea to show some of THAT...
    – Stephen Rauch Mod
    Commented Oct 12, 2018 at 4:32
  • @StephenRauch I found out they don't show the same comments for every survey. Though you might be right, throwing in some more nasty ones may help to keep people's attentions. It's a design flaw of the survey.
    – user3956566
    Commented Oct 12, 2018 at 22:38
  • 1
    @StephenRauch I thought the comments being borderline was the whole point
    – BSMP
    Commented Oct 13, 2018 at 6:08
0

I've evaluated a number of comments and found that overwhelmingly they are just fine. That's good to know and it's even better if we can verify that several people agree with that evaluation. (It's not obvious, but comments are potentially evaluated by other respondents.) I maintain that, on average, comments on Stack Overflow are among the best on the internet and it's nice to have data to back up that assertion.

However, I've also run across comments that are definitely not fine. Some seem likely to shake the confidence of the author (likely without having the intended effect of discouraging participation). And others are definitely not fine. I'm not sure how long to suggest looking for not-so-productive comments, but in my experience, they tend to be clustered on specific posts. So it's possible to go through several pages of the evaluator without seeing anything wrong only to run across a post with a whole mess of less-than-productive comments. (My "favorite" are the ones from high-reputation users complaining about upvotes.)

Whether or not the process of clicking "Fine" is worth your time depends entirely on you, of course. I've done this sort of exercise in the past and it's hard to power through a bunch of dull comments. And it turns out many comments on the site are just sorta uninteresting taken out of context. The good news is that they tend to be pretty quick to evaluate in this experiment.

If we are able to successfully build a model that can classify comments, it seems worthwhile to skip posts with nothing by innocuous comments. But building that model requires getting feedback on a healthy sample of evaluations. So it's kinda a chicken-and-egg problem.

2
  • 1
    so you'd need something like a bunch of people who you can statistically say have a normalized view of what is and isn't OK, and then have them rate a metric ton of comments so you can train an AI algo with that?
    – Magisch
    Commented Oct 15, 2018 at 7:33
  • @Magisch: I'm not sure how many people have evaluated how many comments, but my impression is that we are collecting tons of data. (I don't know what measurement system that's in, however.) The goal is to be able to guess what comments are welcoming based on their content.
    – Jon Ericson Staff
    Commented Oct 16, 2018 at 20:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .