5

First off: I didn't get an invitation for the Comment Evaluator 5000™; I learned about it through related questions here on Meta. I tried it out anyway, so please let me know if I'm not supposed to do so.

Up to now, I spend about 30 minutes rating those comment threads shown to me. I just noticed that a thread of four comments was shown to me for the second time. Why are those comments presented to me again? Is this a bug or is this by intention to make sure that users provide consistent results?

4
  • @CamiloTerevinto: Well, I'm reading quite a lot of posts here on Meta and I followed all "welcome" and "CoC" discussions. Also, I got a few information from related questions. Maybe my evaluations are not perfect, but at least I tried my very best...
    – honk
    Commented Oct 2, 2018 at 13:43
  • @CamiloTerevinto: I agree, we cannot be sure. To my defense: I saw this post which shows an invitation email and I thought that's everything I'm supposed to know. Thanks to the answer by E_net4, I learned that there was also an FAQ...
    – honk
    Commented Oct 2, 2018 at 14:03
  • Ahh, I see now. They also asked not to make the link public... Commented Oct 2, 2018 at 23:44
  • @CamiloTerevinto: Thank you for that information! Unfortunately, I also didn't know that. If I had, I wouldn't have touched the comment evaluator. A note like: "Please don't participate if you didn't get an invitation" at the top of the comment evaluator site would have helped me...
    – honk
    Commented Oct 3, 2018 at 7:44

1 Answer 1

11

The invitation made by e-mail to use the comment evaluator includes a FAQ. One of its entries reads:

What if I see the same thread more than once while rating?

This is expected! We can use these ratings to measure how consistent people are.

This answers the question here, although one might indeed question the significant frequency of repeated threads. This might also suggest that it's not worth spending more than the recommended hour unless you're fine with rating the same comments over and over again.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .