First off: I didn't get an invitation for the Comment Evaluator 5000™; I learned about it through related questions here on Meta. I tried it out anyway, so please let me know if I'm not supposed to do so.
Up to now, I spend about 30 minutes rating those comment threads shown to me. I just noticed that a thread of four comments was shown to me for the second time. Why are those comments presented to me again? Is this a bug or is this by intention to make sure that users provide consistent results?