I've evaluated a number of comments and found that overwhelmingly they are just fine. That's good to know and it's even better if we can verify that several people agree with that evaluation. (It's not obvious, but comments are potentially evaluated by other respondents.) I maintain that, on average, comments on Stack Overflow are among the best on the internet and it's nice to have data to back up that assertion.
However, I've also run across comments that are definitely not fine. Some seem likely to shake the confidence of the author (likely without having the intended effect of discouraging participation). And others are definitely not fine. I'm not sure how long to suggest looking for not-so-productive comments, but in my experience, they tend to be clustered on specific posts. So it's possible to go through several pages of the evaluator without seeing anything wrong only to run across a post with a whole mess of less-than-productive comments. (My "favorite" are the ones from high-reputation users complaining about upvotes.)
Whether or not the process of clicking "Fine" is worth your time depends entirely on you, of course. I've done this sort of exercise in the past and it's hard to power through a bunch of dull comments. And it turns out many comments on the site are just sorta uninteresting taken out of context. The good news is that they tend to be pretty quick to evaluate in this experiment.
If we are able to successfully build a model that can classify comments, it seems worthwhile to skip posts with nothing by innocuous comments. But building that model requires getting feedback on a healthy sample of evaluations. So it's kinda a chicken-and-egg problem.