I don't think that collapsed comments are a waste of time. I think they are useful because I often go past a series of a zillion comments. Especially on high traffic posts. That said, I think that there could be room for improvement.
Your first two points ring home and I do both those things quite often. If I see "35 hidden comments" I will often not look at the set of current ones regardless of their upvotes and click on show all. This is because I have been burned so many times just reading the upvoted comments and missing the context for half of what was written.
Comments do not have downvotes. Thanks, captain obvious. I know, but to me I think is a major concern because if a controversial comment is placed there is no way to tell. It just looks like it was wildly popular. +20, -200? Just looks like +20, and all of a sudden it is featured in the list of "highlighted" comments.
The subset of comments shown is supposed to give an indication of the overall conversation in comments by showing the "popular" comments. However, as there is no context for many of these comments and because the post most often becomes expanded in comments (and not in the post unfortunately) reading only the popular set is not very useful.
Showing just the first set of comments regardless of votes before the cutoff would show an overall indication of what the comment conversation was going to include. I am not sure what the statistics are, but just like quick answers, quick comments often also generate the most upvotes. Mostly because a position is stated which everyone was thinking and instead of restating it they just upvote the comment.
tl;dr; I would assert that showing the first set will still include several highly voted comments from the subset that is shown now, and also that it will give a better indication of the overall comment discussion because it will contain no gaps.