Votes (up or down) are not about the user; they are about the post, the material itself. They mark the material as being good or bad in some person's estimation. It's a "wisdom of the crowd" measure -- over time, and with lots of eyeballs, the chaff sinks and the hand-formed, hearth-baked bread made from locally-grown wheat is elevated. In that sense, one up- or downvote is just a single grain in the bucket. This applies as much to questions as to answers; a clear, findable question about an interesting problem that another person might encounter is the first step to getting an expert solution.
Granted that this doesn't remove the desirability of explanation; if the post can be improved, it's nice to know that as an author, and improving that post would make the site better. Sometimes a post can't be improved, though, or the voter just doesn't want to take the time to explain how and why.
A vote, representing an expert's judgement of the material, is still valuable as an indicator to future readers, even without explanation.
I should point out that this is one of the most-discussed Meta subjects in the history of Stack Exchange. There's a whole lot of discussion to be perused at Encouraging people to explain downvotes and the many linked questions.
On a connected note, I've found that, the majority of the time that I downvote and comment, I either get into an argument or find a string of not-so-mysterious downvotes on my own posts later that day. I'd really like the freedom to explain to someone why and how I think their post should be improved without them getting upset, but it just doesn't happen in my experience. I sometimes get attacked because I comment where someone else has downvoted. I don't know how to fix this problem, but requiring a comment isn't going to help (although an anonymous comment might work); it's going to make for fewer downvotes overall. That wouldn't be good.