Downvoters should not have to explain themselves.
They really shouldn't, and this topic has been beaten to death.
OP's should be able to ask for an explanation of downvotes
Why should a poster not be able to comment "Can the downvoter please clarify their position?" or similar? If they are not able to see the issue with their post, then they are only asking for the exact problem which led to the downvote so they can either fix it (or often delete it once they understand). Further, it is possible that the explanation doesn't even come from the downvoter. Either way, none of this places the actual onus on the downvoter, because they in no way are required to respond to these comments nor does anyone even know they were the one who posted the downvote.
The downvoter should not have the "authority" to prevent user comments
In my opinion, that one user should have "authority" over the subject matter of another user's comments solely based on the fact they downvoted their post is ludicrous. While certain users may respond with a hostile comment asking for an explanation, that just makes them easier to ignore. In no way does that mean that every user should be prevented from asking for explanations.