Yes, I think the best thing is to reply in a helpful manner in 2 specific situations. 

1)The person asking is clearly new to Stackoverflow  OR  2)He/she wants to know **why** they were downvoted and **NOT who** downvoted them.

I usually reply with the  appropriate link from [i-downvoted-because][1] and the Stack Overflow guide on how to ask a good question: *[How to ask a good question][2]*

For example, if the OP made no attempt to solve the problem first, I would put that in my reply and include this link: http://idownvotedbecau.se/noattempt/

I think not letting someone know why they were downvoted only creates resentment towards the community.

  [1]: http://idownvotedbecau.se/ "I downvoted because"
  [2]: https://stackoverflow.com/help/how-to-ask "How to ask a good question"