Regardless of whether you downvoted, or whether you feel like commenting, do flag the comment for removal. No longer needed flags are usually honoured in these cases, as well as on plain undirected "why the downvote?". If the comment is deemed as more provocative towards the user(s) who downvoted ("some trolls downvoted my question for no reason", "whoever downvoted should be banned from this site"), you are in your right to escalate to the Unfriendly or Unkind flag. Yes, they might be upset and in need of venting for what may be felt as an unfair demotion of their content, but such an exhibition of impatience is not only noise to visitors, but also makes potential contributors to that question feel uncomfortable, and will drive them away.
There is indeed a chance that the poster of the subsequent comment becomes a target, and what happens next is not always predictable. Best case: everyone acts in good faith, the OP thanks the user and improves the question/answer with that in mind (if that is possible at all; downvotes are also frequent on unsalvageable questions). Bad case: the OP starts an endless argument with you on how the question is appropriate, leading to more noise and frustration. Worst case: you receive an e-mail accusing you of bashing people that are still learning and of not being a real programmer, even though your comment was constructive.
I have experienced each of these outcomes so far, and yet the consequences of facing a negative reaction far outweigh the ones from good reactions. The life of a Stack Overflow curator that leaves replies to "why the downvote?" will generally have to show some resilience, otherwise they might just "break" after the eventual Nth bad case.
In conclusion, replying to such comments is a bit like playing the roulette, where the stakes are based on the OP's question/answer and mood. It's such a strange game.
The only move that always leaves you unharmed is not to play. Whether you're willing to continue placing bets is up to you. No one should feel obligated to comment, after all.