I have not observed this trend. But even if I had, I would say it is simply a fundamental failing of a system that allows anyone with a token amount of reputation to cast a vote—in other words, a democracy.
It works the other way, too. Plenty of people see an answer that looks superficially good and upvote it. No matter that it is outright wrong, contains obsolete information, or recommends a cure that is worse than the disease ("just add DoEvents
inside of your loop!"). Plenty of these voters don't have the requisite knowledge to judge what a good, accurate, and complete answer is. At best, they think to themselves, "sounds good!", and cast an upvote. At worst, they vote based on the user who wrote the post.
So it kind of makes sense that someone who has little background on which to judge an answer's merits would be easily swayed by a strongly-worded comment. I don't know if it is fair to call this "reactionary vote casting". In some sense, all vote casting is reactionary: you are reacting to the accuracy and usefulness of the post.
The real problem is wrong vote casting. But we don't really have such a notion here—and can't. It is not for one user to judge the veracity of another's vote, so long as they follow the basic ground rules (no sock puppetry, no serial voting, etc.). One person's vote is—on its face—just as valuable as another person's vote. We simply have to hope that an appropriate score will eventually shake out in the end. As such, don't be shy with casting your own votes when you have a conviction that something is useful/correct or useless/wrong!
As for your choice to delete the answer because you didn't want to lose rep, I would strongly advise you to rethink this strategy. I understand that it sucks to lose rep, but they're just imaginary Internet points. If you are convinced that your answer is accurate and useful, then you should leave it visible. Trust that you will be vindicated in the end by future upvotes. An upvote is worth +10, but a downvote is only worth −2, so the odds are on your side. Really, the only time you should delete an answer that is attracting downvotes is because you have seen the error of your ways and decided either that you don't want to fix the answer or see that someone else has already posted a better answer.