I am very much torn here.
On one hand I am behind downvoting answers which are on "really" poor questions, because the question simply does not deserve an answer. It shows questioners that they will get an answer regardless of the quality of their question.
I have even seen questioners on really really bad questions tell me to stfu, and explain that I am among only a few people who think such questions are not welcomed.
They don' engage in comments, because they know someone will come along with an answer for them.
Then they start with comments, under the answer asking for more and more from the poor answerer, or telling them why it doesn't work and to update their answer.
Or worse, I have seen plenty "If you can show code which also does XYZ I will accept your answer".
This makes me want to downvote the question a million times!!
That said
I also don't think it's fair to downvote answers based on the question. The answer is usually ok, and away from a poor question well deserves a few upvotes.
I still only vote based on single merit, that is vote on the question based on the question, and vote on the answer based on the answer. I think that is the only fair way.
I tend not to upvote an answer which is to a bad question, regardless of the answer quality, simply because they have answered a bad question.
I often check back later, and for the most part, these questions are closed off later on.
All that said, voting is very much a personal thing, so vote however you want to - everyone else does!
If you genuinely care for others and the site, then just try to be wise and fair in your voting.