SO says: you should not downvote
It's not about that little reputation point (although it might add up for lower rep users who are very serious about voting a lot).
But in principle it still is a penalty! The reputation system implies that good deeds give you badges and rep, while bad deeds make you pay. The underlying signal of that penalty point, is that SO doesn't want you to downvote.
Possible solution: rep as currency rather than performance counter
Of course (?) rep should be seen more as a currency than just as a reward counter, but there is so little you can spend it on! Maybe there should be more ways of spending this currency. Let people use their rep to buy hats, colored mark-up, other gimmicks, or maybe even more useful features. Instead of automatically handing out more site functions when people gain rep, let them buy it (if they want). That way you can get more into the habit of spending your coin, or just letting it sit on the bank, and it will feel more like just another spending if you downvote. You could even put a cost to upvoting too.
Newcomers say: you should not downvote
Another thing is being welcoming to new users. Being downvoted is a big fear for new users, often expressed in the questions in the form of "please don't downvote me" (sometimes appended with a request for feedback).
This tells me that those users do take it personally, or at least too seriously, and I get the impression that downvotes are part of what makes SO unwelcoming to newcomers. As you say, we're downvoting content, not people, but still there is this perception. That perception needs to be tackled somehow, otherwise there will be an eternal ping-pong game between downvoting poor content and being welcoming, even though they shouldn't be mutually exclusive.
Possible solution: delayed effect of downvotes
Sometimes I see questions that don't even seem that bad. Even though they might be hard to answer you can at least tell that someone tried to ask a good question, but nevertheless such questions easily got 2 or 3 downvotes in very little time. New users who don't have experience with the speed of SO will come back after a couple of hours or a day, only to find their question is at -4 and is pretty much invisible to anyone.
I don't know exactly how this would work, but maybe there could be, sort of, preliminary downvotes. Like "Downvote this question if it won't improve in 2 days".
Automatic improvement detection is probably hard to build, but maybe a reminder will do. I'd love to be able to bookmark questions and get some reminder about them on my next visit.
That way, I can suggest improvements or ask for clarification and postpone my downvote. If I get notified the next day and the question is improved, it's a win for every party. If it isn't, I can still cast a downvote. At least the asker has been given the chance to work on their question.
Sometimes I've used the 'favorite' feature for that, but it feels wrong. I've also bookmarked specific questions in my browser, but that feels cumbersome (especially since I'm working on different machines).