One has to sign-in to participate to Stack Overflow, and then he or she has a nice profile page, with an About description, and perhaps a pretty picture. Also cool badges. And, clearly the most important part, a reputation. Those points, while imaginary, denote effort in writing questions and answers and are real, because they mean time, concentration and passion.
So it is undeniable that Stack Overflow has a human meaning for posters.
As many in the comments have well expressed, it is annoying, depressing or demotivating when one spends time to write a question or an answer and gets welcomed by a cold reply, that superficially could mean "I don't like your post, and possibly others will not as well", but on the poster side, it could mean "My effort, time and passion just don't have value for the people I am seeing as experts and to whom I am asking for help", or in other words "I am not valuable as a member of the largest Internet programmers community". (This applies mostly to downvotes without declared reasons, i.e., anonymous.)
Since the OP question is mainly psychological, consider this: a small child has no value for their own family. They are just a "sink", they "drain" resources from their parents, sisters and brothers. However, in any sane family, they are not seen as a weight. Now, the child goes to their parent and says "look at what I have built", or "I have a serious problem", and imagine the parent answering: "it sucks" or "you are just a child".
While it is true, that Stack Overflow is for adults, psychologically we are also children. If not, we would never joke with co-workers, never get frustrated, bored, etc... always knowing the right answers and questions, and when not, admitting it for the good of others.
So, if (at least one) people (sometimes!) feel hurt by the system of Stack Overflow, there may very well be a reason. The answer should not be "take it or leave it", if it is true that the system is made for all. Some people are more sensible than others, some more insecure, some people may come to Stack Overflow because they have an urgent problem or a problem they have spent days solving it without success, some may just seek a word of advice from good programmers from a very well renowned Internet website...
Upvotes may flatter, no-votes may leave things are they are, and downvotes may give some minor or not so minor pain.
So I do not understand why they are there. Algorithmically, they may give better results, but humanly they are not optimal. But, is Stack Overflow for machines or for humans?
In Stack Overflow, other than technical research there are underlying human values and interactions.
Anyone seeing their values betrayed gets annoyed or depressed by the interaction with other people through the website.
I am not saying to give sugar for everything one does, but I am saying that algorithms and rules may kill the human relations if one is too rigid about them, and a website or software is rigid by nature. So rules should be minimal and acceptable by all, under the majority of circumstances.
What harm would do if downvotes were prohibited? Just upvote interesting posts and ignore the ones you do not like. Are out there really so few readers of Stack Overflow so that just upvotes are not enough to push up interesting questions and answers?
About "giving a signal to posters", votes alone are not a satisfying mean to "give a signal", as many say in the comments. Really, does a voter care about the person on the other side? They have not even seen the other, do not even know if they hate or love what the other loves or hates, nor do they know to what degree the questioner has need for an answer to their -- even vague but perhaps important -- problem. Truth is that one votes up or down based on personal tastes, ideas and beliefs. The voter cannot really care about the other, because the other is just a name and a pure product of thought on Internet forums. Even though it is true that the voter cares, when a child learns from an adult, they learn because they trust the adult, not because the majority of adult people give "correct" downvotes and upvotes. A child is not an AI that learns with punishments and rewards, they have feelings. So we are too, and a comment of 100 characters would be exponentially better than "a signal". As well, a dialog, where the poster and the voter talk to each other, would be even better.
Back to the initial observation, if Stack Overflow refrained from having a profile and a reputation for its users for giving them human (intangible) rewards and, just for example, limited itself on counting points on (anonymous) questions and answers, clearly the problem of being down-voted would not have been felt so much. One may say that a user profile motivates people to do better, but be aware that applying rules for one's own happiness without caring about the happiness of others, may be spelled as "gaming, manipulating, and using others", even if the end result is a "better product for humanity". Note, indeed, that if a great part of the population would be exterminated, much of the world problems would be resolved: overpopulation, poverty, pollution, ... A game is cool if everyone has fun playing in it. It is very important to respect all others.
Finally, note that readers volunteering to review posts do not get paid, but also authors of questions and answers are not paid.
To me, it is a minor satisfaction that downvoters (just of answers) lose reputation. What is the gain? We both lost reputation.
So to answer the OP question, the problem rises because of a confusion of the identity of Stack Overflow. In part, it is a social Q&A website where you always find answers to your questions. In part, it is a game, with strict rules, so when you have a question or want to share an answer, you must be very careful. If the second part is more prevalent, wouldn't it be better to build a moderated wiki like PlanetMath? Or, just have a higher barrier to entry to post questions and answers, so that one is not misguided: when participating, one is working for the site, not asking questions or answering for himself/herself, i.e., the platform is not neutral.
downvote !== calling someone stupid