It's hard to think of a better cause to downvote a question than it being off-topic. Other good causes include a question being unclear and not showing any research effort (i.e., the answer is immediately obvious from glancing at the documentation).
Not coincidentally, these are the reasons listed on the tooltip for the downvote arrow:
This question does not show any research effort; it is unclear or not useful
Asking an off-topic question is pretty clear evidence that you did not expend effort to research our guidelines for an acceptable question, and off-topic questions are absolutely not useful to anyone.
Pragmatically, downvotes on off-topic questions are necessary, and an integral part of the system, because they provide a signal that the question is not one that experts should pay attention to. It is a waste of their time and our limited resources if people have to look at off-topic questions. By the first few viewers downvoting them, it saves everyone else time, which they can then use to focus on questions that deserve an answer.
A common misconception among new users is that downvotes are somehow punitive. They really aren't meant to be. This site focuses on content, not users. Voting is a content rating system; nothing more, nothing less. Bad content should be downvoted; good content should be upvoted. Content that makes you say "meh" probably shouldn't be voted on. However, a fundamental rule of our voting system is that users are allowed to vote for any reason(s) they choose, as long as they are not casting votes fraudulently (i.e., by operating sockpuppets and/or by targeting their votes at a particular user.)
In any case, there is some good news: off-topic questions are generally deleted after being closed, and when a post is deleted, you will get the reputation you lost from downvotes returned. (The only catch is, deleted posts' scores still count towards the automatic quality bans, but as long as you don't make a habit of asking off-topic questions, this will not be a problem for you.)
As for
people got into arrogant arguments with me instead of explaining in two words that the question was off-topic.
Yes, that happens sometimes. People can be jerks. Other times, people are trying to be helpful, but phrase things poorly so that they come across as arrogant. Either way, we don't condone such behavior, and you should flag these comments for moderator attention so they can be removed. More generally, if someone is violating our "Be Nice" policy, then please let a moderator know.