I would be okay with displaying a confirmation message the first time, like you suggested, except that you can easily undo an accidental vote. The downvote arrow glows orange, and a simple click on it reverses the damage.
Moreover, you only earn the privilege to down vote after you've had the privilege to up vote for some period of time. The idea is, therefore, that you already understand how the voting system works, which would make a confirmation unnecessary and annoying.
What happened to you doesn't seem like confusion in understanding how the voting system works, it sounds like a fat-fingered mistake. Which is no big deal, we all make them sometimes. The normal case would have been that you noticed it relatively quickly after casting the erroneous vote, in which case the system would have let you easily reverse it.
However, since you didn't notice it until several hours later, you discovered that the system has locked your vote in. It is necessary to do this to prevent gaming the system through voting. I know that you're an above-the-board person who would never do this, but there are some users that will downvote every competing answer upon answering a question, hoping to drive their own answer to the top of the sort order and thus accumulate even more upvotes from future visitors. Since downvotes on answers cost reputation, they'll come back later once their own answer is decidedly in the lead and remove their votes. Locking in the votes after a certain period of time allows us to prevent this type of fraud, while still allowing honest mistakes to be undone.
There is, however, a workaround, which Martijn has already pointed out: you can edit the post that you've voted on, which will unlock your vote so that it can be undone or converted to a different type. The idea here is that when a post is edited, its content changes, and therefore your opinion of it might change. But it also works as an escape hatch to undo a vote that has been locked in. Doing this regularly would naturally still be considered an abuse of the system. But it serves as a workaround for the rare cases, like this one. Plus, we do ask something of you in return for the privilege to change your vote—we ask that you take the time to improve another's contribution to the site. If you're doing that each time you post a competing answer, well…more power to you.