The inability to remove a down-vote from an answer after a short time has elapsed seems like an unnecessary restriction. One can work around it by editing the answer first, but it seems that this shouldn't be necessary, and what is the point of a restriction such as this if you can work around it anyway?
What prompted me to ask was this answer where someone down-voted and left a comment, but then realised that they were mistaken in their objection to the answer, but were unable to remove their down-vote.