While you are reviewing First answers, Late answers, and Low quality answers, you are performing a civic duty. In essence, it is your duty to downvote answers that are not high quality enough as well as selecting the appropriate review action.
In comparison, when you are simply browsing the front page for questions and find an answer you don't think is high quality, you can choose to downvote it at your own discretion. Presumably, downvoting an answer subtracts 1-reputation point to deter gaming the system, spamming downvotes blindly, or target harassing a single individual.
I don't think that while you are in the review queues the 1-reputation penalty should apply to downvoting answers.