I noted it today, I don't know if it was already this way, but my last reputation was 5 downvotes and 2 upvotes in one question I asked resulting with no changes in the score.
I think it should show a mini-box with a number zero instead of hiding the mini-box. It just seems weird to me :/