A lot of user removals are due to sockpuppet voting. You wouldn't believe how many fake/fraudulent accounts we remove per week.
So the reputation gained from those account votes must be removed, to correct the voting. If it happens that those accounts also voted for you (sometimes a lame strategy to try to fool the moderators -note: it doesn't work-), then the votes are invalidated as well.
Other case is user requesting account deletion. It also happens, more rarely. In the end, the user has been destroyed, and the rep gained from the votes is lost.
Note that when deleting a (non-fraudulent) user with a very high reputation, the community managers can decide to keep reputation from votes of this user, because the vote invalidation would mean that a lot of people would lose a lot of points (statistically, high-reputation users vote a lot all along the numerous years of the account existence)
Because high-reputation users have usually cast a great many votes, removing all of them could be that much more disruptive to other users. In such cases, the staff use a special deletion that preserves the votes, resulting in no reputation change for those who had been voted on by that user.
Doing that for every non-fraudulent account would be a lot of work I suppose. Well beyond the power of the non-SE employees moderators like me.