There are a few different reputation-affecting actions which can be reversed when a user is deleted.
Reputation events listed as user was removed
- User who upvoted your answer is removed (-10)
- User who upvoted your question is removed (-5)
- User who was the final approver of a suggested edit is removed (-2) 1,2,3
- User who downvoted your question/answer is removed (+2)
Looking at this, the most likely answer is that your guess was correct:
- One question upvote removed (-5) and two question downvotes removed (+4)
As Martijn Pieters points out, another possibility is due to the daily reputation cap.
When you are reputation capped, downvote an answer, then receive another upvote on one of your posts, you get a +1 reputation change. If the voter was subsequently deleted, you see a -1 event.
Finally, if the deleted user's posts were also removed, there are a few other possibilities, but these ones seem fairly unlikely.
(The list of reputation events doesn't specify whether these would be listed as user was removed)
- An answer you accepted was deleted (-2)
- An answer you downvoted was deleted (+1)