I have observed it many times. Reputation points go down by 3 or 4 points without leaving any trace under reputations changes.
Why does it happen? I have faced it many times.
Recently it happened again. Yesterday, my reputation points went down by 4. It was 2203 and now today it is 2199.