When you mark an answer as accepted, the owner of that answer gets +15 reputation points. Those are indeed taken away if you unaccept their answer. If you choose a different answer to accept, then that other user will get the +15.
You shouldn't really take that into account though if the answer has a problem. We don't want incorrect answers sticking at the top of the answer list.
Select the best answer that solves your problem, or the one you found the most helpful to you. That can change over time: maybe a new answer comes up with a better solution, or perhaps you've become more familiar with the issue and you realize that one of the other approaches was in fact more appropriate. Change your accepted answer in that case.
You lose reputation when: one of your formerly accepted answers loses accepted status: -15