Bounties do not work that way : they are proposed by somebody for a limited time, but since that moment the bounty value is deduced from the promoter (I call him/her that way because it can be question asker or anyone else) rep and this cannot be undone.
Next, if promoter finds a question deserving the bounty until expiration time, he can manually award it. In that case the answerer receives the full bounty value. Here again nothing can be undone after manually awarding.
If the bounty has not been manually awarded at expiration time, the best answer with at least +2 will automatically receive half value of the bonus
References : How does the bounty system work?
Now to answer your question, it is clear that you cannot give it back. If it makes you feel bad, the best you can do is to try to improve your answer. For the rest, you tried to help, your answer is clearly more than a comment, and OP decided your answer was the best. If another user later posts another answer better than yours, you simply should :
- upvote it if it deserves it
- edit your own answer to reference it, saying that it was posted after bonus award but you thing it is better than yours
That way, even if the quality of that answer would not deserve the bonus per se, you will have done your best to improve the site. And if it is not enough for you to feel good, just do your best to improve the site, by flagging incorrect posts, duplicates, and also try to talk on meta. All that will indeed deserve the 50 points rep !