There's no way to stop a bounty from being automatically awarded if an answer meets the criteria for it to happen.
It sucks, but that's the way it is.
As indicated by @animuson, there's a feature request to make this possible: Explicit "do not award bounty" button