You say that you noticed a lot of this in the last few weeks. So I'd assume you're saying it's worse than it was before. I personally do remember myself having issues with these highly scored bounty audit questions in the past, but have since learned how to spot them pretty well to avoid failing an audit. I've also noticed others complaining about them on meta before recently. Definitely not a new thing.
So I think if you're saying that it's worse than before, then we'd need to gather some data to see if that's the case. Or maybe we don't need to since it's trivial since it's always been a problem just never fixed.
The way you wrote your OP makes it sound like there should be a serial voting type of script that could detect if some users are just going through the featured tab questions and upvoting them all for no apparent reason other than that they're featured. Maybe an employee could work some magic to see if this conspiracy theory scenario could possibly be worsening the issue. But even if some users are doing this for whatever their reasons, there isn't no way to really punish them since they can use their votes how they want I guess.
Or maybe if this is a lot worse than before, maybe there is sock puppet serial voting rings going on that involves bounties in their greedy schemes. Run serial voting / collusion scripts on the weird cases questions/answers you come across.
I don't see how a new design of how they are featured would decrease these unwanted seemingly incorrect upvotes. Unless, you remove the featured tab all together, get rid of the ability to search for them with like hasnotice, or through SEDE. So that the only way you know if it has a bounty currently is if you view the question itself. Yea, that would work, but I'd definitely be less confident in placing a bounty if that were the case, since it would likely get very low views / less potential good answerers seeing the question. Because there would be no way to inform the users that want to answer bounty questions to get more bang for their time without them scanning all questions in their tags for bounties, which would be ridiculous and hard on the server.
You could remove the previously bountied questions from the audits, but I don't know the algorithm that's used, so I just hope there is enough questions with like +5 or whatever score and other necessary criteria to be picked for these audit cases in place of the bountied ones. Because if this would somehow make it for reviewers to be given less audits, then I'd say no. But maybe just don't use previously bountied questions from users that are less than 2 months old or have a score of less than 250 or something. That would likely produce a similar result as removing the bountied question audits all together.