Before I get into my proposed solution for bad review audits, I should state that bad audits are like plane crashes: they are very rare compared to the number of audits used, yet we think they're common because a big deal is made when they happen. I have looked over many, many audit cases since these were put into place, and only a tiny fraction of audits are bad cases.
Therefore, I don't believe the best solution would be to completely rework the way that audits are defined, but to instead handle the rare bad cases themselves. We really don't want humans hand-picking audits, since that won't scale (if done by moderators or staff), could be prone to abuse (if done by the community), and could lead to a whole new class of arguments on Meta if questionable cases were chosen for audits by people.
I still think the best way to deal with bad audit cases is what I describe in this answer. Give reviewers a means of contesting an audit via a button on the audit failure interface. This would then go into a queue for moderators (and the SE team) to review and give a yes / no as to whether an audit was proper or was a bad case. If marked as a bad case, it would be removed from circulation and the reviewer would have that failure be removed from their record (as well as any ban it triggered).
This would allow the community to identify the rare bad cases in a way that doesn't require a Meta protest, and it should be easy for moderators to tell at a glance if something was a good or bad audit case. Anything that takes any amount of thought for a moderator to tell if it was a good audit should be removed as an audit as a default position.
Someone abusing these disputes should be subject to a longer review ban, to avoid bad reviewers trying to annoy us by disputing all legitimate audits they get.
No disputing should be allowed for system-generated suggested edit audits. These are all gibberish, and if you approve them, you have no argument for doing so.
If such a system was implemented, it would eliminate almost all Meta audit protests overnight, give good reviewers a means of identifying and countering bad audits, all while making sure audit generation scaled with the site. It may even give SE information about audit generation techniques with high percentages of failure, and allow them to tweak audit algorithms based on that.