You should be more careful on how you vote when going through the FP queue.
(Well, voting with care should be a given anywhere, but even more so in that queue).
You are reviewing posts of users who are new to the platform. Your actions there are supposed to help and guide these users into becoming productive users of the community.
If you are voting on "hunches" instead of doing your due diligence, you will be sending the wrong signals to these new users. You'll reward poor posts. You'll (encouraging bad habits), or you'll maketreat them unfairly and make feel unwelcome by treating them unfairly by down-voting posts that didn't deserve down-voting (as could been said in the case of the audit you failed).
Personally, I do not see anything wrong in the audit:
If you weren't sure, you should have skipped or researched properly before voting.
If you were sure and you were wrong, no biggie. We all make mistakes. Failing this particular audit will hopefully push you to be even more careful in the future, which is the whole point of audits.
If you were sure of your down-vote and continue to believe the answer is wrong (or poor, or just generally not useful), just down-vote the post outside of the queue. It will no longer be an audit, so you'd be fixing this issue for future reviewers (more below).
And in any case, reading this carefully will help you in continue reviewing successfully.
In the general case: if you disagree that a post should be an audit (and you think that it truly deserves a down-vote) down-voting it outside of the review queue will make it so it's no longer eligible to be an audit in the future.
(This is not valid for all types of audits, obviously; but I think it's true for any audit that's based on a non-deleted post).