It already works that way.
Trusted users are given delete votes, and they are allowed to delete low-quality or non-answers when their score is −1 or lower.
Since the same users can downvote an answer, this effectively means that trusted users have the ability to cast delete votes on answers with a score of 0 or lower. This just functions as a sanity check, to prevent an organized cabal of users from deleting valuable content (measured by the community) merely because they dislike it for some reason. You'd need to get together a large enough cabal for their downvotes to cancel out upvotes of other users.
The answer doesn't even need to be flagged as "very low quality", but if it is, and you come across it in the review queue, one of your options (a big blue button at the top) is "Delete". This casts a delete vote on the post.
Three delete votes and the answer is gone.
Users with delete-vote privileges are given 5 delete votes per day, plus 1 vote for every 1k reputation that they have over 10k, up to a maximum of 30 delete votes per day. (Reference: Deletion FAQ on the Uber Meta)
If someone else has already flagged an answer as Very Low Quality or Not An Answer, you have enough reputation to work the VLQ queue, and you haven't already reviewed 20 very low quality posts that day, can you "directly" vote to delete the answer (rather than flagging it again, which doesn't do any good)?
Correct. Currently, flagging it again does not do any good. But even for users without the "trusted user" privilege, "Recommend Deletion" is one of the options when reviewing posts on the "Low Quality Posts" queue, and six delete recommendations will cause the post to be deleted immediately.
So, even for non-trusted users (i.e., anyone with review privileges), we already have the feature that you propose.
The only stumbling block is that the post needs to have already been flagged as "very low quality" or "not an answer", but that's...not a very high stumbling block. You shouldn't really be deleting posts that don't qualify for these flags (except, I believe, in the rare circumstance that you're an expert on the applicable technology and can stake your reputation on the fact that the answer is dangerously wrong or adds absolutely nothing to the discussion, and the way that we measure this on Stack Overflow is reputation, which is why 20k == trusted user == delete votes).
Oh, I guess another thing to note about this is that it still requires the answer to have a negative score. Six users recommending deletion won't cause the direct deletion of a positively-scored post. Instead, a moderator flag will be automatically raised, alerting us to the fact that a number of users recommended deletion on a positively-scored answer. We'll look at it and decide whether we agree that the answer should be deleted. If we do, we'll delete it. If not, we'll decline the auto-flag and move on. None of the reviewers are ever the wiser; they reached a consensus, so flaggers' VLQ/NAA flags are automatically validated.
An alternative would be to have VLQ flags work like spam and rude/abusive flags if you have enough reputation: if a post acquires enough of them, it's just deleted. (You'd have to figure out the relationship between flags and the queue, then, though, so it may be simpler to just give users with enough reputation to just vote to delete stuff that's already been flagged).
See above—we basically already have that, it's just implemented a bit differently.
Difference 1 is that instead of being implemented by reflagging the post as VLQ, you click the "Recommend Deletion" button. Functionally, it works the same way.
(Granted, I could follow an argument for why you should just raise another VLQ flag on it, but consider that feeling an answer is "very low quality" is not necessarily the same thing as feeling that an answer should be deleted. Except that...that's how the flag is generally interpreted, which is an entirely different point of confusion that I'll leave for another time.)
Difference 2 is that you don't have the auto-downvotes that you have with spam/abusive flags. Of course, this has been proposed, and still makes a great deal of sense to me. Note that the VLQ flag does carry one automatic downvote from Community. If the post is edited, this automatic downvote is removed, without penalty.
Does that answer your question, or have I completely misunderstood what you're asking/proposing?
(Bring on the "not an answer" flags because there's a rhetorical question in here!)