From exhaustive testing and interviewing, we can confidently assert that tagging a question is a process that is very unintuitive to new users. Auto-completing helps people through the process, but taxonomy and nomenclature are the least of someone's worries when all they want to do is ask a question.
Additionally, we ask for tags as a final step in the asking process. This means, should someone not find an available tag that matches their intent, loss aversion alone can compel them to just find something that fits and see what happens. Abandoning efforts put into a title, body, any formatting done, etc would not seem optimal, to say the least.
With that said, it depends. We've had some success using much more descriptive messaging when blacklisting a tag, we've also seen success when warning using a tag, so there's at least some evidence that people are willing to abandon a question if the system makes it clear to them that the experience is not going to be great if they proceed. It's not always the case, though.
And then you have topics where a narrow subset is on-topic (anyone remember lots of discussions about Sharepoint?), questions with historical locks where people don't notice the 'locked' part (sometimes, just because all they see are the titles in search results) ... it's really hard to narrow the brunt of the blame on any single element there, tag included, and it's difficult to show just-in-time help for so many kinds of edge cases.
And then you've got folks that put tags in titles if they can't get them into the tag section, and those go a few hours without editing, get indexed, and 20 more come in. It's honestly a dice roll and the cost of 'doing business' with so many uncontrollable factors.
So the direct answer to your question: We have evidence that just-in-time warning can overcome the loss aversion that would otherwise prevent someone from abandoning a question, despite the existence (or lack thereof) a corresponding tag. However, it's just too hit-and-miss, with too many edge cases to build a solution around.
Does getting rid of bad tags help curtail off-topic questions? That's hard to answer because you can't document a negative. We know as much as we do through events we log when something is abandoned due to help being shown, which is expensive. Anecdotally? Yes. It does help, and certainly won't hurt.
The current strategy of inserting just-in-time help in known problematic tags while keeping them around just for the small subset of questions that are on topic, or just as a warning beacon is better than just accepting that it's just the cost of doing business.
More elegant solutions (like calculating the average score of questions in all applied tags and warning if foo
looks like it's universally crapped on) is too much to do on-the-fly right now. But, eventually, the just-in-time help needs decoupled from the human dependency (perhaps offloaded almost entirely to mods and trusted users, but we'd have to overhaul how we implement the blacklist entirely to do that).
Sorry for the rambling response on this, it's something that I've been poking and peeking into more and more over the last few months as I was examining situations where questions were just kinda doomed no matter what happened to them.