I think there is no hard or fast rule that can be established. When answering questions for a library I maintain, I usually discern the following situations:
Obvious (to me as maintainer) missing feature of the library.
Ask the OP to file an improvement request (or do it myself), and suggest a workaround if one exists.
Depending on the scope this should either be a comment (ask to file an improvement request) or an answer (ask to file an improvement request + provide workaround). Example:
New bug where the reason is clear to me as the maintainer.
Similar to 1, I will either ask the OP to create a ticket, or create it myself. If a workaround is possible, provide one. Example:
Known bug or limitation.
Inform the OP of the bug or feature request (link to tracker or FAQ), and provide an explanation and a workaround if available. Example:
Not so obvious problems or bugs.
If I don't understand a question, or the exchange in the comments gets too long (or if I suspect it is going to take too long to identify the problem), I will usually ask the OP to post their question on our mailing list or bug tracker (depending on whether I think something is a question on usage or a bug in the library I prefer one over the other). If possible, I may post a summarizing answer later. Example:
Point 4 is the hardest, and it is not easy to say when the point comes when you need to back off and ask people to go to a more suitable venue (mailing list, bug tracker).
I'd say that after asking for clarification 3-4 times and not getting closer to the root, or if you provided an answer to what you thought the problem was and then the response by the OP indicates the problem is nowhere close to what you answered, that might be a good time to take it elsewhere. And as a lost resort: if Stack Overflow suggests that you should move your comments to chat, that should be considered a stern reminder that you have used too many comments to troubleshoot on Stack Overflow.
In other cases, if while reading the question you already get the feeling it is going to take some digging, then immediately suggest to take it elsewhere. If possible, when you have come to a solution, post a summarizing answer.
This doesn't always work though, as sometimes people are not willing to take it elsewhere. In that case, consider just voting something to close as Too broad.