If the question explicitly states that answers should not involve that library, or libraries in general, then the answer is not a valid answer and should be downvoted.
If the answer does not explicitly state that the answer uses a 3rd party library (or it is otherwise not immediately obvious that this is the case), then the answer is incomplete and should either be edited or downvoted.
Otherwise... it's dealer's choice, really. Some would say that solving a problem by employing a widely available and popular library is a perfectly valid answer. Others might say that if the asker was using a particular library, they'd have mentioned it, so it's rude to suggest that they use something they may not wish to.
Personally, unless the library was really obscure, outdated, requires loads of dependencies, or unquestionably terrible, I wouldn't downvote for it. Even if it isn't helpful to the OP, the answer can still help someone else with that problem who does use the library.