Interesting read. Detecting unfriendliness is a form of sentiment analysis, and my employer has been doing something similar for the last 20 years so I recognize quite a few of the challenges.
We fully agree with the notion that you can't just take humans out of the loop. But AI is very good at prioritizing work for humans. You can train neural networks to also produce confidence scores. These shouldn't be interpreted as "percentages correct", but only to sort the flags for the human moderators to judge. Start with the ones that the network classifies as unfriendly with high confidence, and work down the list.
As you gain experience with the AI, you may find that the highest confidence scores are certain hits, so you can in fact deprioritize those. Instead, pending human verification you label them as unfriendly.
One area we're actively looking into is the ranking of sentiment. Your current classifier is a binary classifier, because you train it on two classes. Something is either friendly or unfriendly. But your human baseline can be more nuanced than that, and your AI can learn that.
As you might notnotice, unlike Jason Punyon and Dave Robinson, I have no hesitation using the term AI. To me that is a broad category. I agree that this is also Machine Learning, and also Pattern Recognition. Those are overlapping subfields of AI.