TL;DR: Yes, it can.
Background:
The New Answers To Old questions page in the moderator tools is one of the best tools that can be used to assist moderation and has often been described as a place to find flaggable answers. There are nearly 9000 new answers to old questions per week.
A few users (myself included) used to devote a lot of our time aggressively patrolling the same answers and often used to end up out of flags even when there were still a lot of NAAs flowing. We faced a few issues:
- The NAAs used to pile up too quickly for users to notice and would be forgotten in the pages of history, as the tool listed only 500 posts (25 pages X 20 posts per page).
- The tool was not a real time feed of Late Answers.
- The tool was available only for users above 10k, there were a lot of users below 10k who were interested in curating this content.
After examining the Stack Exchange API, we, i.e, the SOBotics team, wrote a chatbot that could report these posts to us. That way we could solve all the issues we faced! Thus, Natty was born.
It worked well but after looking at some stats, we realized that most of the posts are quite obvious:
(tp indicates a true positive (i.e. a flaggable post), fp a false positive, which is a wrongly detected post, tn is a post that was not detected but was reported manually and ne indicates a badly formatted post in need of an edit)
A large number of posts (868) caught with a detection value of above 7.0 were all flaggable! Only 3 posts (which were all really badly formatted answers) were wrongly detected. (The 1 manually reported post was because of the API backing the bot off for a minute).
A few filters were performing really well compared to the rest:
So we decided that it would be better to automatically flag those instead of waiting for a human to revisit them and flag it. Everytime we flagged, we used to add an automatic comment from a list of canned comments. Ideally, this would have educated the OP that their post was NAA and hence they would delete the post. In reality, the 4-5 hour delay between posting and flagging meant that the OP was no longer around to look at our comments.
The initial plan was to flag the obvious NAAs and leave a canned comment on their post. Detecting which comment to use was simple - there were 3 categories of NAAs being detected. Link only posts, (which were flagged VLQs), "me too" answers by <50 rep and the NAAs by >50rep. Link-only posts were detected as they consisted only of a link. The reputation of the user, returned by the API, was used to determine the type of NAA. If the bot detects the post as non-English or a gibberish post, it doesn't leave any comment.
After talking to Shog9, a community manager, about this system, he advised us to:
- Improve the comments.
- Delete the comments if the flags are declined or remain unattended for an extended period of time.
After trying to improve the comments, Shog himself provided us with a set of comments.
What are we doing at the moment?
At the moment we are:
- Flagging obvious NAA.
- Adding a relevant comment on the post.
- Deleting the comment, if the post remains for a day.
What problem are we trying to solve?
All the posts that are flagged go directly to the Low Quality Posts Queue (a few after a delay of 15 minutes). Many of the posts which could have been autoflagged were lying around for 6 or more hours (until a human user checked those and flagged them). Hence autoflagging them would have cleaned those within an hour. Apart from that the NAAs enter the moderator queue after an hour and are prioritized based on when the post was flagged and not when the post was created. Adding a comment on the post as soon as they post such an answer would make the user aware of the rules. This would result in either the OP deleting the NAA as informed or tailoring their post into an answer for that question. All in all, it would help clear NAAs faster.
Why am I not running it under my own account instead of a bot account?
Initially it was decided that I would run it on my own account as I could retract bad flags if any and delete the wrong comments. But then something happened and I could no longer use my account. At the moment, the only disadvantage of this is that the bad flags cannot be retracted. There is a Room Owner only command to delete the bad comments.
Are there a few examples of true positives and false positives?
Sentinel is a small web dashboard that was written to track all the posts that we have detected. Most of the true positives can be seen there. A few of them are:
- direct link — Image for <10k
- direct link — Image for <10k
- direct link — Image for <10k
- direct link — Image for <10k
- direct link — Image for <10k
The 3 false positives were:
- Downloading and adding javax.enterprise.context.RequestScoped to my project — Which was edited to a proper answer before the end of the grace period
- Need to find reports on Search. E.g. Top searched keyword, categories; (Image for <10k)— Which, even though wasn't a link only answer, was deleted as such.
- How to skip values on particular indexes in Line Chart using MPAndroid Chart Library? — Which was a rhetorical question posted as an answer
More Stats
Every day we detect more than 150 bad posts, out of which nearly 125 are flagged. On November 25th, the number went to 227 flagged posts out of 269 with 30 of them in between 0900-1000 UTC. The graph for the past week is:
Date, Detected, Flagged
2017-01-30, 176, 160
2017-01-31, 191, 170
2017-02-01, 213, 190
2017-02-02, 201, 181
2017-02-03, 255, 226
2017-02-04, 122, 110
Out of the 1036 NAA flags this week, 100 of them would have been flagged and auto-commented (10%).
Can't SE directly stop such posts?
I have asked about this on Meta Stack Exchange when Tim requested ideas for the "Second Iteration of the Stack Exchange Quality Project".
Discussion
Now that you know the details of this system, what are your thoughts/comments/ideas?
Title courtesy: Can a machine be taught to flag comments automatically?