4

This. This is enraging.

No, not the user's post. Why Stack Overflow does not attempt to detect such non-answers is enraging and raise an automatic NAA flag. Just like we detect and block questions that don't appear to be in any language at all, or how we detect non-code formatted code.

These rules for detection can be:

if the answer is posted by the question poster and the user has 2-30 reputation:
    add 40 points if the string contains less than three space-delimited 
    when you remove these words*:

    <space> i I got found the solution solved problem fixed a :) . 
    thanks it myself me nvm nevermind

    add 10 points if the question the above words account for more than half the answer

    add 20 points if the question is shorter than 100 chars



    if >60 points:
        prevent user from posting
    elif >40 points:
        warn user
        flag as NAA if still posted
    elif 30 points:
        flag as BAA

 *And their lexicographically similar variations (ie typo'd versions); see lexicographic distance.        

I don't think the false-positive rate would be high; attempt to imagine a situtation where this would fail.

The sensitivity can be adjusted as SO wishes.

5
  • Isn't the cost vs benefit ratio fairly low for this? Benefit: users don't have to cast NAA flags, which we have a hundred of each. Cost: dev time, wasted mod time for false positives. You may be interested in checking out Natty. Also, how would users figure out that an auto-flag was raised? I won't run this calculation every single char, and I don't expect everyone to install a UserScript or something similar.
    – Kyll
    Commented Dec 11, 2016 at 23:34
  • @Kyll 1. mods don't answer NAA flags 2. because otherwise the flag dialog would've indicated so Commented Dec 12, 2016 at 0:20
  • 1
    Moderators handle hundreds of "not an answer" flags a day. I believe (but I need to check on this) that we handle the majority of these, even after they were placed in the Low Quality Posts review queue. They're generally the easiest and fastest flag to act on, given their narrow scope.
    – Brad Larson Mod
    Commented Dec 12, 2016 at 0:57
  • 1
    @BradLarson Uhuh. We should review more in that place. Commented Dec 12, 2016 at 1:00
  • @noɥʇʎPʎzɐɹC If you want to test your algorithm on real data, ping me in Sobotics, I've created a few test data dumps, out of which one of them might be useful for you. (Though it has only 11k records).
    – Bhargav Rao Mod
    Commented Dec 17, 2016 at 17:37

1 Answer 1

11

There are (or were) heuristics in place for detecting bad answers. At the time they were implemented, new users were presented with a "how to answer" page if an answer ranked highly enough as a "thanks" non-answer. That hasn't stopped people from posting those. The low-quality-posts heuristic that automatically inserted posts in review has such a terrible false positive rate that I believe it should be removed or completely reworked.

That said, I'd love to have a more automated system for detecting non-answers. If you have a demonstrably reliable heuristic for determining non-answers, please do present it, along with evidence of its hits and false positives. The analysis is important here, because in all the schemes I've seen for finding non-answers to flag, none have been useful without human oversight. Simply asserting that "I don't think the false-positive rate would be high" isn't good enough, I want to see some statistics on this.

For example, the phrase "found the solution" is present in a ton of good answers. It's not inherently bad to state that you've found the solution, and what it is, and this really isn't the most common kind of non-answer I come across. You're far more likely to see someone trying to post a question in an answer or say thanks than that.

My gut feeling is that your proposal wouldn't hit a very large number of answers, and has the potential to have a troubling false positive rate if used to block posts or automatically flag them. I could be convinced otherwise with detailed statistics, though.

2
  • It appears Natty is extremely accurate you can find it in this chatroom Natty - Bringing 10k moderation to All Commented Dec 12, 2016 at 1:25
  • 3
    @noɥʇʎPʎzɐɹC "Extremely accurate" is kinda an overstatement. At the moment, we are hitting an accuracy of ~90% , which goes to 99% with an increased level. The complete stats are here, gist.github.com/Bhargav-Rao/567753705a953043fedd96f806814a08 (Look at the Level and the ptp columns). We are running the bot at a lower level of accuracy (77%), in the sobotics room. That said we can actually move on to auto flagging at level 7 (~100% tps), But the issue is only 300 or so answers are found, out of the 3321 NAAs detected in 1 week.
    – Bhargav Rao Mod
    Commented Dec 12, 2016 at 7:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .