Thanks for the interesting blog post. Some random thoughts (I don't have experience with NLP).
- Is it planned to take into account context like the comments immediately before and after each comment? My feeling is that sometimes tension builds up gradually until someone kind of crosses the unfriendliness border. It might be important what was said before or after (mostly before I would guess).
- If I understood that right, moderators do mark 3.5% of random comments as unfriendly and about 70-80% of an automatically preselected set of 1% of all comments as well as 70% of a manually preselected set of 0.14% of all comments. What about increasing the workload even more and maybe feeding the moderators 2% of all comments (if that is a feasible workload), some even maybe randomly selected? That should result in more training material of the kind that we may be missing and even more removed unfriendly comments. If it's not feasible permanently, it should at least be possible for short times like feeding the 10% of most suspicious comments of a single day and then identifying which new unfriendly comments appear that the robot would normally not detect and overweight them in training.
- I wonder how long unfriendly comments live on average on the site before they are deleted? Is it minutes, hours, days? And how many people have seen them before they are deleted approximately?
- What about feeding those comments where the robot is very, very, very sure that they are unfriendly and very likely to be deleted to a separate review queue that has a higher priority and actually hides the comments from view until they are cleared by a moderator (awaits moderator approval shown to comment poster)? That way the decision is still with a moderator but the damage is further minimized.
- Just to be sure, it could be that moderators with an increased supply of potentially unfriendly comments have also lowered or increased their threshold for unfriendliness. Is there any indication of a changing ground truth, for example by feeding comments from the past that were already reviewed within some kind of audit?
- If I remember correctly, the UC-R1 did show a lowered performance towards the end of its lifetime. Is there a similar effect visible for version 2 so far, i.e. is the performance constant or changing significantly with time?
- Finally, from the blog post it seems that it's not always clear why a moderator deleted a comment. Are comments that are deleted by moderators but were not flagged (for example when just reading content and not working a review queue) included as training data? Maybe moderators should have different delete buttons (delete because unfriendly, delete because spam, delete because other... maybe 3-4 most common options). That could make the robot even better in the future.