This is sort of two questions: "How are we measuring 'negative comments'?" and "Did the number of negative comments on SO decrease by ~50% in Q4 2019?" My team (along with several others) worked on both sides, so I'll do my best to answer both.
Measuring Negative Comments
First we have to be very specific about what we mean by "negative comments." Stack Overflow is a bit unique in that the "negative" comments that survive are not the sorts of negative comments you find on a lot of the internet - we're not talking about hate speech, slurs, targeted harassment, and the like that many other online communities have to wrestle with. We normally call these really egregious things "rude or abusive," like in the flag dialog. The community and moderators of Stack Overflow do a really good job of keeping the "rude or abusive" content cleaned up, very little of it survives.
The sort of negative comments we have to contend with are the "unkind, unwelcoming, or unfriendly" ones - those with condescending, dismissive, or otherwise subtly hostile phrasing. I feel like often there is confusion around this point since, in contrast to the "rude or abusive" cases, the intent of the commenter is rarely hostile. In my experience this can make discussion of the issue more difficult, since it requires separating the comment and its impact on a reader from the commenter and their intent - which is very unnatural, especially emotionally.
A few years back we started work on quantifying the magnitude of our unwelcoming comment problem - we had heard, both anecdotally and in more structured user surveys, that it was an issue but it's hard to measure progress or make rational decisions in the absence of data. To cut a long story short, we started with manually labeling new comments and progressed to using user generated flags and a human-in-the-loop automated flagging process.
The manual labeling started with employees, then rolled out to moderators, and then a wider group of volunteers. That data suggested that somewhere between 5% & 10% of new comments were perceived, by at least one person, as unwelcoming in some way. Later, we rolled out new flagging options for comments that allowed users to flag unwelcoming comments for moderator attention.
The data that was gathered from manual labeling and from the comment flags was used to train an auto flagger1 (internally, we call it the Unfriendly Robot). The robot doesn't delete anything, but it raises concerning comments for moderator attention - a setup often called "human in the loop". We have iterated on the robot, mostly by taking the cases where moderators have disagreed as test cases, and the flags it raises are now at least as accurate as the flags raised by regular Stack Overflow users. It is with this robot that we can now plot the change in negative comments over time.
In summary, we measure comment negativity using a custom algorithm that is trained on users' comment flags (and refined with moderators' responses to both those flags, and the algorithm's flags). Today when the algorithm raises flags, they are accepted at rates as high or higher than human raised flags.
Reducing Negative Comments
One nuance right away, while Prashanth spoke about the absolute number of negative comments, internally we mostly talk about percentages. In this case there isn't a practical difference, but he did make a (reasonable, IMO) simplification.
Given a way to classify negative comments, we can use it to look at the past and see how the prevalence of negative comments has evolved over time. This lets us evaluate the impact of our efforts.
Cumulatively, the robot thinks they've taken Stack Overflow from something like 1.5% of new comments being unwelcoming to a little less than 1%. This translates to something like a thousand improved comments a week, which get read by tens of thousands of people in the same period - and since content on Stack Overflow is consumed far more often than it is produced, the cumulative number of people impacted is even larger.
We've also used the robot to investigate specific changes, rather than just global trends. For example, Yaakov's work around post notices flow appears to have reduced the % of unfriendly comments on duplicate questions by ~10%.
Caveats, Footnotes, Etc.
The big caveat to acknowledge is that it's not possible to split test most changes around welcoming-ness, and thus we can only infer causation - not prove it. While, when we look at history, changes correlate to actions we'd expect to have had an impact - that is still only suggestive of causation, not proof of it.
Another thing to acknowledge is that, while I believe our efforts have made an impact on new content, there is still a very long tail (a decade or more of it) of old content. Changing perceptions of Stack Overflow's overall welcomingness to newcomers is a very large undertaking, and won't be accomplished by a single feature shipped or algorithm implemented.
Members of my team are planning at least one blog post that goes into the process of building the algorithm in detail, though it is a large enough and nuanced enough topic that it could still take a while to get published. I also expect they'll want to explore some of the nuances between preventing unfriendly comments (with things like policy changes), addressing unfriendly comments (with things like flags), and handling the long tail of old content that still gets frequently viewed. There are already some older posts that have alluded to bits of this topic: like these three.
A final thing I'd like to share - a lot of third-party approaches to these sorts of problems focus on users, building up profiles and histories to influence how they score particular pieces of text from those users. We explicitly decided against that approach. Our algorithms only consider the text of comments, no user specific details or user history are considered. This decision follows naturally from the observation that these subtly unwelcoming comments are often not intended to be unwelcoming.
1Because these are subtle and automated flags, they're a bit special. Mostly, any of the systems that consider flag history or volume ignore these flags.
Edit (from Jason)
Was it Q4 2019?
To answer the title question: No, this particular thing definitely didn't happen in Q4 2019. Sorry for the confusion. In the original post Prashanth mentioned a bunch of things that did happen in Q4 and I think this was "thrown in for good measure". We haven't really talked about it much publicly for reasons and I think he was right to add it in, as it has happened.
Was it half?
For the 30 days leading up to "Stack Overflow is Unwelcoming" the average daily unfriendly comment percentage (by robot V2 reckoning) is 1.41% and the average count of daily unfriendly comments is 493.
For the 30 days leading up to today the average daily unfriendly comment percentage is 0.821% and the average count of daily unfriendly comments is 244.
On a percentage basis it's a stretch (we reduced the percentage 41.7% relative). On an absolute basis it's true (we reduced the number 50.5%).
"Did you try anything?"
"Did you try anything?" gets a .73 from UnfriendlyRobotV2. "Did you even try anything?" gets a .941. "Did you even bother to try anything?" gets a .978. The maximum is 1 and the threshold for an automated flag is .907. So no, "Did you try anything?" isn't getting anyone in trouble.
Can we say that there are more negative comments from post authors or from readers?
UnfriendlyRobotV2 marks 1,015,098 of 70,878,654 comments (1.43%) from Not-PostAuthors as unfriendly. It marks 323,182 of 32,148,152 comments (1.01%) from PostAuthors as unfriendly. The 95% confidence interval of a proportion test is 0.422% - 0.431% (this is the difference between the two percentages) with a p-value of 2.2e-16.
UnfriendlyRobotV2 thinks Not-PostAuthors are unfriendly 41%-42% more frequently than PostAuthors and the difference is statistically significant.
Can we say that there are more negative comments under questions or under answers?
(Jason: I'm getting tired of saying UnfriendlyRobotV2, so I'm just gonna call it the robot from here on out).
The robot marks 508,183 of 50,763,777 comments (1%) on Answers as unfriendly. The robot marks 931,338 of 55,658,832 comments (1.67%) on Questions as unfriendly. The 95% confidence interval of a proportion test is .668% - .677% with a p-value of 2.2e-16.
The robot thinks that comments on Questions are 66%-67% more frequently unfriendly than comments on Answers and the difference is statistically significant.
Did 3CV change anything?
We moved the close vote threshold around in the second half of 2019. Did that have any effect on the percentage of unfriendly comments?
This was not an A/B test, so there's no control to compare against. All we can really do is look at a trend line. I don't see much of a change during the original 30-day experiment period. There's some rolloff in 2020, but I'd be hard pressed to attribute any of it to the final change which was made on December 3.
How helpful is the robot?
What percentage of robot flags are helpful? Here's a plot that puts it alongside the same metric for humans. (Human flags here are both Rude/Abusive and Unfriendly/Unkind as that's what the robot is trained on)
Robot V1 was definitely not up to the task (there were tons of false positives). Robot V2 looks as if it's doing well but it's jumped around a bit and recently has fallen to "merely" human levels. It's natural to come up with another metric which we call "The Robot Rating" and calculate as Robot %Helpful / Human %Helpful. That plot looks like this...
That isn't the whole story though, because the robot never sleeps. There is no comment that escapes its roboty eye. Let's plot helpful flags by source...
We see that even though V1 wasn't so accurate it still raised tons more helpful flags than humans did. (V1, though diligent, did get sick at the end of its tenure and dropped off a bit to "merely" human). How much are we improving detection using the robot? We get a multiplier by taking the ratio of Robot Helpful Flags to Human Helpful flags...
V1 helped us find 2X-3X more unfriendly comments than humans flagging alone. V2 has routinely been above 3X.
The robot gives a score to each comment. How helpful is the robot conditioned on score? If we round robot scores to the nearest percent and calculate percent helpful at each rounded score we get this...
We can see that in both versions the higher the score the more likely our moderators agree with the robot. We also see a jump in helpfulness across all scores that came with robot V2.