I'm heading out on vacation for a couple weeks, but I wanted to try and address this before I left. To do that, I have to tell a story...
Return with me now to those thrilling days of yesteryear
Before we get to numbers, let's review the original motivation for Triage:
On April 17, 2014, we split Meta Stack Overflow into two sites: this one (which started almost empty with a handful of questions migrated in) and Meta Stack Exchange (which kept most of the content that the old MSO formerly housed). This seemingly-obscure bit of housekeeping had an immediate and dramatic effect: suddenly, thousands of additional people could vote on MSO due to the reputation they'd earned on Stack Overflow, and "hot" meta posts were promoted hourly in the main site's sidebar. For the first time in years, Stack Overflow had a discussion site of its very own, and folks wasted no time in letting us know exactly what was on their minds - within days, these two discussions blew up:
- Why is Stack Overflow so negative of late?
- Question quality is dropping on Stack Overflow
Both discussions focused on a controversy that'd been simmering for years: how to handle an ever-increasing wave of questions, many of them of such poor quality as to be unanswerable, without being complete jerks about it.
Needless to say, the overwhelming popularity of the despair and frustration expressed in these discussions caused some amount of consternation within the company. So within days...
...David Fullerton (now our esteemed President but back then only a lowly Vice President) kicked off the Stack Exchange Quality Improvement Project with this plea: Allow users to optionally filter out low-quality questions
David followed this a month later with a more specific problem statement and proposal: The Stack Overflow homepage is over-emphasizing bad questions (and a proposed solution)
...Which was followed within weeks by a new homepage design, which was iterated on for several months culminating in a complete set of filters with which folks could adjust the criteria to their heart's content.
Meanwhile, back in Quality Improvement Project land... I was getting really fed up with the First Posts review queue, since we'd tried to make it do too much and ended up with it not doing much of anything. After some internal discussions, I tossed up an idea for reviewing problematic posts as a prerequisite for making it onto the homepage. This was the genesis of Triage Review: a pre-filter for the homepage... Which, if you recall, we were simultaneously working on beefing up such that it would actually be useful. By the time Triage rolled out, that Recommended view for the homepage had been polished up as much as possible: you could filter by numerous criteria, with the caveat that you only got one view: your tag preferences + whatever homepage options you set.
I'm gonna take a break from this timeline now, because... We could have stopped right there, in December of 2014. All the pieces were there: more powerful filtering on the homepage, triage to keep embarrassing stuff off the homepage, and a Help & Improvement queue to maybe allow some of the less embarrassing stuff to come back into play. Granted, all of them needed a lot more refinement - Triage alone took months to really "finish" - but the core ideas were there.
However... The homepage was an even bigger headache. At its core it was still driven by a mixture of tag preferences and magic machine learning; the filtering options were powerful but confusing. It wasn't very predictable or fast, and for some uses it scaled poorly. It looked and worked completely differently from all the other question lists. And finally, the Recommended homepage was one of four default homepages: the Interesting tab was still alive (though a good bit of the work on Recommended would be backported), the Active tab was used for all sites outside of Stack Overflow, and a variation on the Active tab (with fewer questions, built-in tag filters, and slightly different selection criteria) was used for anonymous readers outside of Stack Overflow.
Navigation was a mess. And... we weren't really making it better by tacking on yet another view. We'd already been discussing improvements for a while, but around the same time everything else described above was happening we'd begun thinking about this in a more structured way: how were folks using the site, and what could we do to make that work better?
Sam the Brand kicked things off with a discussion on meta after doing a bit of investigation via Google Analytics.
Then Sklivvz took and ran with it, diving deep into data on how users who answered questions navigated the site, tracking back via referers from when an answer was posted to identify the view the author had used to get there. By early 2015, he'd published a report internally, and the evidence was damning:
The single biggest source of answers was Questions By Tag - Newest, the view you see when you manually filter by one or more tags.
The next biggest source of answers, responsible for roughly half the answers of #1, was the homepage. Specifically, the Interesting tab.
The third biggest source of answers, responsible for less than half of #2, was unknown: either the user's browser wasn't passing a referrer, or the referrer was external (cough Google cough).
Sources trailed off pretty fast from there, but there was a bit of a long tail... Sklivvz, being the thorough sort, didn't stop with a list of referrers: he went on to calculate which sources were the most profitable for answerers as well (resulted in answers that got lots of votes): this demonstrated that sources like Unanswered By Tag, which were responsible for only a fraction of all answers, were none the less some of the best places to go looking for questions to answer if you wanted your answers to do well. The homepage was middling in this regard, probably only a little more profitable than answering questions you stumbled onto via Google.
Finally, Sklivvz did a bunch of analysis on user behavior as it related to answering. Not surprisingly, he found that most users did not behave optimally when it came to earning reputation. We were doing a poor job of guiding answerers toward rewarding questions.
The conclusion of this study led to...
the introduction of the New Navigation project in March of 2015. The goal here was to replace all question lists with a unified interface, easily customizable and consistent throughout. New Nav saw continual improvements throughout 2015, culminating in a Release Candidate.
Then New Nav was abandoned and eventually retired.
It's notable here that New Nav was not predicated on Triage, nor was Triage predicated on New Nav - they were separate projects with separate goals. But, thanks to Sklivvz' research, it is clear that Triage's goal - improving the homepage - was never going to be sufficient on its own; far more questions are answered outside of the homepage, where Triage had no effect. The only practical way to benefit the majority of answerers would have been to make this distinction irrelevant: and this is precisely what New Nav set out to do.
So what good is Triage?
Ok, now we can get to some numbers. From my perspective, it's still a better First Posts review than First Posts Review, and a better way of handling VLQ flags on questions than Low Quality review. But that doesn't directly help folks just trying to browse the site without bumping into lousy questions.
So let's focus on views. In May of 2018, 256,414 questions were posted on Stack Overflow. Of those, 31586 entered Triage, and 25508 were reviewed completely (that is a consensus was reached). The average view count across all questions posted in May was 89; the average view count for questions that entered Triage was 64; the average view count for questions that completed Triage was 67.
Broken down by Triage consensus:
Triage result |
Questions |
Avg Views |
Looks Good |
11379 |
76 |
Should Be Improved |
9825 |
66 |
Unsalvageable |
4299 |
48 |
Conclusion
Triage has not done everything we hoped it would, because our hopes were based on an understanding of how folks used the system that didn't reflect reality. In many ways, including but hardly limited to Triage, we've been unable to adjust the system such that it matches up with the reality of how folks are best able to use it effectively. Hopefully that will change.
However, Triage is not unsuccessful. Beyond its utility as a tool for moderation, it does at least partially fulfill the original goal: ensuring that fewer people have to look at terrible questions.