94% of waste is... worrying. Even Shog's lower estimate of ~42% is pretty high for something that should have these numbers as close to 0% as possible. So I asked our data team to give us some more reliable numbers, and Kevin Montrose came up with numbers that, while still not ideal, are way less worrisome than either of the other two estimates. We've pulled together some more numbers, going all the way back to 2015. TL;DR: it fluctuates, but 10~20% seems about right for "waste" if we assume all reviews and flags are legitimate.
Exactly why and how ReviewTasks
get invalidated is... complicated, and we don't have great history, hence the "seems about right" in the TL;DR above. Definitions of waste are tricky too, but here goes:
Definitions
We considered a review to be finished if its associated review task is deleted. We ignored audit reviews.
Aged out reviews are review tasks that have been invalidated, AND are either at least 14 days old when they were deleted OR the 3 days have passed since the last close-like flag was cast on the associated post. Because there are other rules we didn't simulate that prevent aging away close votes, this will overestimate the number of aged out reviews.
A review didn't result in a post state change if its review task was invalidated AND no close-like history row was created on the associated post after the review was created. Because this considers all post changes after a review (even those not directly caused by it), this will tend to underestimate the number of reviews that didn't lead to a state change.
And finally, the graphs you've all been waiting for:
Like I mentioned before the numbers would ideally be even lower than this, but these look a bit less worrisome than either of our earlier estimates. That being said, there's still work to be done here, obviously.