59

This is posted as a follow-up to my answer on another question, as requested by a Community Manager.

SEDE apparently doesn't accurately reflect the current situation in terms of how well we're doing with reviews, and specifically how many percent of reviews aren't handled to completion. Shog mentioned ~42% of close votes and flags age away, but that doesn't fully explain the review situation.

Can we have some more accurate statistics on close vote review task invalidation, as well as statistics on close vote aging?

12
  • 9
    An interesting statistic would also be, for the reviews that don't age away, how many votes came through the queue and how many by people voting on the question directly. I assume that most of the times, when the queue works, it's just one or two votes coming through the queue and the rest by people just visiting the question, but that's just a guess.
    – Erik A
    Commented Jun 27, 2019 at 18:26
  • 4
    Wouldn't surprise me. Most of my close votes haven't been cast from a queue at least. It would be interesting seeing those stats as well. Commented Jun 27, 2019 at 18:34
  • 1
    I've been spending some time lately at the CV queue, checking questions for the tags I follow and I've been wondering this myself as well, mostly because I've no clue if I'm spending my time wisely. Commented Jun 28, 2019 at 6:33
  • 1
    I would prefer this query (doesn't matter SEDE or internal SO database) to exclude reviews on questions that are less than 30 (better 40) days old. This would account for 30-days roomba cleanup. One could of course still complain about review being abandoned for whole 30 days and especially about askers deprived of guidance provided by closure with respective banner, but as of now we probably better focus on optimistic estimate and exclude questions deleted by roomba
    – gnat
    Commented Jun 28, 2019 at 7:55
  • 7
    @gnat I'm not sure if I agree. It might be best to do separate analyses, if the queue is clogged up with stuff that will roomba anyway and gathers useless review votes in the meantime, that's useful knowledge, especially since it might lead to a feature request to prioritize reviewing stuff that won't roomba to make the queue more effective.
    – Erik A
    Commented Jun 28, 2019 at 12:31
  • @ErikA that's a good point, I missed that there can be two separate analyses, that would be better for sure
    – gnat
    Commented Jun 28, 2019 at 12:34
  • @ErikA If there's going to be prioritization of questions which won't roomba, then we would need to determine in what time-frame will that potential Roomba action need to be in order to cause a question to not be prioritized. Are you thinking 1 day, 1 week, 1 month, 1 year, etc.? A lot can change in a short time. All it takes for something to no longer qualify for the 30-day or 365-day Roombas is for there to be an answer added, which is one of the points of getting it closed. In some cases, all it takes is an upvote. So, how much time should there be between now and the potential Roomba?
    – Makyen Mod
    Commented Jun 29, 2019 at 18:19
  • 4
    If we're looking at close vote reviews aging away, then I'd be very interested in seeing categories for how many aged away without being edited after entering the queue and those that aged away after they were edited while in the queue (e.g. where the OP added requested information or clarification).
    – Makyen Mod
    Commented Jun 29, 2019 at 18:21
  • 4
    I'd also be interested in knowing how many aged away based on the 4 day expiration of close-votes (>=100 views) vs. the 14 day expiration (<100 views). I'd also be interested to know how many views actually occurred while the question was in the Close Vote Review Queue. Even better would be the number of views by users capable of casting close-votes. However, I don't think the data for the latter two things is recorded.
    – Makyen Mod
    Commented Jun 29, 2019 at 18:25
  • you are not going to get this information, because closing and deleting are bad for the welcoming initiative and reduces eyeballs.
    – user10677470
    Commented Jul 1, 2019 at 14:13
  • 5
    @JarrodRoberson considering a full request was requested by a community manager, I think they're going to answer. Probably takes a while to colelct the data. Commented Jul 1, 2019 at 15:38
  • 13
    Another interesting point would be the number of reopen votes that age away. We leave open huge numbers of questions that should be closed, but we probably also have questions that should be reopened, but never get the views and votes they need.
    – Baum mit Augen Mod
    Commented Jul 1, 2019 at 20:32

1 Answer 1

26

94% of waste is... worrying. Even Shog's lower estimate of ~42% is pretty high for something that should have these numbers as close to 0% as possible. So I asked our data team to give us some more reliable numbers, and Kevin Montrose came up with numbers that, while still not ideal, are way less worrisome than either of the other two estimates. We've pulled together some more numbers, going all the way back to 2015. TL;DR: it fluctuates, but 10~20% seems about right for "waste" if we assume all reviews and flags are legitimate.

Exactly why and how ReviewTasks get invalidated is... complicated, and we don't have great history, hence the "seems about right" in the TL;DR above. Definitions of waste are tricky too, but here goes:

Definitions

We considered a review to be finished if its associated review task is deleted. We ignored audit reviews.

Aged out reviews are review tasks that have been invalidated, AND are either at least 14 days old when they were deleted OR the 3 days have passed since the last close-like flag was cast on the associated post. Because there are other rules we didn't simulate that prevent aging away close votes, this will overestimate the number of aged out reviews.

A review didn't result in a post state change if its review task was invalidated AND no close-like history row was created on the associated post after the review was created. Because this considers all post changes after a review (even those not directly caused by it), this will tend to underestimate the number of reviews that didn't lead to a state change.

And finally, the graphs you've all been waiting for:

enter image description here

Like I mentioned before the numbers would ideally be even lower than this, but these look a bit less worrisome than either of our earlier estimates. That being said, there's still work to be done here, obviously.

6
  • 2
    the way you explain it reads to me like, 80-90% of all questions that entered review queue either resolve to Leave Open or eventually end up closed and / or deleted. Would it be correct to say so?
    – gnat
    Commented Jul 26, 2019 at 5:33
  • 1
    It would, yes, @gnat :)
    – JNat StaffMod
    Commented Jul 26, 2019 at 19:14
  • 1
    this is good to hear. This means, even if review expires by timeout, other parts of the system eventually catch up and finish the job (10-20% losses sounds like sensible fail rate to me). From perspective of askers aged out reviews are probably not that great as this deprives them timely guidance for improving (30- and 365-days roomba are rather obscure, especially for newcomers) - but this is a different angle and possibly a different stats request. What I learned for now sufficiently addresses my concerns, thank you!
    – gnat
    Commented Jul 26, 2019 at 20:01
  • 2
    Wait, where did the 94% number come from at the top of this answer?
    – TylerH
    Commented Aug 9, 2019 at 18:55
  • 2
    @TylerH 94% is from my answer here (where it all started). It's based off SEDE, which for some reason contains incorrect data (also the reason why this question exists - see the first comment from JNat, which was also linked in this question) Commented Aug 9, 2019 at 20:47
  • 1
    I see, I tried to find it earlier but think I stopped somewhere around paragraph 15 of the answer you provided with no luck.
    – TylerH
    Commented Aug 9, 2019 at 21:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .