28

Related: Is this request for statistics of skipping in reviews hard to accomplish or does it somehow lack visibility?


This is sort of a follow-up request based on stats of how many reviewers performed certain amounts of reviews shared a while ago by SE team (details for those interested are on the bottom of this post).

I would like to get related stats for frequency of skips: what is average frequency of skip actions for users who performed certain amount of reviews.

Like, those who did 50 or less reviews skip 10% average, those with over 50 but less that 250 reviews skip 11% average, those with over 1000 reviews skip 12% etc.

I am also interested in similar stats for audit failures, like, users who failed less than 5% audits skip 20% average, 5 to 10% skip 10% percent average etc. For the purpose of these stats skipped reviews would probably better count as passed: although I don't do this myself I can perfectly easy imagine someone skipping audits with the purpose to do more real reviews instead.

Preferably (though not necessary) stats would be separate per review types - close, LQ, triage etc.

I expect requested stats to help me understand how likely it is that better educating inexperienced reviewers to use skip could help them be more productive. To me it is certainly the case but without statistics it is hard to tell whether there is some common pattern like that or not.


Per discussion in comments I won't mind if stats are limited for simplicity to reasonably recent ones like last 90, or 120, or 180, 360 etc days.

Another simplification that comes to mind is to limit stats to users who had sufficient rep for review at the start of picked period and who performed, say, no less than 5 reviews in that period.


As promised, details of past stats mentioned above:

https://i.sstatic.net/0BBq0.png

Sharp drop at 10-50 likely indicates that many users find it difficult to work in review queue. Drop after 250 (silver badge) suggests that even after substantial amount of reviews, many users still fail to discover a way to work productively...

15
  • 2
    Skip was not an option for some time, wasn't it? So you would need to take that into account, too.
    – false
    Commented Jun 26, 2017 at 10:44
  • 1
    that's a good question @false - I need to re-check older meta posts for that, it will take some time
    – gnat
    Commented Jun 26, 2017 at 10:49
  • 3
    @false as far as I can tell skip was there in all "modern" review queues for almost 5 years now: it is listed for all queues in the introductory announcement at MSE New Feature: Community Review Tasks - Now in Beta. I know there were some review queues for 10K users before that but I can't make my mind whether it is worth mentioning them here, need to think about it some more time
    – gnat
    Commented Jun 26, 2017 at 10:56
  • 1
    Mnm, my first review was 2012-04-28. And, I believe that there has not been any option to skip.
    – false
    Commented Jun 26, 2017 at 11:04
  • if it was so @false the only reason I could think of is bug in beta implementation. Because announcement is fairly straightforward on that from the very first revision, all the queue options lists include "skip it if you are Not Sure"
    – gnat
    Commented Jun 26, 2017 at 11:07
  • 2
    ...above discussion made me think that maybe it would be simpler to just limit stats request to few recent years, 2 or 3 or maybe 4
    – gnat
    Commented Jun 26, 2017 at 11:30
  • 2
    @gnat There was a suggested edit review queue before that post, but in the earlier implementation it didn't show one edit at a time, it showed a whole page of (I think 50) edits, and you could act on each one. There was no button to skip because to "skip" an item you just kept scrolling right on past it and didn't click any button at all, so there was no recorded review action. That's old enough though at this point that I doubt it'll pollute the stats notably, and it's easy enough to just ignore all review stats from that time entirely anyway.
    – Servy
    Commented Jun 26, 2017 at 15:14
  • thanks @Servy, so you think it would be okay to ignore stats that are, say, over 4-5 years old? I only vaguely remember how it was back then but if memory serves there was simply much less reviews (and the site itself was much smaller) so maybe these indeed won't make a difference
    – gnat
    Commented Jun 26, 2017 at 15:37
  • 1
    @gnat There was a review system for suggested edits, but there was no analogous version of any of the other queues. There also tended to be fewer people doing more reviews back then (as there was no review limit), and that combined with it just being a long time ago means that the number of users whose stats are dominated by reviews from back then has to be a pretty small number, but it'd be easy enough to just write the query to only go back 5 years, rather than for all time.
    – Servy
    Commented Jun 26, 2017 at 15:41
  • 2
    Looking for stats on how frequency of skip actions correlates with percentage of review actions by type and audit failures... For example: users who skipped 0-10% approved 85% of suggested edits on average with a standard deviation of 5%, then show that for 10-20%, etc. Commented Jun 27, 2017 at 14:51
  • @Dukeling that's an interesting twist, I haven't thought about it (guess I'm too much focused on productivity matters after that scary histogram with so much folks dropping off after just 50 reviews:). I am curious about these too, not adding it to my request only to keep its scope limited
    – gnat
    Commented Jun 27, 2017 at 14:55
  • @false have you encountered any bugs on this site due to your username? SO is pretty well coded, but your name is an interesting test case I wouldn't have tried. Commented Jun 27, 2017 at 16:36
  • @LordFarquaad: FALSE is probably more of a challenge
    – false
    Commented Jun 28, 2017 at 5:27
  • 3
    @gnat I think I slowed then stopped reviewing because it was too hard to keep "being nice" after the 4 or 5 first crappy reviews. And as usually recommended here: "don't comment if you you can't avoid being snarky and pass your way", so I did pass my way and stopped having a look there to avoid arguments with the "be nice with help vampires" team, (and a little less for arguments with post authors).
    – Tensibai
    Commented Oct 13, 2017 at 13:01
  • @Tensibai I like CV queue because it doesn't have an option to comment. (Granted there is a custom close reason but I made a habit to pick Unclear every time I feel like writing something like "I have no friggin' idea what kind of help you expect here with your friggin' homework dump")
    – gnat
    Commented Oct 13, 2017 at 13:12

1 Answer 1

7

As requested, the first thing that I pulled was the percent of skips used in each review queue based on the total number of reviews performed. I used Total Review buckets of < 50, 50-249, 250-499, 500-999, and 1k+. That query produces a graph that looked a bit underwhelming:

enter image description here

The data used to make that is:

+-------------------+--------------+------------+
|    ReviewType     |  # Reviews   | PctSkipped |
+-------------------+--------------+------------+
| Close Votes       | < 50         |         40 |
| Close Votes       | 50-250       |         37 |
| Close Votes       | 250-500      |         38 |
| Close Votes       | 500-1000     |         37 |
| Close Votes       | >= 1000      |         46 |
| First Post        | < 50         |         51 |
| First Post        | 50-250       |         40 |
| First Post        | 250-500      |         41 |
| First Post        | 500-1000     |         50 |
| First Post        | >= 1000      |         68 |
| Helper            | < 50         |         89 |
| Helper            | 50-250       |         86 |
| Helper            | 250-500      |         82 |
| Helper            | 500-1000     |         70 |
| Helper            | >= 1000      |         89 |
| Late Answer       | < 50         |         42 |
| Late Answer       | 50-250       |         29 |
| Late Answer       | 250-500      |         33 |
| Late Answer       | 500-1000     |         38 |
| Late Answer       | >= 1000      |         59 |
| Low Quality Posts | < 50         |         38 |
| Low Quality Posts | 50-250       |         34 |
| Low Quality Posts | 250-500      |         36 |
| Low Quality Posts | 500-1000     |         43 |
| Low Quality Posts | >= 1000      |         50 |
| Reopen Vote       | < 50         |         37 |
| Reopen Vote       | 50-250       |         32 |
| Reopen Vote       | 250-500      |         27 |
| Reopen Vote       | 500-1000     |         24 |
| Reopen Vote       | >= 1000      |         42 |
| Suggested Edit    | < 50         |          6 |
| Suggested Edit    | 50-250       |         31 |
| Suggested Edit    | 250-500      |         33 |
| Suggested Edit    | 500-1000     |         40 |
| Suggested Edit    | >= 1000      |         54 |
| Triage            | < 50         |         37 |
| Triage            | 50-250       |         33 |
| Triage            | 250-500      |         31 |
| Triage            | 500-1000     |         39 |
| Triage            | >= 1000      |         61 |
+-------------------+--------------+------------+

I found those numbers uninteresting, so I decided to take your request a bit further and provide a breakdown of the percent skipped in review based on different rep levels.

+-------------------+--------------+---------+-----------+-----------+-----------+-------------+------+
|    ReviewType     | TotalReviews | 500-999 | 1000-1999 | 2000-4999 | 5000-9999 | 10000-19999 | 20k+ |
+-------------------+--------------+---------+-----------+-----------+-----------+-------------+------+
| Close Votes       | < 50         |       0 |         0 |        42 |        38 |          39 |   40 |
| Close Votes       | 50-249       |       0 |         0 |        36 |        37 |          33 |   40 |
| Close Votes       | 250-499      |       0 |         0 |        42 |        38 |          32 |   40 |
| Close Votes       | 500-999      |       0 |         0 |        32 |        40 |          41 |   35 |
| Close Votes       | >= 1000      |       0 |         0 |        35 |        59 |          48 |   38 |
| First Post        | < 50         |      54 |        49 |        47 |        43 |          39 |   42 |
| First Post        | 50-249       |      43 |        37 |        40 |        46 |          41 |   42 |
| First Post        | 250-499      |      40 |        44 |        34 |        38 |          60 |   10 |
| First Post        | 500-999      |      56 |        50 |        43 |        56 |          41 |   39 |
| First Post        | >= 1000      |      73 |        73 |        60 |        63 |          98 |   81 |
| Helper            | < 50         |       0 |         0 |        90 |        86 |          91 |   85 |
| Helper            | 50-249       |       0 |         0 |        85 |        85 |          85 |   95 |
| Helper            | 250-499      |       0 |         0 |        86 |        76 |          87 |    0 |
| Helper            | 500-999      |       0 |         0 |        89 |         0 |          68 |    2 |
| Helper            | >= 1000      |       0 |         0 |        87 |        95 |           0 |   94 |
| Late Answer       | < 50         |      45 |        39 |        42 |        40 |          41 |   35 |
| Late Answer       | 50-249       |      28 |        28 |        27 |        37 |          47 |   23 |
| Late Answer       | 250-499      |      28 |        35 |        32 |        44 |          24 |   49 |
| Late Answer       | 500-999      |      45 |        37 |        41 |        28 |          51 |   30 |
| Late Answer       | >= 1000      |      84 |        59 |        44 |        72 |          70 |   80 |
| Low Quality Posts | < 50         |       0 |         0 |        42 |        34 |          37 |   30 |
| Low Quality Posts | 50-249       |       0 |         0 |        35 |        36 |          36 |   28 |
| Low Quality Posts | 250-499      |       0 |         0 |        41 |        37 |          26 |   29 |
| Low Quality Posts | 500-999      |       0 |         0 |        40 |        43 |          51 |   55 |
| Low Quality Posts | >= 1000      |       0 |         0 |        54 |        59 |          60 |   35 |
| Reopen Vote       | < 50         |       0 |         0 |        39 |        34 |          37 |   39 |
| Reopen Vote       | 50-249       |       0 |         0 |        34 |        29 |          22 |   41 |
| Reopen Vote       | 250-499      |       0 |         0 |        30 |        29 |          30 |    8 |
| Reopen Vote       | 500-999      |       0 |         0 |        14 |        16 |          36 |   28 |
| Reopen Vote       | >= 1000      |       0 |         0 |         0 |        47 |          49 |    4 |
| Suggested Edit    | < 50         |       0 |         0 |        18 |        15 |          18 |   14 |
| Suggested Edit    | 50-249       |       0 |         0 |        32 |        32 |          29 |   31 |
| Suggested Edit    | 250-499      |       0 |         0 |        32 |        37 |          34 |   33 |
| Suggested Edit    | 500-999      |       0 |         0 |        39 |        41 |          42 |   40 |
| Suggested Edit    | >= 1000      |       0 |         0 |        55 |        61 |          60 |   53 |
| Triage            | < 50         |      38 |        35 |        36 |        38 |          38 |   41 |
| Triage            | 50-249       |      34 |        31 |        34 |        37 |          39 |   42 |
| Triage            | 250-499      |      31 |        29 |        30 |        33 |          21 |   46 |
| Triage            | 500-999      |      41 |        37 |        38 |        18 |          50 |   52 |
| Triage            | >= 1000      |      73 |        72 |        40 |        39 |          34 |   86 |
+-------------------+--------------+---------+-----------+-----------+-----------+-------------+------+

I did the same thing for the audit failures. Here is a breakdown of the percent skipped in review by rep level, but instead of using total reviews the columns are the percent a user fails audits. Meaning a user failed audits 1-10%, 11-20%, etc. of the time. Note: you requested that we count audit skips as a pass, however I only checked if a user failed an audit presented to them. Also, there are no Helper queue stats because we don't have audits in that queue.

+-------------------+-------------+-------+--------+--------+--------+--------+--------+--------+--------+--------+---------+
|    ReviewType     |  TotalRep   | 1-10% | 11-20% | 21-30% | 31-40% | 41-50% | 51-60% | 61-70% | 71-80% | 81-90% | 91-100% |
+-------------------+-------------+-------+--------+--------+--------+--------+--------+--------+--------+--------+---------+
| Close Votes       | 2000-4999   |    25 |     28 |     10 |     25 |     18 |      6 |      0 |      0 |      0 |      15 |
| Close Votes       | 5000-9999   |    35 |     20 |     15 |     35 |     27 |      4 |     25 |      0 |      0 |      17 |
| Close Votes       | 10000-19999 |    32 |     19 |     42 |     18 |     26 |      0 |      0 |      1 |      0 |      21 |
| Close Votes       | 20k+        |    35 |     40 |     14 |     17 |      5 |      0 |     21 |      0 |      0 |      12 |
| First Post        | 500-999     |    42 |     38 |     32 |     32 |     34 |     30 |     14 |     19 |     21 |      31 |
| First Post        | 1000-1999   |    48 |     31 |     27 |     20 |     29 |     36 |     12 |     34 |      0 |      22 |
| First Post        | 2000-4999   |    35 |     30 |     22 |     34 |     26 |      0 |      4 |      0 |      0 |      21 |
| First Post        | 5000-9999   |    40 |     57 |     15 |     26 |     42 |      0 |     15 |      0 |      0 |      15 |
| First Post        | 10000-19999 |    39 |     38 |     43 |     85 |     17 |      0 |      0 |      0 |      0 |      26 |
| First Post        | 20k+        |    28 |     55 |      0 |     52 |     26 |      0 |      0 |      0 |      0 |      29 |
| Late Answer       | 500-999     |    27 |     31 |     25 |     25 |     28 |      8 |     11 |     11 |     12 |      17 |
| Late Answer       | 1000-1999   |    34 |     21 |     23 |     18 |     28 |      3 |     16 |     32 |      0 |      17 |
| Late Answer       | 2000-4999   |    25 |     23 |     14 |     22 |     25 |      5 |      0 |      0 |      0 |      16 |
| Late Answer       | 5000-9999   |    25 |     29 |      3 |     31 |     29 |      0 |      0 |      0 |      0 |       9 |
| Late Answer       | 10000-19999 |    17 |      0 |      0 |      9 |     20 |      0 |      0 |      0 |      0 |      19 |
| Late Answer       | 20k+        |    25 |      0 |     12 |     16 |      0 |      0 |      0 |      0 |      0 |      21 |
| Low Quality Posts | 2000-4999   |    38 |     26 |     20 |     23 |     30 |      0 |      0 |      0 |      0 |      24 |
| Low Quality Posts | 5000-9999   |    38 |     28 |     33 |     35 |     18 |      0 |      0 |      0 |      0 |      17 |
| Low Quality Posts | 10000-19999 |    37 |     12 |     27 |     13 |     16 |      0 |      2 |      0 |      0 |      17 |
| Low Quality Posts | 20k+        |    32 |     20 |      8 |     37 |     22 |      0 |      0 |      0 |      0 |      11 |
| Reopen Vote       | 2000-4999   |    38 |     19 |     24 |     26 |     18 |      0 |     45 |      0 |      0 |      15 |
| Reopen Vote       | 5000-9999   |    30 |     18 |     18 |     23 |     22 |      0 |      0 |      0 |      0 |      19 |
| Reopen Vote       | 10000-19999 |    41 |     20 |      4 |     23 |     14 |      4 |     45 |      0 |      0 |      24 |
| Reopen Vote       | 20k+        |     0 |     33 |     18 |     19 |     37 |      0 |      0 |      0 |      0 |      21 |
| Suggested Edit    | 2000-4999   |    29 |     23 |     24 |     26 |     17 |      0 |      9 |      0 |      0 |      11 |
| Suggested Edit    | 5000-9999   |    22 |     16 |     30 |     13 |     14 |     15 |      0 |      0 |      0 |      12 |
| Suggested Edit    | 10000-19999 |    26 |      9 |      5 |     30 |     22 |      0 |      0 |     18 |      0 |      10 |
| Suggested Edit    | 20k+        |    10 |     11 |     19 |     31 |     42 |      0 |      8 |      0 |      0 |      16 |
| Triage            | 500-999     |    32 |     23 |     23 |     18 |     24 |     10 |     36 |     17 |      0 |      24 |
| Triage            | 1000-1999   |    30 |     19 |     29 |     18 |     18 |     22 |      0 |      0 |      0 |      20 |
| Triage            | 2000-4999   |    22 |     12 |     25 |     20 |     20 |      0 |      0 |      0 |      0 |       9 |
| Triage            | 5000-9999   |    22 |      3 |     20 |      0 |      0 |      0 |      0 |      0 |      0 |      11 |
| Triage            | 10000-19999 |    21 |     37 |      3 |      3 |     15 |      0 |      0 |      0 |      0 |       0 |
| Triage            | 20k+        |    64 |      0 |     62 |     28 |      0 |      0 |      0 |      0 |      0 |      14 |
+-------------------+-------------+-------+--------+--------+--------+--------+--------+--------+--------+--------+---------+

Instead of trying to graph or chart this, I figured someone could do something much fancier than I could, so have at it with the numbers above.

9
  • great food for thought, thank you! Few quick observations: 1) reviewers appear to be aware of Skip and actively using it from the very start (except for Suggested Edits where active usage of skip seems to start at ~50 reviews, which looks early enough as well), 2) those who do over 1K reviews tend to skip more (possibly due to absence of "badge progress pressure"), 3) skipping more seems to somewhat help avoid audit failures
    – gnat
    Commented Nov 7, 2017 at 16:10
  • @gnat The one thing that jumped out was the high percentage of skips in the Helper queue, which to me makes sense. Those posts tend to need a lot more work and if you're at all unsure, I'd think you'd be more likely to skip than dive into it.
    – Taryn StaffMod
    Commented Nov 7, 2017 at 16:12
  • @bluefeet most of the posts I've seen in the helper queue should have been in the close queue instead. When I'm in that queue, I usually do a lot of skipping unsalvageable posts, I dont skip very many due to them being hard to fix (while actually being salvageable), that occurs much less often.
    – user4639281
    Commented Nov 7, 2017 at 16:16
  • 2
    yes, Helper queue shows much more skips and your explanation of the reason for that makes good sense. To me this looks really good, kind of proof that reviewers are well aware of Skip and don't hesitate using it (and I am personally pleased that this data saved me effort of making and pushing feature requests related to promotion of using Skip:)
    – gnat
    Commented Nov 7, 2017 at 16:16
  • ...on a further thought, additional Skip requirement for silver and gold badges now looks entirely feasible and harmless. Though if you're interested in pursuing this idea you better check these stats with data scientists because to me for example it's hard to tell by just looking at these numbers whether adding such a requirement could be beneficial
    – gnat
    Commented Nov 8, 2017 at 7:41
  • (rather tangentially related) I recently posted yet another stats request, would appreciate if you take a look when you have time: here. Technically it is probably trickier than this one because I am trying to estimate differences between weekends and weekdays for a fairly non-trivial value (efficiency of close voting)
    – gnat
    Commented Mar 29, 2018 at 13:22
  • 1
    @gnat I've seen the request and I'm guessing someone on the Community Team has as well, but with my new role I'm not sure I'll have bandwidth to dig in anytime soon.
    – Taryn StaffMod
    Commented Mar 29, 2018 at 13:28
  • 1
    I see, thanks. Didn't notice your role change, sorry (it was totally obscured by a name change:)
    – gnat
    Commented Mar 29, 2018 at 13:44
  • 1
    @gnat I'm just trying to confuse everyone. :)
    – Taryn StaffMod
    Commented Mar 29, 2018 at 13:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .