TL:DR: Compared to 2016, in last year of 2022 there were 6.3 times less reviews of suggested edits and 3.6 times less total reviews. Is this estimate accurate and if yes, what does such a decrease mean?
In my first several years at Stack Overflow I've been actively using reviews and suggested edits and seen others actively using these. Seeing this and reading meta made me feel that these are important features. There were active discussions and developers invested quite some effort into maintaining these features.
Some time later I started getting a feeling that something goes wrong. For quite a while I ignored this feeling thinking that it's just an old fart nostalgy, when I was youger grass was greener yadda yadda. Recently I figured that there is statistics data available to find out what's really going on, instead of feelings and guessing.
I found moderation reports for years 2016, 2018, 2019, 2020, 2021, 2022, and studied their data about suggested edits and total reviews. I was quite surprised with the trend I observed (guess I wouldn't if I paid closer attention to last years complaints about suggested edits).
There were fairly strong decreases year by year in the range of 20-30-40%, with total drop between 2016 and 2022 aggregating to massive 3.6 times less in total reviews and 6.3 times less reviews of suggested edits.
To me this looks like features I considered important in the past are essentially disappearing. And I wonder what could that mean and what could we expect next? Should we expect them to eventually fade into peripheral features of low impact?
Below are details about data I used - feel free to skip if you're not interested in detailed assessment of this study.
Here are counts I've got for total reviews in all queues:
Year | Count | Notes |
---|---|---|
2016 | 6,867,383 | |
2018 | 4,966,450 | 28% decrease |
2019 | 4,346,456 | 13% decrease |
2020 | 3,435,850 | 21% decrease |
2021 | 2,613,983 | 24% decrease |
2022 | 1,910,184 | 27% decrease / about 3.6 times less reviews than in 2016 |
Here are counts I've got for suggested edits reviews:
Year | Count | Notes |
---|---|---|
2016 | 1,864,076 | |
2018 | 1,177,058 | 37% decrease |
2019 | 1,141,695 | 4% decrease |
2020 | 817,980 | 29% decrease |
2021 | 529,477 | 36% decrease |
2022 | 295,589 | 45% decrease / about 6.3 times less reviews than in 2016 |
I tried to estimate maybe people are just unaware of reviews and it looks like this is not the case: over a million Custodian badges indicate that at the very very least 130 thousands users have tried reviews at least once (this is based on most pessimistic estimate that all badges were given to 3K users who can earn them 8 times).
Also, after observing such a sharp drop in suggested edits I decided to re-check how much it contributed to decline of total reviews. For that I calculated difference between 2016 and 2022 for all reviews excluding suggested edits. I've got about 3.1x decrease which is less than 3.6 but not substantially.
I also studied some events in site history that could potentially impact what I observed. I could not figure why decrease in 2019 looks lower compared to other years, there is a chance of a mistake somewhere in my calculations (when drafting and re-checking this question I found at least one mistake that had a substantial impact on my analysis).
I considered changes in review thresholds for suggested edits in May 2017 and for close votes in Dec 2019. Couldn't figure if these had any substantial influence on the data above.
I also considered influence of welcoming campaign rolled out in April 2018. It wasn't explicitly about reviews but maybe these were considered kind of an obstacle for a new party line, especially close reviews. Statistics of company efforts related to content curation that dropped to zero in 2018 (discussed here) suggests possible connection here, although smaller decreases in 2019 indicate that it's not really straightforward.
As was suggested in comments I additionally estimated total amount of review items that entered the queues, including those that weren't acted on. Purpose was to check if maybe this could somehow factor in decrease of reviews.
Per my observations, there was a trend of decrease from 2016 to 1021, although less pronounced than in performed reviews:
Year | Count | Notes |
---|---|---|
2016 | 4,010,000 | |
2017 | 3,650,000 | 9% decrease |
2018 | 3,410,000 | 7% decrease |
2019 | 3,190,000 | 7% decrease |
2020 | 2,950,000 | 8% decrease |
2021 | 2,770,000 | 6% decrease |
2022 | 2,820,000 | 2% increase / about 30% less items than in 2016 |
I could not figure how to estimate per-queue amount of review items and because of that it is difficult to make definitive conclusion of what above means. However, from data available it looks like the most likely explanation is that this decrease reflects one in suggested edits reviews.
Reason why I think so is that suggested edits seem to be the largest part of reviews, making for almost a third of all the reviews in all queues and these suffered most decrease in reviews. And what is specifically important here is the blocking nature of suggested edits queue meaning that the fewer reviews are made, the fewer items are allowed to enter this queue.