Stack Overflow is using post revision related badges, such as Copy Editor and Strunk & White, to encourage users to revise posts to maintain and improve the quality of posts. However, by mining the Stack Overflow data (i.e., sampling ~90K answers that have at least 5 revisions), we have some thoughts about the current revision-related badges that we want to share. The current badges are primarily encouraging the quantity of edits (e.g., a badge is awarded after making X number of edits) while ignoring the quality of the edits to some extent.
We find that some users perform spikes of revisions in a single day when pursuing a badge. However, such spikes of revisions could be low-quality, which may increase the chance of the revision being rolled back and sometimes may even offend others.
Below we share some points that we found by mining the revisions of posts in detail:
In general, users who have revision-related badges are 12 times more likely to perform “spikes” of revisions (i.e., perform much more revisions than usual in a single day) compared to the users without revision-related badges. These kind of spikes are more likely to occur right before a user is rewarded a badge. Users perform spikes of revisions on 20.6% of the days that they are rewarded badges.
The following figure presents such an example of a user. The Y-axis shows the number of revisions per day and the X-axis represents the timeline. We see that there are spikes around the day when the user was awarded badges (i.e., Strunk & White, Copy Editor, and Archaeologist).So why spikes of revisions are bad? We find that making too many revisions a day increases the chance of a revision being rolled back (i.e., rejected by earlier answerers). The following figure presents the probability of making a revision that will be rolled back against the number of revision per day (RevisionPerDay). We can see that the probability goes up as RevisionPerDay increases.
We also did a manual study on the reason of the rollbacks, and we find that users that have revision-related badges are more likely to perform unwanted text-related changes (e.g., unwanted formatting).
Some users tend to perform text and simple (e.g., in terms of the size of changes) revisions, especially when they perform many revisions in a day. The figures below present the ratio of different types of revisions and size of revisions against RevisionPerDay.
We manually check hundreds of such revisions and observe that users tend to help others with performing shallow revisions (e.g., formatting) during the spikes rather than in-depth revisions.
In summary, we observe that some users may prefer the quantity of revisions instead of quality when pursuing badges.
Based on our findings, we wonder if Stack Overflow should consider improving the way that these badges are rewarded? Or do you have any thoughts about how to improve these revision-related badges?