45

Stack Overflow is using post revision related badges, such as Copy Editor and Strunk & White, to encourage users to revise posts to maintain and improve the quality of posts. However, by mining the Stack Overflow data (i.e., sampling ~90K answers that have at least 5 revisions), we have some thoughts about the current revision-related badges that we want to share. The current badges are primarily encouraging the quantity of edits (e.g., a badge is awarded after making X number of edits) while ignoring the quality of the edits to some extent.

We find that some users perform spikes of revisions in a single day when pursuing a badge. However, such spikes of revisions could be low-quality, which may increase the chance of the revision being rolled back and sometimes may even offend others.

Below we share some points that we found by mining the revisions of posts in detail:

  • In general, users who have revision-related badges are 12 times more likely to perform “spikes” of revisions (i.e., perform much more revisions than usual in a single day) compared to the users without revision-related badges. These kind of spikes are more likely to occur right before a user is rewarded a badge. Users perform spikes of revisions on 20.6% of the days that they are rewarded badges.
    The following figure presents such an example of a user. The Y-axis shows the number of revisions per day and the X-axis represents the timeline. We see that there are spikes around the day when the user was awarded badges (i.e., Strunk & White, Copy Editor, and Archaeologist).

  • So why spikes of revisions are bad? We find that making too many revisions a day increases the chance of a revision being rolled back (i.e., rejected by earlier answerers). The following figure presents the probability of making a revision that will be rolled back against the number of revision per day (RevisionPerDay). We can see that the probability goes up as RevisionPerDay increases. enter image description here

    We also did a manual study on the reason of the rollbacks, and we find that users that have revision-related badges are more likely to perform unwanted text-related changes (e.g., unwanted formatting).

  • Some users tend to perform text and simple (e.g., in terms of the size of changes) revisions, especially when they perform many revisions in a day. The figures below present the ratio of different types of revisions and size of revisions against RevisionPerDay. enter image description hereenter image description here

    We manually check hundreds of such revisions and observe that users tend to help others with performing shallow revisions (e.g., formatting) during the spikes rather than in-depth revisions.

In summary, we observe that some users may prefer the quantity of revisions instead of quality when pursuing badges.

Based on our findings, we wonder if Stack Overflow should consider improving the way that these badges are rewarded? Or do you have any thoughts about how to improve these revision-related badges?

26
  • 1
    I know that edits on questions that were deleted afterwards don't count towards the badge but do edits that were then rolled back still count? That would seem kind of odd.
    – Keiwan
    Jun 7, 2017 at 16:39
  • 1
    @Keiwan pretty sure as long as the edit is part of the post history of an undeleted post, it counts. The counting is only done based off visibility, not validity. Jun 7, 2017 at 16:41
  • 2
    @Keiwan the badge count is based on the number of posts, not actually based on edit. so no matter how many edits you do on one post, only count as one. Jun 7, 2017 at 16:47
  • 5
    SO has rate-limits on almost anything you do, specifically to avoid ab/use like this. But not on edits for some rather mysterious reason. Plenty of other trouble caused by this, from DoS attacks on users' Interesting page view to very current problems with the edit review queue. Jun 7, 2017 at 17:19
  • 3
    @HansPassant don't conflate suggested edits with full editor privs. The former is capped to N per user and a total of M on the site itself and require review. Most edits will "bump" the post again, so it gets eyes on it, and they can be looked at and flagged if abuse is happening. Jun 7, 2017 at 18:44
  • 3
    Is there a similar effect for the review task badges (like Steward)?
    – ryanyuyu
    Jun 7, 2017 at 18:46
  • 1
    @ryanyuyu We got the data of revisions from the post history.xml which is published on archive.org/details/stackexchange. Not sure about where to get the review task data. I also checked the data on data.stackexchange.com, the review task doesn't have userid, so probably hard to do this. Jun 7, 2017 at 19:01
  • 1
    Ah well, it's a lot of good research already. I was just hoping you happened to have the review info too. Thanks.
    – ryanyuyu
    Jun 7, 2017 at 19:04
  • 4
    Considering the point about rollbacks, one way to discourage the "quantity over quality" behavior would be to have the badges not credit any edit submitted on the same day as a rolled-back edit (or alternatively, no credit for edits submitted within 24 hours following submission of an edit later rolled back, since this can be computed using a single pass through the edit history).
    – Ben Voigt
    Jun 8, 2017 at 3:07
  • 14
    @WangShaowei your post is focusing on rollbacks, but as a revisionist (12,433 posts edited), I have no way to know which of my edits got rollbacked, so I can't improve myself on that. I believe we should first improve on communicating on when an edit got rollbacked. And possibly why an edit got rollbacked by eventually adding a rollback-message feature.
    – Cœur
    Jun 8, 2017 at 3:41
  • 13
    You use the word "we" a lot here, I'm curious who else was involved in creating this analysis? Interesting work, not sure what, if anything, can be done about it though.
    – DavidG
    Jun 8, 2017 at 10:13
  • 6
    I've always wanted to be able to upvote good edits. Perhaps something along those lines could be used to help determine "high quality edits". Coupled with number of edits, it would be similar to a tag badge (e.g. 20 edits with a total score of 20, etc.).
    – Shepmaster
    Jun 8, 2017 at 14:38
  • 2
    As far as quantity vs quality, it seems to me that the only moderately simple way to quantify quality is to put in a voting mechanism for edits themselves. Then allow only >2000 rep users to vote on that. Of course something like that would be complicated and likely have its own unintended consequences. Jun 8, 2017 at 14:42
  • 2
    @Cœur that's a good point. I never know SO doesn't get back to users about the rollback... As I know, one way to know the rolled back revisions is to query from data.stackexchange.com. Jun 8, 2017 at 14:51
  • 1
    @Cœur: That would encourage edit wars, which do much more damage than some unwarranted rollbacks. Jun 8, 2017 at 15:26

1 Answer 1

4

I would suggest adding something about the quality of the edit to all of those badges.

For example Strunk & White which is: Edit 80 posts. may become Edit 80 posts and more than 90% edits are accepted.

Percentage can be discussed.

It will apply only for -2,000rep users, as 2,000+ rep edits are instantly accepted. I assume 2,000+ users should know what a good and a bad edit are.


Note that we already have this kind of think in our badges, such as the Electorate badge:

Vote on 600 questions and 25% or more of total votes are on questions.

4
  • What about users with >2000 rep? Their edits are automatically accepted right.
    – CinCout
    Jun 9, 2017 at 9:27
  • @CinCout As I wrote in my answer: "It will apply only for -2,000rep users, as 2,000+ rep edits are instantly accepted. I assume 2,000+ users should know what a good and a bad edit are."
    – Mistalis
    Jun 9, 2017 at 9:43
  • Technically, you edited your answer later ;) Thus making it a good edit :D
    – CinCout
    Jun 9, 2017 at 11:06
  • 5
    @Mistalis The spikes are usually from professional revisors which usually have high-reputation. So more attention should be paid to high-reputation users. Jun 9, 2017 at 14:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .