36

I've been finding a lot of instances of plagiarism on Stack Overflow over the past few weeks, and having to point each instance out to mods for cleanup is a very time-consuming process, and frankly I just don't have the time to spend on hunting down each case alone like I've been doing.

I would be interested in starting a discussion on how we can improve the process of cleaning up plagiarism (beyond just manually going through a user's answers and googling each and every one for an original source), but first I'm interested in finding out some stats about it, such as (but not limited to),

  1. How many cases, on average, are reported to mods over a period of time (like a month)?
  2. Of the total reports, how many are true positives that result in answer deletion?
  3. How many true positives are merely edited to properly cite the copied source, rather than get deleted?
  4. How many users, on average, report plagiarism (like in a month)?
  5. Of users who are found to have plagiarised, what is the distribution of their reputation (before answer deletion occurs)? For example, do 50% have < 1,000 rep, 40% have < 10k rep, and 10% have > 10k rep?

    Why is this an interesting stat? Because I've found a lot of high-rep (over 10k) users who have plagiarised answers. Some were even close to 25k, "trusted user" status. If we come up with a new process for cleaning up plagiarism, I'd rather not have users who are potentially plagiarisers involved in plagiarism review.

  6. Of users who are found to have plagiarised, what is the distribution of their account age?
  7. Of plagiarised answers, what is the distribution of their age before they are discovered and resolved (either by deletion or editing)?
  8. Of plagiarised answers, what is the distribution of their score before they are discovered and resolved?
  9. Of users who are found to have plagiarised, how many are caught with new instances of plagiarism? In other words, how many continue to plagiarise with new answers, after they've been caught and warned about their first offense?
  10. Of users who are found to have plagiarised, how many are later reported for additional discoveries of old instances of plagiarism that they committed? This is different from #9, in that the user hasn't re-offended, but not all of his/her old offenses have been discovered and cleaned up yet.
  11. This is probably going to be controversial, but I'd like to know if any interesting demographic data can be derived from the plagiarism reports, such as age, gender, and country distribution of offenders?

    I think this would be interesting because the overwhelming majority (like 95-99%) of plagiarism cases that I find are committed by individuals belonging to a particular geographical region and culture. I know Tim Post has talked about the influence of culture on voting patterns among users before. I'm wondering if culture is going to be an issue if more SO users are involved in a new plagiarism review process? For example, would users belonging to a culture X tend to "go easy" on their reviews of other users from X who are discovered to be plagiarising?

  12. Of users found to have plagiarised, what is the distribution of the percentage of their answers that are found to be instances of plagiarism? For example, out of all discovered plagiarisers, do 10% have 50% of their answers reported, 40% have 30% of their answers reported, and the remaining 50% have only 5% of their answers reported?

  13. Of the X new SO accounts that are created every month, Y% will probably go on to commit plagiarism (either intentionally or otherwise) at some time in the future. I'd like to know what X and Y are if possible.

  14. Etc.

Is it possible for a moderator or an SO employee to provide these stats?

21
  • 2
    Probably at least 90% of flags that use some form of the word plagiarism end in answer deletion. Once you include things like "he copied my answer" that percentage starts tanking. We never edit attribution into answers unless the answers have a very good reason to stay (e.g. the only answer, high votes, accepted) - other than that the burden is on the user to do it.
    – BoltClock
    Oct 8, 2015 at 12:55
  • 1
    I don't think any of us have been keeping track of these, nor do we have access to raw numbers, so they'll have to come from an employee.
    – BoltClock
    Oct 8, 2015 at 12:56
  • 17
    #5 oh man that gets my goat so much, you don't even know. The thought alone of someone reaching 10k or trusted user status largely through stealing content makes me physically sick - and I've had the displeasure of handling a few such cases.
    – BoltClock
    Oct 8, 2015 at 12:59
  • @BoltClock not super important to know, but I've noticed that Brad seems to handle a lot of these cases...does he end up handling most of the plagiarism reports among the SO mods?
    – Bob
    Oct 8, 2015 at 12:59
  • Indeed, he does. He's also one of those most vocal about it on meta, besides myself.
    – BoltClock
    Oct 8, 2015 at 13:00
  • 9
    Respect to bob, signed up 24 days ago and already on the hunt for injustice with an epic meta bullet list to back it up.
    – Gimby
    Oct 8, 2015 at 13:30
  • 7
    The temptation to copy/paste the above question body into a question of my own is almost overwhelming.. Oct 8, 2015 at 13:31
  • 3
    Man 13+ questions is a lot. Honestly I stopped reading after about number 3. Perhaps you should ask only the most important questions?
    – ryanyuyu
    Oct 8, 2015 at 13:34
  • 1
    @ryanyuyu I don't know, they all seem pretty important to me. I want to better understand the big picture of the problem, because all I have is the very limited, very narrow view of all the cases that I've personally found. I don't know what the mods know, and I don't have access to super-secret employee-only data like Shog. Also, perhaps you need to drink some coffee to aid your attention-span. Kidding...
    – Bob
    Oct 8, 2015 at 13:37
  • 3
    @ryanyuyu: I actually think it's fine, but then again, I'm a mod, and thus part of the target audience...
    – BoltClock
    Oct 8, 2015 at 14:32
  • 5
    #11 if experience has taught me anything, it is yes, people who don't understand are going to approve plagiarized posts simply because they don't see any harm in them. And that frightens me because how are we going to reconcile our ideals with theirs on such a polarizing issue?
    – BoltClock
    Oct 8, 2015 at 14:34
  • 6
    @BoltClock we should simply have an official guideline for it, like every university on the planet. That guideline would be law, much more so than a mushy community consensus. The team seems to be shying away from the idea, no idea why.
    – Pekka
    Oct 8, 2015 at 14:45
  • @Pekka 웃: Probably because this is basically a given in the Western world (or even just academic settings if not everywhere). Stealing is not okay. Virtually any reasonable person at least in our culture knows that.
    – BoltClock
    Oct 8, 2015 at 14:47
  • 9
    @BoltClock, the problem with plagiarism without attribution is much deeper in a Q&A setting. It's not just ethical. It cuts the link between the answer and its origin. This inhibits further research by anyone who wants to look into the topic further. This corrodes the investigative process and by doing so it undermines the purpose of the forum. Oct 8, 2015 at 21:48
  • 1
    @Dmitry Rubanovich: Completely agreed.
    – BoltClock
    Oct 9, 2015 at 3:47

3 Answers 3

37

It'd be nice if plagiarism had its own custom flag, and that flag went into a queue that could be handled by the community. Five votes that it's plagarism causes the post to be deleted and we get notified once (An auto review queue) just so we can make sure it worked out (the dates of each post are really important here).

There should be strict guidelines on what plagiarism actually is:

Plagiarism is:

  • Copying someone's post wholesale and pasting it without attribution from another Stack Overflow post
  • Copying someone's post wholesale from another site and pasting it into a Stack Overflow answer without attribution

Plagiarism is not:

  • two answers using the same method to solve a problem.
  • two answers posted within minutes of each other that use the same format for solving a problem.
13
  • 2
    This is a great idea. It might even make sense to have a separate field for the plagiarized source, for faster opening
    – Pekka
    Oct 8, 2015 at 14:43
  • 1
    I've thought about how a community-driven solution would work, and I've always had adding some sort of new review queue in mind. But this is why I ask for stats on the reputation levels of plagiarisers. Who gets access to this new queue? It leaves me uncomfortable to make this merely a 10k or 25k privilege, because as I've pointed out, plagiarisers at those rep levels are not uncommon. It almost makes me think that it should only be available at like 100k rep, but of course I've also read arguments against new privilege levels.
    – Bob
    Oct 8, 2015 at 14:44
  • 2
    Yeah, rep based privileges just don't scale anymore, period.
    – BoltClock
    Oct 8, 2015 at 14:44
  • 1
    @Bob is it really a problem if plagiarizers can access the queues? I'm having a hard time imagining how they could seriously obstruct the process
    – Pekka
    Oct 8, 2015 at 14:44
  • 1
    @Pekka 웃: They could thwart the review process by okaying everything in there. Like robo-reviewers, but actually decisive.
    – BoltClock
    Oct 8, 2015 at 14:45
  • 4
    You could simply not allow someone to act on their own post or a post they're affiliated with. Oct 8, 2015 at 14:48
  • But is that really likely to happen? Wouldn't someone have to literally lurk in the queues 24/7 to protect their plagiarized posts?
    – Pekka
    Oct 8, 2015 at 14:49
  • @George Stocker: Own-post is a no-brainer. But how would you define affiliation?
    – BoltClock
    Oct 8, 2015 at 14:49
  • @BoltClock If you've answered, commented, or voted on any posts related to that post (so if it's an answer, if you've voted, answered, or commented on any answers on that question; as well as the other post that it's either the source of the plagiarism or the target of plagiarism). Oct 8, 2015 at 14:54
  • 14
    Sounds good, though I'd leave out users who voted on or own the source posts as they often know best if the answers were indeed stolen. I've seen stolen answers and recognized them right away without even having to do a check because they were my own writing.
    – BoltClock
    Oct 8, 2015 at 14:58
  • @GeorgeStocker: Well, the question is whether that would freeze out too many users who over a long time contributed immense amounts, compared to those actually having an ax to grind, or not. As (presumably), they would also be the best possible reviewers... Oct 8, 2015 at 14:58
  • 1
    @BoltClock: Re rep-based privileges: I got an idea for that that I'm working on, so I'm glad to hear the dissatisfaction expressed! ;) Oct 8, 2015 at 22:33
  • George, I'm getting some very strange, confusing signaling on my proposed solution, should I open a new question for discussing solutions on how to improve handling plagiarism, and then you and I can re-post our proposals there?
    – Bob
    Oct 9, 2015 at 0:29
27

I can't answer all of your questions, but I can present some statistics from what I can parse out of messages / comments we have left in cases of plagiarism.

In the last month, 24 users have been messaged by moderators for plagiarism. We tend to message only for cases where there is more than one plagiarized post, and leave comments for individual plagiarized posts. Of those messaged, only one was above 5k in reputation, 3 in the range 2k-5k, 3 in the range 1k-2k, 4 in the range of 500-1k, and 13 below 500 reputation.

As you can see, this is a relatively rare occurrence, given the huge number of active users we have on this site. These numbers are even higher than usual, given the recent push by some to seek out plagiarists. In this same period, we messaged over five times as many people for running sock puppets, for example.

However, each plagiarist can dump a lot of stolen material on the site before they are caught, so cleanup can be harder to handle. I've personally handled maybe a third of the 24 users we messaged, yet I've deleted 90 answers for plagiarism over the same period. While not common, when a plagiarist is found it can be a pain to dig through all their answers and verify original sources.

Again, this can seem like a huge problem when you run queries and find dozens of incidents of plagiarism, but relative to the amount of content that comes in every day, it's actually surprisingly rare.

2
  • "While not common, when a plagiarist is found it can be a pain to dig through all their answers and verify original sources." I can definitely attest to this, and this is what motivated me to ask this question. I've given up on investigating answers for some plagiarisers I've found, simply because I lack the time to check them all out, and that bothers me a lot, because I know that there is probably still plagiarism out there just lying around undiscovered on the site.
    – Bob
    Oct 8, 2015 at 15:01
  • I don't expect you to have records going this far back, but by any chance do you remember what these same stats looked like around the same time last year? There was a big, organized cleanup of a lot of plagiarism by a group of users then.
    – Bob
    Oct 8, 2015 at 15:10
-3

It was my intention to get a better handle on the nature of the problem of plagiarism on Stack Overflow before attempting to brainstorm a solution, but since George already got the ball rolling, I might as well add what's been on my mind...

I can't search and flag a plagiarist's answers alone anymore, automate and crowd-source the process for me please

As Brad points out in his answer, and as I can firmly attest to, once a plagiarist is discovered, it is a time-consuming process to examine all of his or her other answers to check whether or not they have also been plagiarised, especially for users who have posted a hundred (or hundreds) of answers. I've seen accounts less than a year old post a lot of answers, so the volume of answers to check isn't a problem unique to old accounts.

I'd like my workflow to be automated and crowd-sourced to other reviewers. So instead of I, a single user, combing through a plagiarists answers alone, I propose that once a new plagiarist is discovered through a flag on a single answer, that all of that user's answers enter a new plagiarism review queue. Then each answer could be examined to determine whether or not it has also been plagiarised. That way we can keep track of what's already been reviewed and what hasn't, and if I only have time to review a single answer or two, that's fine, because then some other reviewer(s) can take up the slack.

Then I'd imagine that the review process in that queue might go something along the lines of what George has already proposed.

11
  • 15
    This sounds like a feature request instead of an answer to your own question. I think you should post this as a separate feature-request to keep the discussions more focused. You'll get more feedback on people's thoughts to your specific feature request that way. Link back to this question for context if you need to.
    – ryanyuyu
    Oct 8, 2015 at 16:08
  • There will be robo-reviewers. Then, there will be answers which are plagiarism, but no one will bother determining because they passed the reviews.
    – user12205
    Oct 8, 2015 at 20:07
  • 2
    +1; this doesn't deserve -4.
    – whoKnows
    Oct 8, 2015 at 20:08
  • 1
    @ryanyuyu my request gets downvoted to -4, yet George's request for an additional feature gets upvoted to a score of 19? You Meta voters are a very strange, silly lot indeed!
    – Bob
    Oct 8, 2015 at 23:53
  • 3
    Yep. But you should still have posted a new question to separate two distinct posts. This post is not an answer in any way to your question about how much plagiarism mods handle. To be fair, George's answer isn't either, but he is at least a moderator who potentially can answer this question accurately. The way you phrased the title question made it basically impossible for any non-mod to answer with any degree of authority. Just post a new question with your feature request. Then the voting will be more accurate on what the community believes.
    – ryanyuyu
    Oct 9, 2015 at 1:10
  • 1
    @ryanyuyu I don't see how George having a mod diamond next to his name makes any difference in this case. If you have an issue with these posts not addressing the question posed, you should have brought the issue up with George as well, since he broke convention first. However, not a big deal in the end, I'm conferring with George about starting up a new discussion.
    – Bob
    Oct 9, 2015 at 1:19
  • 1
    You know what, I thought the same thing. I was even going to make a joke comment about incoming NAA flags when he first posted his answer. Then decided it was too stale of a joke.
    – BoltClock
    Oct 9, 2015 at 3:58
  • 2
    I don't know why the others down-voted, but I down-voted because I don't like the fact that a single answer being flagged for plagiarism puts every other answer by the same user into a queue. This does not seem to me like a very scale-able system. Imagine just one flag putting 1,000 answers into a review queue...
    – Cypher
    Oct 9, 2015 at 21:12
  • @Cypher I don't see that as being a big deal at all. The Close Review Queue currently contains 10.5k questions under review. There are plenty of high-rep users (10k+ and 25k+) available to review 1,000 answers. I honestly don't see that workload as a problem.
    – Bob
    Oct 9, 2015 at 23:20
  • 1
    @Bob: The reason why the close vote review queue on SO contains over 10k questions is because the rate at which new posts get thrown into that queue grew beyond the capability of reviewers to handle years ago, and the backlog has been growing ever since. After various attempts to address the issue ... Oct 11, 2015 at 13:20
  • 1
    ... such as encouraging users to review more, allowing more reviews per day per user, reducing the number of reviews needed, and even lying about the size of the queue to make it seem less daunting, it seems that SE has basically decided to give up on it, and just made unreviewed questions eventually age out of the queue. We don't even get topbar notifications about that queue any more, because the number in the topbar would be enormous, and would never go down. Oct 11, 2015 at 13:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .