35

Plagiarism is a known problem on Stack Overflow. I do not mean to be hyperbolic and imply that this is a rampant problem; judging from the number of instances that I've been able to find, versus the number of users who use Stack Overflow (currently 4.6 million), my educated guess is that the number of plagiarisers at any given time is a very tiny percentage of the user base.

Regardless of the severity of plagiarism on Stack Overflow, it's a problem that does exist, to some degree or another, and probably always will. Given that, I was wondering if anyone has any ideas about how the Stack Exchange Data Explorer (SEDE) can be utilized to find potential instances of plagiarism?

Although I have provided some tools of my own, I'm interested in gathering improvements to these tools. My goal is to provide the community with better tools to help clean up plagiarism on the site. Some instances of plagiarism (non-exact-match answers) go undetected by my toolset, and manually searching Google for original sources is a time-consuming process.

For anyone interested in poking around SEDE, you can find the documentation at Database schema documentation for the public data dump and SEDE.

13
  • 2
    Factoring SEDE into the question gives this a feel of, well, begging the question. I would not expect any immediate answers which substantially differ from yours.
    – tripleee
    Commented Sep 16, 2015 at 6:36
  • 1
    Let me know if this really gets closed as a dupe so I can vote to reopen.
    – Pekka
    Commented Sep 16, 2015 at 7:26
  • @Pekka 웃: Maybe TZHX is saying OP should have edited the original question instead of posting new.
    – BoltClock Mod
    Commented Sep 16, 2015 at 7:32
  • @BoltClock true, good point - but now that it's here we could do it the other way round, too, and keep the upvoted version around.
    – Pekka
    Commented Sep 16, 2015 at 7:52
  • 1
    @Pekka웃 - I reversed the direction on this to point from the old post to this newer, better one.
    – Brad Larson Mod
    Commented Sep 16, 2015 at 15:13
  • 8
    I'm interested why you seem to care so much, you obviously put a lot of good work into this question and answer, but you've never asked or answered a question on main. Are you afraid to answer because you think your work will be stolen?
    – durron597
    Commented Sep 16, 2015 at 19:34
  • 14
    @durron597 I am irrelevant. Why should anyone care about plagiarism? Because plagiarism always sucks, and is harmful to everyone, including even the plagiariser, who will fail to grow by articulating and constructing their own thoughts and ideas.
    – Bob
    Commented Sep 17, 2015 at 1:22
  • 1
    Well said, @Bob. Just out of curiosity, are you a professor?
    – Brad Werth
    Commented Sep 17, 2015 at 6:22
  • 1
    @durron597 As a guess, sock puppet account? Commented Sep 17, 2015 at 20:44
  • 1
    Fighting the temptation to copy this word-for-word into a new Meta question. Commented Sep 17, 2015 at 21:03
  • 2
    Identifying plagiarism on trivial issues can and will often be nigh impossible because the correct answer can simply be one-to-one code-wise.
    – Etheryte
    Commented Sep 17, 2015 at 21:29
  • @Nit I agree. However, many cases of plagiarism on Stack Overflow are of non-trivial issues, and are comparatively less difficult to detect.
    – Bob
    Commented Sep 18, 2015 at 0:58
  • Although I have provided some tools of my own lol that must be the first time i have seen a question link to an answer already showing on the same page, by same person :) Come on everyone can see that first answer already. Don't take it seriously though :) Commented Sep 18, 2015 at 15:43

2 Answers 2

34

Finding Exact-Match Answers

There are two types of plagiarism that can be found on Stack Overflow: exact-match instances, and non-exact-match instances. Of the two types, exact-match plagiarism is relatively trivial to detect: it merely requires a simple equality comparison between two "strings" (or whatever text-field datatype is being used to store answers in a particular database).

When detecting exact-match instances of plagiarism on Stack Overflow, the "hard part" comes in efficiently surfacing these cases from the database, i.e. the Stack Exchange Data Explorer (SEDE).

A Naive Approach

Here is a naive query that can be run against SEDE to find exact-match answers,

SELECT  u.Id AS [User Link],
        p1.Score,
        FORMAT(p1.CreationDate, 'yyyy-MM-dd') AS [Date],
        p1.Id AS [Post Link],
        p2.Id AS [Post Link]
FROM Posts p1 INNER JOIN Posts p2 ON p1.Body = p2.Body -- SUBSTRING(p1.Body, 0, 100) = SUBSTRING(p2.Body, 0, 100)
              INNER JOIN Users u ON p1.OwnerUserId = u.Id
WHERE p1.PostTypeId = 2 AND p2.PostTypeId = 2 -- 2 is the post type for answers
  AND DATEDIFF(month, u.CreationDate, GETDATE()) <= ##MaxMonthAge:int?12##
  AND p1.Id <> p2.Id
  AND p1.OwnerUserId <> p2.OwnerUserId
  AND p1.CreationDate BETWEEN ##StartDate:string?2013-01-01## AND ##EndDate:string?2015-12-31##
  AND p2.CreationDate < p1.CreationDate
  AND p2.CreationDate <= ##EndDate:string?2015-12-31##
  AND ##MinLength:int?100## <= LEN(p1.Body)
ORDER BY  u.Id,
          p1.Score DESC,
          [Date]

The key part of this query is the self-join of the Posts table on itself, using the equality of two post bodies as the condition for the join,

FROM Posts p1 INNER JOIN Posts p2 ON p1.Body = p2.Body

Note that in this query, user u represents a potential plagiariser, p1 represents a potentially plagiarised answer, and p2 represents the (possible) original source. Thus, the following conditionals filter the set of self-joined post bodies down to those where

  1. The post bodies are an exact match.
  2. The posts are answers.
  3. The answer IDs aren't the same (don't include the join of an answer with itself).
  4. The answer owners are different.
  5. p1 was created after p2, i.e. p2.CreationDate < p1.CreationDate.

Expressed in SQL, these are

WHERE p1.PostTypeId = 2 AND p2.PostTypeId = 2 -- 2 is the post type for answers
  -- ...
  AND p1.Id <> p2.Id
  AND p1.OwnerUserId <> p2.OwnerUserId
  -- ...
  AND p2.CreationDate < p1.CreationDate

When running such a query against a data set as large as SEDE, I found it helpful to also add the following conditionals to limit the result set, making the query run faster, but at the cost of potentially missing some instances of plagiarism,

  1. For a user u, only return results if that user's account is less than X months old. The idea behind this is that "new" user accounts are more likely to engage in plagiarism than accounts belonging to more established users...though in some cases, I have found high-rep users with accounts that are years old, who have engaged in a disappointingly large number of instances of plagiarism.
  2. Limit potential positive p1 to answers made within a certain time frame.
  3. Limit potential source p2 to answers created before a certain date.
  4. Limit potential matches to the ones where the post bodies are at least X characters long, with the idea being that short answers are more likely to be false positives.

Expressed in SQL, these conditionals are

AND DATEDIFF(month, u.CreationDate, GETDATE()) <= ##MaxMonthAge:int?12##
-- ...
AND p1.CreationDate BETWEEN ##StartDate:string?2013-01-01## AND ##EndDate:string?2015-12-31##
-- ...
AND p2.CreationDate <= ##EndDate:string?2015-12-31##
AND ##MinLength:int?100## <= LEN(p1.Body)

Try It Out

You can test out the above query on SEDE. I have given the query some default parameters that will currently return results (about 140 of them, to be exact), though if moderators start removing true positives, then the result set may eventually be reduced to zero.

Feel free to play around with the parameters, or even make modifications to the query itself.

Finding Serial Plagiarisers

Using the tools and techniques that I have presented above (or using your own methods), you may come across a legitimate instance of plagiarism. I have often found it to be the case that when a user plagiarises once, they are likely to plagiarise many times. It can be an extremely time consuming process, but if you are feeling so inclined, you might decide to check a known plagiariser's other answers to see if they too are additional cases of plagiarism.

In the case of exact-match answers, this process can be automated somewhat by using a query such as the following,

DECLARE @lastEditDate datetime2 = (
  SELECT MAX(LastEditDate)
  FROM Posts
  WHERE OwnerUserId = ##UserId:int## AND PostTypeId = 2 -- answers
)

SELECT ROW_NUMBER() OVER (ORDER BY p1.Score DESC, p1.CreationDate) AS Rank,
       p1.Id AS [Post Link], p1.Score, DATEDIFF(day, p1.CreationDate, GETDATE()) AS [Days], p2.Id AS [Post Link]
FROM Posts p1 INNER JOIN Posts p2 ON p1.Body = p2.Body 
WHERE p1.PostTypeId = 2 AND p2.PostTypeId = 2 -- answers
  AND p1.Id <> p2.Id
  AND p1.OwnerUserId <> p2.OwnerUserId
  AND p1.OwnerUserId = ##UserId:int##
  AND p2.CreationDate < @lastEditDate -- don't check answers that are newer than the user's last edit date
ORDER BY p1.Score DESC, p1.CreationDate

This query returns the entire set of potentially plagiarised exact-match answers for a given user. Much like the very first query I presented, it does a self-join of the Posts table, using equal text-bodies as the join-condition. A key addition, however, is the condition that narrows the search results down to answers belonging to a specific user (the known plagiariser),

AND p1.OwnerUserId = ##UserId:int##

As I mentioned, this query only returns exact-match cases of plagiarism. If a known plagiariser (also) uses non-exact-match answers, then you might just have to find original sources manually by using Google and SO's search engine, as I have mentioned.

Can You Improve These Queries (or Make Better Ones)?

There are many areas of improvement to the SQL queries that I have presented, such as making them run more efficiently, or returning a more complete and exhaustive set of results. One interesting enhancement might be to modify the queries to search the revision history for answers, in case an original source and a copy "drift" away in similarity as a plagiariser (or well meaning SO users) edit them over time.

Another simple enhancement would be to strip HTML and Markdown formatting from the post bodies before comparing them. Someone also once suggested using the Levenshtein distance between posts for comparison.

Also, I am not a source control expert, and I would be interested to know if any text comparison techniques used in version control can be applied to this problem. I've also heard about tools that people working in academia use to determine whether a student's paper has been plagiarised from some other source, and I would be interested in hearing about whether these tools can detect non-exact-matches, and about whether their algorithms can be applied to this problem.

Update: Calculating Levenshtein Distance and SEDE

I read up on how Levenshtein Distance is calculated, and it seems too computationally expensive to implement in a SEDE query. I might just make a GitHub project in .NET or Ruby someday that implements this instead (perhaps with an additional public-facing web interface that people can use).

In the meantime, here's a pseudo-code description of the algorithm, along with my notes, derived from the English description in Levenshtein Distance, in Three Flavors. Note that my pseudo-code style is a mish-mash of mathematical notations and syntaxes borrowed from a variety of languages, e.g. Haskell, Ruby, Java/C# etc.

/*
 * - Each iteration of the `for` loops computes the Levenshtein Distance for
 *   the sub-matrix that begins at cell M(1,1) and terminates at cell M(i,j).
 * - Addition and deletion are inverses of each other, and modify string length.
 * - Substitution can only occur along diagonals, i.e. iff index i == j, and
 *   does *not* modify string length.
 */
EditDistance(string a, string b) -> int
{
    int n = a.length
    int m = b.length
    return m if n == 0 // `a` is empty string
    return n if m == 0 // `b` is empty string
    
    M = new Matrix(n + 1, m + 1)
    for int i from 0 to n {
        M(i, 0) = i
    }
    for int j from 0 to m {
        M(0, j) = j
    }

    int cost = 0
    for int i from 1 to n {
        for int j from 1 to m {
            cost = 0 if a(i - 1) == b(j - 1) else 1
            M(i,j) = min(M(i - 1, j) + 1, M(i, j - 1) + 1, M(i - 1, j - 1) + cost)
        }
    }
    
    return M(n,m)
}

Update: Have My Edits Been Rolled-Back?

Occasionally, when you're searching for plagiarism, you'll come across answers that technically cite the source(s) that they're copied from, but do so poorly. For example, by not using quote-formatting, and "hiding" the source link at the very bottom of the answer, in non-descriptive anchor text like "SEE THIS" or "LINK".

Such answers come across as a little dishonest because they're not very forthright about the fact that they're copied from another source. Indeed, I have actually mistaken such answers for plagiarism at first glance.

In such cases, you may want to help the answer author improve his/her citation of the copied source through editing. Even in the case of actual plagiarism, you may want simply cite the source for the author, rather than flag him/her for plagiarism.

It has been my experience, however, that sometimes the owners of these plagiarised and almost-plagiarised answers will later undo such edits. In order to make it easier to determine if such a rollback has occurred, I have written another SEDE query that takes your Stack Overflow user-id, and returns all of your edits (made in the last X days) that have been replaced by subsequent edits. Enjoy!

SELECT ph.Id, ph.PostId, ph.CreationDate, ph.Comment
INTO #Edits
FROM PostHistory ph
WHERE UserId = ##UserId:int##
  AND ph.PostHistoryTypeId IN (5,8) -- Edit, or Rollback
  AND DATEDIFF(day, ph.CreationDate, GETDATE()) <= ##MaxDayAge:int?14##
  
SELECT ph.PostId AS [Post Link], ph.CreationDate
FROM PostHistory ph INNER JOIN #Edits e ON e.PostId = ph.PostId
WHERE ph.PostHistoryTypeId IN (5,8) -- Edit, or Rollback
  AND e.CreationDate < ph.CreationDate -- new revision
ORDER BY ph.CreationDate DESC

A note on answers with poor citation

When you see an answer that poorly cites a copied source, you may in fact want to check the revision history to see if in fact it was originally plagiarised. It has also been my experience that sometimes someone other than the answer poster will come along and cite the plagiarised source for the plagiariser. If you find an answer like that, you may want to investigate the author's other answers for signs of more plagiarism.

Update: New Detection Query, Droid Hunter

After finding additional cases of plagiarism over the weekend, I came to the realization that plagiarism often occurs in the [android] tag. Apparently, Android developers are among the most frequent plagiarists on Stack Overflow (sorry guys, the numbers don't lie).

So I've created a new query, "Droid Hunter", that searches all answers to [android] questions that are exact-match duplicates of other answers,

SELECT a.*
INTO #AndroidAnswers
FROM Posts q INNER JOIN Posts a ON a.ParentId = q.Id
             INNER JOIN PostTags pt ON pt.PostId = q.Id
             INNER JOIN Tags t ON t.Id = pt.TagId
WHERE t.Id = 1386 -- Android
  AND a.PostTypeId = 2  -- answer
  AND 100 < LEN(a.Body) -- reduce false positives

SELECT -- TOP ##Limit:int?100##
  ROW_NUMBER() OVER (ORDER BY a.OwnerUserId, a.Score DESC, a.CreationDate) AS Rank,
  a.OwnerUserId AS [User Link],
  FORMAT(u.Reputation, '#,###') AS Reputation,
  a.Id AS [Post Link],
  p.Id AS [Post Link],
  a.Score,
  DATEDIFF(day, a.CreationDate, GETDATE()) AS [Days]
FROM Posts p
  INNER JOIN #AndroidAnswers a ON p.Body = a.Body
  INNER JOIN Users u ON a.OwnerUserId = u.Id
WHERE p.PostTypeId = 2 -- answers
  AND p.Id <> a.Id
  AND p.OwnerUserId <> a. OwnerUserId
  AND p.CreationDate < a.CreationDate
ORDER BY a.OwnerUserId, a.Score DESC, a.CreationDate

The query currently returns 62 possible cases of plagiarism.

12
  • 2
    For exact matches, compare hashes. When a question or answer is posted, take the body of the post and run it through a hash function (md5 would be sufficient), and check if it already exists. These hashes are stored in a database along with other relevant information on the post and can be searched efficiently. Should the hash already exist in the database, you have a perfect match and the post should not be allowed.
    – Tro
    Commented Sep 17, 2015 at 8:35
  • @Tro that is a very interesting idea, and seems like it could definitely help make querying the database more efficient. However, I want to focus on solutions that make use of SEDE's API, because only SO's devs are able to make changes to their database (unless anyone wants to actually populate their own private database with SEDE data...). If the SO devs ever become interested in building plagiarism-detection tools, however, then they definitely might be interested in using your hash idea.
    – Bob
    Commented Sep 17, 2015 at 8:51
  • Try comparing post bodies by stripping away excess characters at the start or end. That should account for copied answers plus a header or footer. Compare only the smallest length of both posts. Or, compare a fixed number of chars such as 500. If that many are copied it's bad.
    – usr
    Commented Sep 17, 2015 at 20:51
  • Consider adding a parameter to have answers posted within, say 10 minutes of each other excluded. While good answers are unlikely to be identical, code-only ones can be, and for non-exact matches 2 people can say the same thing independently in such a time frame. Commented Sep 17, 2015 at 21:12
  • If you only search for answers posted to the same question, you can throw out most of the limiting stuff and it won't timeout. data.stackexchange.com/stackoverflow/query/365689/… 71 results at the time of posting.
    – user4639281
    Commented Sep 17, 2015 at 23:29
  • @TinyGiant thanks for the idea. However, I'm afraid that your modification isn't very useful, because it returns results that have an extremely high chance of being false positives. The answers returned are for the same question, so of course there's an extremely high chance that two users arrive at the same answer independently.
    – Bob
    Commented Sep 18, 2015 at 0:59
  • @TinyGiant In addition to what Bob said, plagiarism of an answer on the same question is much easier to spot anyway since it's right there. Commented Sep 18, 2015 at 8:44
  • @usr the problem with stripping away characters from the beginning and end of a post is that sometimes a plagiariser removes content, rather than add it. Stripping away characters seems like it will only work for exact matches, and in those cases, we haven't really gained anything by doing so. In the case of non-exact matches, without knowing beforehand if answer B uses non-attributed content from answer A, how will we know how many characters are necessary to remove, and from where in the answer?
    – Bob
    Commented Sep 18, 2015 at 9:21
  • 2
    Truncate the longer string to the size of the shorter. That's what I meant. This will strictly result in more matches than before. It's a primitive improvement over exact match.; I once had to solve a similar problem (finding duplicate articles). What I did was try different truncations and use Levenshtein distance. That detects most duplicates. For finding plagiarized answers I would add something to match up paragraphs as well.; But really my point was to suggest a super easy to make improvement that can be done in SQL without going insane.
    – usr
    Commented Sep 18, 2015 at 9:28
  • @usr ah, I understand what you mean now. That definitely sounds like a simple yet effective improvement, I'll keep it in mind when I make new versions of my queries.
    – Bob
    Commented Sep 18, 2015 at 9:45
  • @usr Calculating Levenshtein distance in SEDE would take a lot of time, with the timeout it would probably fail from that before returning one result. Now if you download the datadump then you can run whatever queries you want on the data from your computer.
    – user4639281
    Commented Sep 18, 2015 at 14:39
  • Too bad this query seems to slow now as the data has grown. I would like to have seen how effective it was. Using post length seems to time out a lot of queries. Also, 7 years later, it turns out your observation about [android] plagiarism was way more spot-on than you may have realized. In the past few weeks, I've raised hundreds of flags on answers under that tag alone, and most of them were submitted pre-2015. There are likely hundreds, if not thousands, more at the rate I'm finding them. It's a little ridiculous. Commented Aug 31, 2022 at 19:33
7

this q&a are more about playing with SEDE and not realy about plagiarism.

  1. Any unique entry may be plagarised from out of SO.
  2. Two identical entries could be made by different people who have not read each other. For example two identical answers to two different question.
  3. An entry can plagiarism two or more sources
  4. An entry can be 95% duplication and 5% attribution and thus be legit
  5. A person cannot plagiarise themselves

Anyow to be of any real use for plagaris detection it needs to identify for each entry its bag of longest "rare" substring. The bag needs to adjusted for attribution. Then the bag needs to be compared to other entries' bags and to their incidence on other pages on the web as well. Next you need to establish authorship and date - not a trivial task. Sinaly you need to rank the results.

This is well beyond the scope of what can be done by SQL anyhow, even if you only are looking within SEDE.

5
  • Regarding your #2, it is certainly possible for two people to independently arrive at the same solution, especially when their solutions involve short snippets of code. However, when someone's multi-paragraph answer is identical to someone else's, then it is near-indisputable that someone has plagiarised off of someone else.
    – Bob
    Commented Sep 17, 2015 at 8:56
  • Regarding #3, yes, it is certainly possible for someone to plagiarise more than two sources in an answer. This is difficult to detect using exact-match techniques, and is one of the problems that I would be interested in seeing a practical solution to. Manually googling for the original sources is possible in these cases, but it is also time-consuming.
    – Bob
    Commented Sep 17, 2015 at 8:58
  • 1
    Regarding #4, as long as proper attribution is given, then yes, an answer would be "legitimate", in the sense that it does not try to pass off ideas as belonging to person B when in fact those ideas originated from person A. However, I'm interested in detecting cases of actual plagiarism, where no attribution has been given, at all. These sorts of answers do exist on SO, unfortunately.
    – Bob
    Commented Sep 17, 2015 at 9:01
  • Finally, it is not impossible to detect plagiarism on SO using SEDE. Indeed, I have given an exhaustive introduction to the subject, and you can even see some preliminary results yourself.
    – Bob
    Commented Sep 17, 2015 at 9:04
  • Regarding #5, self-plagarism is real concept (questioned by some), but usually on Stack Overflow it means that the questions which contain self-plagiarised answers are duplicates. Commented Sep 18, 2015 at 17:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .