Looking at the rules I am not sure if the following question would be allowed.
I have gathered about 5.000 quotes. But some quotes are the same, just worded a little different.
For example:
Character is not something you were born with and can not change, like your fingerprints. It is something you were not born with and must take responsibility for forming. - Jim Rohn
and
Character isn't something you were born with and cannot change, like your fingerprints. It is something you weren't born with and must take responsibility for forming. - Jim Rohn
(Will not happen in my case, because I translate isn't to is not, but it is the idea.)
I would like to write some code to find quotes that resemble the quote to be added, so that if there are quotes found it is not inserted in the database until it is verified that it is not the same quite. Would I be allowed to ask for pointers to make an algorithm for this problem?
Edit
Sample inputs
The ‘problem’ is that this is not answerable. There are infinite possibilities I am afraid. But I would be happy to get a solution for X and would not mind not getting a solution for Y, even when I also asked for Y. Every pointer will help me on my way.
Expected outputs
I would want a list of all quotes that could match, because only after rejecting ‘all’ possible matches a new quote should be added.
Known invariants
I am only interested in the quote itself. A lot of quotes are wrongly accredited. For example the (shortened) quote:
We are like dwarfs on the shoulders of giants, so that we can see more than they, and things at a greater distance, not by virtue of any sharpness of sight on our part, or any physical distinction, but because we are carried high and raised up by their giant size.
Is often accredited to Isaac Newton, but he was quoting Bernard of Chartres.
Or a more common one:
The journey of a thousand miles must begin with a single step.
Is often attributed to Confusius, but is from Lao Tzu.
The quotes will be in several languages. It would be nice when an American English and a British English variant will match, but I expect this to be quite hard. If the same English version would be found, there would be already a big improvement.
Constraints
I think I will ‘solve’ it in Clojure. But I think that it would be a good idea to keep the question general. I think I will not be the only one that is going to implement it and when I see some C code that seems to work, I can translate it to Clojure (or the language I am going to use).
Use case
I want to run it for every quote that is proposed to be added. (Proposing can be done singly or in batches. I think it will mostly done in batches.) To minimize the change of duplicated quotes.
Probably it would be handy to do it also with already entered quotes. There will be doubles. But that is just a generalisation. Should not be a problem after I have the other variant.
I have already been toying with some ideas. (For example counting of words and if the counts ‘look alike’ comparing them manually.) But until know I found them lacking vastly.
By the way: I was not ready to ask it yet, but the response certainly will help me creating a question.