I've come across this [paper](http://papers.www2017.com.au.s3-website-ap-southeast-2.amazonaws.com/proceedings/p1221.pdf) from about a month ago titled *Detecting Duplicate Posts in Programming QA
Communities via Latent Semantics and Association Rules*. It features SO as one of its test subjects and claims to have improved the way to find duplicate Q&A.

From its abstract:

> Programming community-based question-answering (PCQA)
websites such as Stack Overflow enable programmers to find
working solutions to their questions. Despite detailed posting
guidelines, duplicate questions that have been answered
are frequently created. To tackle this problem, Stack Over-
flow provides a mechanism for reputable users to manually
mark duplicate questions. This is a laborious effort,
and leads to many duplicate questions remain undetected.
Existing duplicate detection methodologies from traditional
community based question-answering (CQA) websites are
difficult to be adopted directly to PCQA, as PCQA posts
often contain source code which is linguistically very different
from natural languages. In this paper, we propose
a methodology designed for the PCQA domain to detect
duplicate questions.

> ...

> Experiments
on a range of real-world datasets demonstrate
that our method works very well; in some cases over 30%
improvement compared to state-of-the-art benchmarks. As
a product of one of the proposed features, the association
score feature, we have mined a set of associated phrases
from duplicate questions on Stack Overflow and open the
dataset to the public.

I didn't read most of it yet, but considering the queue we have on Close Votes, the direct application of their method on Stack Exchange site and their claim for results, would it be worthwhile to look into and attempt to implement their method?