I just want to point out this recent paper on arXiv: Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow, by Maria del Rio-Chanona, Nadzeya Laurentsyeva, Johannes Wachs, submitted 14 Jul 2023.
Here's its abstract:
Large language models like ChatGPT efficiently provide users with information about various topics, presenting a potential substitute for searching the web and asking people for help online. But since users interact privately with the model, these models may drastically reduce the amount of publicly available human-generated data and knowledge resources. This substitution can present a significant problem in securing training data for future models. In this work, we investigate how the release of ChatGPT changed human-generated open data on the web by analyzing the activity on Stack Overflow, the leading online Q&A platform for computer programming. We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, and to similar forums for mathematics, where ChatGPT is less capable, activity on Stack Overflow significantly decreased. A difference-in-differences model estimates a 16% decrease in weekly posts on Stack Overflow. This effect increases in magnitude over time, and is larger for posts related to the most widely used programming languages. Posts made after ChatGPT get similar voting scores than before, suggesting that ChatGPT is not merely displacing duplicate or low-quality content. These results suggest that more users are adopting large language models to answer questions and they are better substitutes for Stack Overflow for languages for which they have more training data. Using models like ChatGPT may be more efficient for solving certain programming problems, but its widespread adoption and the resulting shift away from public exchange on the web will limit the open data people and models can learn from in the future.
I'm fairly sure the users here would be highly interested in it. Perhaps you'd like to share your thoughts on it.
I gave it a quick read, and these observations seem worth highlighting here:
The authors observe a substantial drop in Stack Overflow posts after the launch of ChatGPT, which they didn't find at other sites (e.g. math.SE):
They estimate "a 16% decrease in weekly posts on Stack Overflow" which subsequently worsened to 25%:
By the end of April 2023, the estimated effect stabilizes at around 25%. Interestingly, ChatGPT use, in general, peaked around this time.
and observe that some tags typescript, python, and r were more affected than others (the authors also mention JavaScript).
Regarding voting, they write:
Indeed the relative stability of voting behavior suggests that the quality of posts on Stack Overflow has not meaningfully changed after the introduction of ChatGPT.
One main discussion point is that GenAI-written posts make publicly available Q&A less useful for training GenAI. Moreover, while private use of GenAI (such as users asking GenAI the questions directly instead of asking on Stack Overflow) will offset this GenAI-training problem, this will allow major GenAIs to monopolize knowledge that would otherwise be freely available.