58

In a forthcoming article in the New York Times (non-paywalled link), Jon Gertner explores the future of Wikipedia in the times of generative artificial intelligence. Near the end of the article, Gertner discusses a preprint "The curse of recursion: training on generated data makes models forget" introducing the concept of "model collapse" that occurs when training future generations of LLMs with content generated by previous ones. One of the conclusions of the paper is that "To make sure that learning is sustained over a long time period, one needs to make sure that access to the original data source is preserved and that additional data not generated by LLMs remain available over time."

My question is how Stack Overflow can adapt to preserve value in this environment. Is there a way to incentivize human-written content? Are there new ways of documenting authorship (e.g. a short video talking about what you just posted) that will replace a simple signature under a hand-written document. Is there a way to document the provenance of contributions? Is there a way to disincentivize humans posting questions and/or answers generated by A.I. just to get imaginary reputation points?

How can we continue to motivate knowledgeable humans to make voluntary contributions to a repository not tainted by A.I. regurgitation? The answer to this question ensures the future of Stack Overflow and other repositories of knowledge and, as a bonus, would prevent model collapse.

15
  • 73
    Supposedly, by deleting AI-generated content actively...?
    – Andrew T.
    Commented Jul 18, 2023 at 14:00
  • 18
    "How can we continue …" as far as I can tell, the goal of SE Inc is the exact opposite so we might need to look at "reverse" instead of "continue". Commented Jul 18, 2023 at 14:01
  • 1
    Edit history exists both on Wikipedia and Stack Exchange sites for a reason -- even back then it was obvious that collaborative efforts might be subverted sometime, someway. As long as edit history remains, we can always filter any change made after late 2022 (or nuke them from orbit) as a last resort. Commented Jul 18, 2023 at 14:20
  • 1
    Just put the whole site into read only mode or simply remove further reputation gain /s :^)
    – Lino
    Commented Jul 18, 2023 at 15:18
  • 28
    Just wanted to add that this (like a lot of things with AI lately) isn't a new problem; Wikipedia already had to deal with the issue of recursive sources prior to AI, the only new thing here is the speed at which the problem can occur. The real danger here is that AI-supplanted sources regurgitate each other so quickly that it'll be hard to sort out what's actually true and where it came from... the internet already had to deal with flawed, human researchers doing and publishing imperfect research citing other flawed, imperfect research, so that part, or dealing with it, isn't really new.
    – zcoop98
    Commented Jul 18, 2023 at 16:01
  • 4
    How about people training AI models do their own data curation? Oh wait, they are, but they pay exploitative wages and have terrible work environments. theverge.com/features/23764584/… I cannot be made to care about creating free clean content for someone else to train AI with.
    – ColleenV
    Commented Jul 18, 2023 at 19:13
  • 4
    With the amount of garbage questions and answers added to SO over the past decade by human contributors, which doesn't get adequately cleaned up anymore because cleanup is not "welcoming", I don't think we need to worry about reducing the suitability of SO content as a training data due to AI-generated posts. All the human-created garbage should have a significantly higher impact...
    – l4mpi
    Commented Jul 19, 2023 at 14:48
  • Humans will have to learn the language the AIs create rather than the other way around. Commented Jul 20, 2023 at 11:05
  • @AndrewT. That would make sense, but the moderators were complaining that Stack Overflow executives have forbidden them from deleting AI-generated content. Commented Jul 20, 2023 at 11:06
  • 5
    @user253751 To clarify, forbidden to suspend users because they are allegedly (and obviously) posting AI-generated content. Ideally, posts with bad content from AI would be treated the same way as bad content written by humans. The practical problem is that AI is so much faster in generating bad posts compared to humans painstakingly writing bad posts, and the AI contributions are better at sounding good while being bad. Commented Jul 20, 2023 at 11:59
  • Another interesting question is what will happen once all the hype ends? Right now it's all on the rise, but will it withstand the test of time?
    – Alejandro
    Commented Jul 20, 2023 at 14:32
  • 1
    The only thing I can think of is go back to snail mail and pay people at StackOverflow to type it in. But that's just silly! Commented Jul 20, 2023 at 16:13
  • "disincentivize humans" no rep? Not sure if this helps increasing the amount of human written content. Commented Jul 20, 2023 at 20:59
  • 1
    @zcoop98 "ll be hard to sort out what's actually true" Isn't voting supposed to give that information on the exchanges? At least in programming it should be still doable to run that code and check the truth. Commented Jul 20, 2023 at 22:24
  • 2
    Maybe I'm dumb, but I'm not following the question. Why should I care about model collapse, as a Stack Overflow user? Generative AI is already banned here.
    – wjandrea
    Commented Jul 21, 2023 at 2:17

6 Answers 6

64

Wait, why do I care if AI's models collapse on themselves from reading Stack Overflow content?

It's honestly the best thing that could happen. In spite of it being banned for now, someone posts an AI-generated answer here on Stack Overflow for clout (and yes, this is a very real thing). Depending on how many people do this and for how long it perpetuates, the value of the data here is poisoned but not to an untrained eye. Then, someone trains a model on it, and the model is worthless.

I say: "Mission freaking accomplished."

The problem isn't that AI models are training off of AI-generated data. It's that people want clout and want to pass off the results of something AI-generated as their own. You'll fix that problem around the same time you solve plagiarism.

3
  • 10
    This supposes that SE Inc. won't just automate posting AI-generated answers. Commented Jul 18, 2023 at 21:29
  • 2
    Just like the solutions to so many problems, the solution to literally every AI problem is to have no AI
    – Ja Da
    Commented Jul 20, 2023 at 11:04
  • "Wait, why do I care if AI's models collapse on themselves" Let's call it professional curiosity. It's like watching children trying to be their own teachers about subjects they don't understand and failing.
    – Clockwork
    Commented Jul 20, 2023 at 14:56
50

Curate like a madman. Tell your friends to curate first, answer second, and ask third. If someone complains about curation, educate them until you are out of bubblegum and then educate them some more. Be vocal about wanting expert answers, not just any answers. Act like you want expert answers, not just any answers.
Edit mediocre stuff. Downvote bad stuff. Close vote off-topic stuff. Flag, vote, flag, vote, then buy more bubblegum and repeat.

The main problem about AI content is that people do not see a problem with inaction in the face of mediocre content, off-topic content, and oftentimes even bad content. AI content is posted because content like it is tolerated, accepted and oftentimes even praised.
Don't want that? Then do not praise such content. Do not accept such content. Do not tolerate such content.

Optimise for pearls, not sand. Hard.

8
  • 1
    Couldn't have said it better. :)
    – Dan Mašek
    Commented Jul 18, 2023 at 20:26
  • 12
    And comprehensive answers, not just "Try this" answers (code dumps) Commented Jul 18, 2023 at 20:33
  • I feel like optimizing for AI indigestible pearls.
    – Joshua
    Commented Jul 18, 2023 at 22:18
  • @PeterMortensen ... and tested code, not just "one liners"
    – MT1
    Commented Jul 19, 2023 at 10:22
  • 6
    Not that I disagree with this answer, but I fear SO the company does. They don't want you to curate and educate or care about quality, they would prefer that you simply answer everything, as that's better for growth (eyes on ads). SO discarded "optimise for pearls" ages ago during the "welcoming" saga. In the python+java tags, I could easily spend all my daily DVs in an hour if I cared... which I don't anymore, as I don't see any value in working towards a different goal than the company. So IMO a better answer would be, stop curating, let this site burn, build something better from the ashes.
    – l4mpi
    Commented Jul 19, 2023 at 15:11
  • 2
    @l4mpi Well, as far as I can tell "let others learn from your history of failure" has a longstanding history of failure. Commented Jul 19, 2023 at 16:23
  • 1
    Stack Overflow executives said no, because AI is a profit center for them. They don't want you to "answer everything" but they do want you to come here to find answers, and that means having things that vaguely look like answers, which can be generated by AI because it's less fussy than actual users. That's the real model collapse. Commented Jul 20, 2023 at 11:07
  • 2
    Take your last big paragraph, replace "AI content" with "content generated by non-SME to gather Internet bonus points" and it would have described 2020ish with peopel joining SO as "joungones" to copy and paste blatant copies of ever the same solutions (already found on SO) to ever the same questions. I feel SO has been on a decline for a decade (prob. longer, not that active before that) - AI just makes the slope more slopier and more slippery drowning the pearls in ever so more grains of sand. Commented Jul 21, 2023 at 10:41
25

My question is how Stack Overflow can adapt to preserve value in this environment. Is there a way to incentivize human-written content?

AI-written content is banned on SO.

Are there new ways of documenting authorship (e.g. a short video talking about what you just posted or - more futuristic - a brain scan demonstrating that you actually thought about what you posted) that will replace a simple signature under a hand-written document.

That's... Science fiction.

Is there a way to document the provenance of contributions?

No, not with any measure of accuracy (plagiarism aside).

Is there a way to disincentivize humans posting questions generated by A.I. and answers generated by A.I. to get imaginary reputation points.

Yeap, remove their contributions and apply suspensions or even full-out bans on repeat offenders.


Now, this whole "model collapse" isn't an SE problem. It's literally of no concern to the site, as the site does not run on these models.

22
  • 3
    True, the site does not run on these models. However, these models run on this site. Commented Jul 18, 2023 at 14:03
  • 33
    That's a "them" problem. Not our problem, @KarstenTheis. We're not responsible for keeping their data source free of data they do not want.
    – Cerbrus
    Commented Jul 18, 2023 at 14:04
  • 2
    "True, the site does not run on these models" It does, just the mods aren't allowed to enforce these rules and as a result (among other reasons) are on strike. Prior to the strike users that posted AI generated content had that content mod deleted, and repeat offenders were suspended for at least a week.
    – Thom A
    Commented Jul 18, 2023 at 14:04
  • @Cerbrus As a human, I am affected by "them" problems as well. The Times article has a nice bit about the interdependence of Wikipedia and Google, both in terms of content and in terms of finances. Commented Jul 18, 2023 at 14:11
  • 26
    How you are affected by AI companies getting poor data is not an SO problem.
    – Cerbrus
    Commented Jul 18, 2023 at 14:15
  • 5
    Based on the capability of the LLM today we are likely at least a decade from having a LLM that determine the actual context of words and determine if they are actually relevant or not when generating a response to input. The LLM today do not have that capability, that is the reason, they talk about absolutely random topics of a question even mentions a word that’s relevant to the question but not the actual response to that question. This fatal flaw makes it absolutely trivial to determine what’s generated by LLM. Commented Jul 18, 2023 at 14:50
  • 25
    Given that the company wants to sell the Q&A data to companies creating LLMs, model collapse and how it relates to having AI-generated content on the platform is a problem which SO, the company, should be seriously considering. Given that model collapse is a real issue for LLM creation, a dataset that has a lower volume of AI-generated content has more value, which means that the dataset can be sold to more companies at a higher price. If the dataset has a high volume of AI-generated content, then it's worth less, or even has negative value, due to the expense necessary to clean it.
    – Makyen Mod
    Commented Jul 18, 2023 at 16:11
  • 26
    In other words, if SO, the company, really is wanting to sell the Q&A dataset, then they should be strongly encouraging the ban on AI-generated content and the measures by the moderators and community to remove AI-generated content, because having moderators and the community, who the company doesn't need to pay, perform the hard work of identifying and removing AI-generated content significantly increases the value of what the company wants to sell without costing the company money to do it themselves.
    – Makyen Mod
    Commented Jul 18, 2023 at 16:22
  • 1
    @Makyen my impression is that SO the Company both wants to sell the Q&A dataset AND also attract tons of new low-effort users. They want to have and eat their cake, reality be damned
    – vbnet3d
    Commented Jul 19, 2023 at 14:34
  • 2
    @Makyen would it make sense for a LLM company to buy the Q&A data? Given the CC BY-SA license, they don't need to. I really see no incentive for the company to attract low-quality content other than trying to increase the user-base or number of pages/ads displayed (which might be a "good" short term strategy, but a terrible one on the longer term).
    – mozway
    Commented Jul 20, 2023 at 10:53
  • 1
    @mozway That will be the next step: turning off the free access to data. Twitter did it. Reddit did it. Stack Exchange will do it too. The money is just too irresistible and that's why we have to abolish capitalism. Commented Jul 20, 2023 at 11:09
  • 1
    "and that's why we have to abolish capitalism." Hahahahahahahah yeah, not gonna happen.
    – Cerbrus
    Commented Jul 20, 2023 at 11:44
  • 1
    @mozway If it makes sense for the LMM company to purchase the content is, really, a business and legal matter which the companies involved would need to resolve. To a large extent it depends on if machine learning models, specifically LLMs, are legally considered to be derivative works of their training data. If they are derivative works, then the creators and model must comply with the individual licenses from each and every work which is in the training dataset from which the model was created, which, obviously, would include the CC BY-SA licenses for SE/SO content obtained for free.
    – Makyen Mod
    Commented Jul 20, 2023 at 15:44
  • 1
    Overall, if a LLM is a derivative work is a large legal question, which will need to be settled over time in basically every jurisdiction. It will take at least years, if not decades, to be resolved and will be the result of either or both of political action and/or many expensive lawsuits. Being able to have the SE/SO data without the encumbrance of the CC BY-SA licenses could have value to an LLM company, even prior to LLMs being derivative works being resolved, as it would reduce what they could be sued over and who could sue them.
    – Makyen Mod
    Commented Jul 20, 2023 at 15:44
  • 1
    @mozway The existing Terms of Service (TOS) results in users granting two licenses. The license we all are quite aware of is the CC BY-SA license (of various different versions, depending on date). That's the license for user content distributed on SE/SO sites, SEDE, and the data dumps. There's another license which users grant to SO, the company, which doesn't have the same restrictions, but does have some other restrictions, depending on the TOS version. What a LLM company would, presumably, be getting by paying is a license to use a copy of the user data under that other license.
    – Makyen Mod
    Commented Jul 20, 2023 at 18:00
7

I would like to share this other pre-print: Self-Consuming Generative Models Go MAD (PDF)

In this paper, image generating AIs are used and self-fed with their content over several generations.

Two figures illustrate well two potential threats that we are facing by allowing AI-generated on our platforms.

loss of quality

Figure 1: Training generative artificial intelligence (AI) models on synthetic data progressively amplifies artifacts.

enter image description here


loss of diversity

Over time, self-feeding can decrease the diversity of the generated content:

Figure 23: Generation t = 1 of a fully synthetic loop with bias λ = 0.7

enter image description here

Figure 25: Generation t = 5 of a fully synthetic loop with bias λ = 0.7

enter image description here


In contrast to images, most of us already think that AI-generated code is of poor quality. I would thus expect self-feeding to have a dramatic effect faster than for image generating AIs.

I believe Stack Overflow should do everything to ban AI-generated content as normal Q/A on this platform. If there is an interest in including AI features, this should eventually be done automatically for each Q&A in a transparent way, but certainly not trough random users in a way that could be mistaken for human generated content.

4
  • I don't know exactly what the number means but you mentioned that "bias λ = 0.7" for the last two figures (repeated the same number). Was that a typo?
    – 41686d6564
    Commented Jul 21, 2023 at 1:03
  • @41686d6564standsw.Palestine no, what matters is that it's different generations for the same bias. If you want to have the details, read the paper. My goal was not to focus on the details, I just wanted to share an excerpt of it that I thought was quite illustrative of the risks of self feeding AIs and why it should be an incentive for the company not to allow posting of AI-generated content. In the end I don't personally care about a potential model crash, but I do care that SO doesn't get polluted in the process.
    – mozway
    Commented Jul 21, 2023 at 5:13
  • 5
    So, in a few generations, all AI code will be "hello world" xD
    – Cerbrus
    Commented Jul 21, 2023 at 6:43
  • 5
    more likely "hell worst" @Cerbrus :)
    – gnat
    Commented Jul 21, 2023 at 11:36
1

How can we continue to motivate knowledgeable humans to make voluntary contributions to a repository not tainted by A.I. regurgitation?

I don't think that a model collapse is imminent because we still have curation and especially voting on Stack Overflow (AIs could for example only learn on positively scored input) and because AI content pollution is probably still only a minority of all created content.

But if you assume that AI is not going to stay fully excluded on Stack Overflow, one solution might be to clearly mark AI content as such, for example by auto-generating it for every question and simply displaying it with a clear description somewhere on the page, maybe with an additional comment thread.

If every AI content is marked as such, it can be filtered out in subsequent learning runs. And we humans can better delineate us from AI content and work with it at the same time. There would be less incentive to recreate AI content.

I wonder if for example ChatGPT keeps a log of all its output and filters out all similar elements in future input.

Summary: If we cannot effectively ban AI content forever at the very minimum we must ensure that it is clearly marked as such for our sake and for the sake of AI.

3
  • 2
    I agree, that what I tried to suggest as a conclusion in my answer. I'd rather have no AI content, but if this is inevitable then let's have one automatically generated AI answer (ideally with an option to hide it) and strictly ban users from inputing such content.
    – mozway
    Commented Jul 21, 2023 at 6:08
  • 4
    Auto-generating answers and why that's a bad idea for a plethora of reasons has been discussed ad nauseam in the LLM ban announcement.
    – Cerbrus
    Commented Jul 21, 2023 at 6:41
  • 2
    "I wonder if for example ChatGPT keeps a log of all its output and filters out all similar elements in future input." It doesn't.
    – Cerbrus
    Commented Jul 21, 2023 at 6:48
-17

Upvote good human-written content because it's good, not because it's human-written. Ignore bad AI-written content because it's bad, not because it's AI-written. Anything else is a genetic fallacy.

9
  • 10
    Why ignore bad content? Commented Jul 21, 2023 at 18:48
  • @MisterMiyagi Because one man's garbage is another man's treasure. An enormous amount of valuable information has been removed from this site because it didn't fit some pedant's criteria for good content. Commented Jul 21, 2023 at 20:50
  • @MisterMiyagi Because negative curation appeals to the same kind of mental defect as planned economies. It reflects a profound inability to grasp emergent phenomena. Commented Jul 21, 2023 at 20:54
  • 1
    I did not know the technical term: "The genetic fallacy is a fallacy of irrelevance in which arguments or information are dismissed or validated based solely on their source of origin rather than their content. In other words, a claim is ignored or given credibility based on its source rather than the claim itself". Wikipedia Commented Jul 21, 2023 at 22:30
  • 10
    This particular dead horse has been beaten to a mushy pulp repeatedly on this site. The reasons AI-generated content is problematic is adequately explained in the still-valid official policy.
    – tripleee
    Commented Jul 22, 2023 at 7:51
  • What does "genetic" refer to? This? - "The genetic method is a method of teaching mathematics coined by Otto Toeplitz in 1927. As an alternative to the axiomatic system, the method suggests using history of mathematics to deliver excitement and motivation and engage the class." Can you elaborate? Commented Jul 22, 2023 at 9:24
  • @PeterMortensen en.wikipedia.org/wiki/Genetic_fallacy It's the category of fallacies that includes ad hominem. Commented Jul 22, 2023 at 16:31
  • 3
    "Ignore bad AI-written content because it's bad" And since when are we not allowed to downvote bad content?
    – Cerbrus
    Commented Jul 24, 2023 at 7:09
  • It's always been allowed, and never been a good idea. Commented Jul 25, 2023 at 0:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .