If you're watching current trends, Stack Overflow is losing ground to LLMs and fast. I feel that if this continues, it will become hard to keep paying the bills and I hope it never happens, but if Stack Overflow has to close the books:
Will all this knowledge now be locked behind an LLM-paywall or behind the walls of the companies that have trained these LLMs as they probably have a copy of that data but we, the public, do not.
I was wondering if there is a project or a solution that is being worked on, to preserve this, as I really think, just like it is the case with research papers, this is invaluable, public domain, and definitely for the benefit of mankind. And it would be a shame and disaster if it were lost.
If nothing exists maybe we should start building a failsafe, but the robots.txt prohibits crawling the tree, so if a project was there perhaps the staff could allow its crawler some extra permissions on the site.
I tried finding a mirror for this site, or download all threads as an archive link on this site, and/or I tried finding how big everything would be as plaintext (but hyperlinked)-HTML or as HTML.
I looked on the Wayback Machine (archive.org) and while it does have many snapshots, many of them seem linked to the actual site and be gone as well, and apart from being really slow, no search is possible as they only have the frontend, not the backend of the site.