The company has refused to honestly explain their plans to the community, but the data dump was late, and a former employee graciously did the community the favour of confirming that the company has discontinued the data dump on orders of senior management:
DISCLAIMER: I was recently impacted by the Company's layoff. I am going to carefully respond in a way that ensures I don't reveal anything the Company may feel is confidential--particularly with regard to strategy, or future plans. Any knowledge I have on strategy or future plans is both dated and confidential, and thus it would be irresponsible for me to say more. As a result, this answer may feel incomplete. I suspect that the CM team is rather busy this week with other topics. I'm offering what I can to uphold the Company's values of Transparency & being Community-centric.
The upload to the Internet Archive has been disabled.
The job that uploads the data dump to Archive.org was disabled in March, and marked to not be re-enabled without approval of senior leadership. Had it run as scheduled, it would have completed on the first Monday after the first Sunday in June.
I mention the timing, as this change long pre-dated the current moderator strike and related policy changes. Some comments have suggested otherwise, so I thought it an important detail.
Is it going to stay that way?
Hopefully the Company will provide an answer that includes this.
How can I access that data?
Stack Exchange Data Explorer (aka SEDE) contains a subset of all data for all sites, with PII removed. The same data available in the data dump is also available on SEDE.
SEDE is updated via a weekly full refresh (every weekend). The Data Dump that is uploaded to Archive.org is a dump of the SEDE databases. The weekly SEDE refresh runs, then the data is dumped to XML & 7zipped, then the 7z files are uploaded to the Archive.
SEDE can't address all the use cases of the Data Dump, nor vice versa. However, there is overlap, and the data is at least queryable.
Hopefully any questions around "why" or "what's next" will be addressed by an official response.
Original Question
Wired just published a new article, Stack Overflow Will Charge AI Giants for Training Data with some interesting new statements from CEO Prashanth Chandrasekar. I've quoted the pieces that seem most relevant to me below, but I encourage you to read the full article in case I've accidentally omitted any important context.
Stack Overflow, a popular internet forum for computer programming help, plans to begin charging large AI developers as soon as the middle of this year for access to the 50 million questions and answers on its service, CEO Prashanth Chandrasekar says.
“Community platforms that fuel LLMs absolutely should be compensated for their contributions so that companies like us can reinvest back into our communities to continue to make them thrive,” Stack Overflow’s Chandrasekar says. “We're very supportive of Reddit’s approach.”
Chandrasekar says proper licensing will only help accelerate development of high-quality LLMs.
They offer downloadable “data dumps” or real-time data portals to help software to access their content known as APIs. In Stack Overflow’s case, LLM developers are getting their hands on data through a mix of dumps, APIs, and scraping, Chandrasekar says, all of which today can be done for free.
But Chandrasekar says that LLM developers are violating Stack Overflow’s terms of service. Users own the content they post on Stack Overflow, as outlined in its TOS, but it all falls under a Creative Commons license that requires anyone later using the data to mention where it came from. When AI companies sell their models to customers, they “are unable to attribute each and every one of the community members whose questions and answers were used to train the model, thereby breaching the Creative Commons license,” Chandrasekar says.
Neither Stack Overflow nor Reddit has released pricing information.
Stack Overflow and Reddit will continue to license data for free to some people and companies. Chandrasekar says Stack Overflow only wants remuneration only from companies developing LLMs for big, commercial purposes. “When people start charging for products that are built on community-built sites like ours, that's where it's not fair use,” he says.
I understand that plans are preliminary and lots of things are up-in-the-air, but this is touching on some topics that the community can be very sensitive to. If possible, any clarification on what we can expect would be appreciated. In particular:
Is the company intending to change the licensing of user content again, and if so, would they attempt to apply those changes retroactively?
Prashanth's statement argues that the existing Creative Commons license would require direct attribution of any information sourced from Stack Overflow, but that AI models don't make it practical to do that. Does this imply that they're planning to be able to offer a different license arrangement to paying customers, which would in turn require contributors to agree to that new license?
Will the company be maintaining its commitment to (roughly-) quarterly data dumps, or are those at risk due to this situation?
This has been a pillar of the company's commitment to the community. Is the company planning to restrict it in some way, or are licensing and real-time-API restrictions sufficient for their objectives?
Again, I understand that matters are still at a preliminary stage and the company is probably unable to give specifics. But these are very sensitive subjects to a lot of us, and it would really be appreciated if you could help set expectations, before we have to find out what you've decided from the press.