64 votes
Accepted

Will Stack Overflow disallow the GPT web crawler?

Yes, Stack Overflow has blocked OpenAI's web crawler, GPTBot from crawling the sites. The robots.txt file contains the following lines: User-agent: GPTBot Disallow: / which is the suggested way from ...
Andrew T.'s user avatar
  • 4,711
40 votes

Can I scrape Stack Overflow job postings?

You could use the RSS feed instead. From the linked answer: For jobs, our “API” is the RSS feed. You can get a feed of any search of jobs, look for the RSS icon below the search results, e.g. ...
Christian Gollhardt's user avatar
22 votes
Accepted

Can I scrape Stack Overflow job postings?

Technically we don't have a problem with scraping of /jobs for non-commercial purposes so long as your scraper acts like a responsible citizen. What does that mean? Respect throttling limits. If we ...
Dean Ward's user avatar
  • 4,793
8 votes
Accepted

Are there existing tools to help track a tag and gather statistics?

Some of this info can already be found in existing SEDE queries: Number of Questions per Time Period: Day: https://data.stackexchange.com/stackoverflow/queries?q=questions+asked+per+day Month: ...
TylerH's user avatar
  • 21.1k
1 vote

Why is `www.xszz.org` hosting exact copies of SO questions and answers

The content on SO is under a CC-BY-SA licence so other sites can host that content as well, if they comply to the requirement that they give proper attribution. There is nothing wrong or against the ...
rene's user avatar
  • 41.9k

Only top scored, non community-wiki answers of a minimum length are eligible