Am I allowed to scrape Stack Overflow job postings?
I love Stack Overflow and don't want to do anything that will get me in trouble.
Technically we don't have a problem with scraping of /jobs for non-commercial purposes so long as your scraper acts like a responsible citizen. What does that mean?
429 Too Many Requests
then back things offnofollow
, noindex
, robots.txt
. That includes in the rel
attribute on a
elements, the X-Robots-Tag
header, etc.I'd also highly recommend using the RSS feeds available on all search pages rather than manually scraping. It doesn't have as much data but it's unlikely to be changed out from under you when we make layout changes.
You could use the RSS feed instead. From the linked answer:
For jobs, our “API” is the RSS feed. You can get a feed of any search of jobs, look for the RSS icon below the search results, e.g. here:
Also take a look at API Terms of Use:
All Applications must ensure they visually indicate that the Stack Exchange Network is the source of the content provided through the API Services [...]
robots.txt
for StackOverflow doesn't specifically disallow the job posting pages