59

Am I allowed to scrape Stack Overflow job postings?

I love Stack Overflow and don't want to do anything that will get me in trouble.

7
  • 18
    Have you checked the terms of service? I believe that outside of API access, downloading is permitted only for personal use.
    – Martijn Pieters Mod
    May 9, 2017 at 12:20
  • 52
    Hey, grats for actually asking rather than going ahead anyway!
    – Moo-Juice
    May 9, 2017 at 18:29
  • 2
    IMO no, you are not allowed to scrape the job postings. However, that also depends on what you mean by "scrape". Are you just trying to get these once an hour (or some sane timeframe, even once every 15 minutes) to keep yourself informed? In that case, you are fine. Are you trying to automate a system which hits the jobs every n milliseconds that attempts to bypass the request throttle in order to post these job ads elsewhere? In that case, you are probably going to get banned.
    – Travis J
    May 9, 2017 at 19:12
  • 2
    @TravisJ I'm a beginner now so it's more similar to your first option. Just a personal project to help me practice accessing data from something I'm interested in.
    – Frank Harb
    May 10, 2017 at 4:04
  • 2
    @FrankHarb If it's for a personal project to practice techniques, that is a relevant fact that should be edited into your question. That might affect the kind of answers you get.
    – Brian J
    May 10, 2017 at 13:19
  • 1
    The robots.txt for StackOverflow doesn't specifically disallow the job posting pages
    – John
    May 10, 2017 at 13:57
  • 2
    @John that's irrelevant, allowing search engines to crawl the page doesn't automatically mean the page is allowing everyone to scrape it for whatever purpose
    – im_chc
    May 12, 2017 at 2:17

2 Answers 2

21

Technically we don't have a problem with scraping of /jobs for non-commercial purposes so long as your scraper acts like a responsible citizen. What does that mean?

  • Respect throttling limits. If we return a 429 Too Many Requests then back things off
  • Respect nofollow, noindex, robots.txt. That includes in the rel attribute on a elements, the X-Robots-Tag header, etc.
  • Don't start applying to jobs programmatically. It angers employers and therefore angers us.

I'd also highly recommend using the RSS feeds available on all search pages rather than manually scraping. It doesn't have as much data but it's unlikely to be changed out from under you when we make layout changes.

3
  • Now I wonder if there's shady recruiting services that scrape SO jobs and sell the resulting lists as careers product.
    – Magisch
    May 11, 2017 at 14:14
  • @dean-ward it is safe, and ok, to scrape even from a logged in account? I'm asking because I wouldn't like to do that if it is not an acceptable thing to be done. Apr 9, 2019 at 20:54
  • @DiegoZoracKy logged-in or logged out makes no difference, same rules apply. If you’re curating a list for yourself it can often make more sense because the default sort order is using our matching algorithm so you’ll get more relevant jobs
    – Dean Ward
    Apr 9, 2019 at 23:11
39

You could use the RSS feed instead. From the linked answer:

For jobs, our “API” is the RSS feed. You can get a feed of any search of jobs, look for the RSS icon below the search results, e.g. here:

Also take a look at API Terms of Use:

All Applications must ensure they visually indicate that the Stack Exchange Network is the source of the content provided through the API Services [...]

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .