If you were intending to data mine StackOverflow, even for a pet project, this post and answer from a SO mod will help guide you in the right direction. In particular, the existence of a SO API seems pertinent.
Here is his answer:
We have scripts that check for unusual / abusive access patterns, and a daily "top n" traffic summary report of inordinate and anomalous usage.
We regularly block (IP range ban) unknown scrapers that do not identify themselves and/or have poor behavior patterns. These bans are permanent until someone emails us to make a case that they should be removed.
If you don't want to get blocked, here's how:
- Use GZIP requests. This is important! For example, one scraper used 120 megabytes of bandwidth in only 3,310 hits which is substantial. With basic gzip support (baked into HTTP since the 90s, and universally supported) it would have been 20 megabytes or less.
Use GZIP requests. This is important! For example, one scraper used 120 megabytes of bandwidth in only 3,310 hits which is substantial. With basic gzip support (baked into HTTP since the 90s, and universally supported) it would have been 20 megabytes or less.
Identify yourself. Add something useful to the user-agent (ideally, a link to an URL, or something informational) so we can see your bot as something other than "generic unknown anonymous scraper."
Use the right formats. Don't scrape HTML when there is a JSON or RSS feed you could use instead. Heck, why scrape at all when you can download our cc-wiki data dump??
Be considerate. Pulling data more than every 15 minutes is questionable. If you need something more timely than that ... why not ask permission first, and make your case as to why this is a benefit to the SO community and should be allowed? Our email is linked at the bottom of every single page on every SO family site. We don't bite... hard.
Yes, you want an API. Now there is one! see http://stackapps.com for all the info you could possibly want, and more.
- Identify yourself. Add something useful to the user-agent (ideally, a link to an URL, or something informational) so we can see your bot as something other than "generic unknown anonymous scraper."
- Use the right formats. Don't scrape HTML when there is a JSON or RSS feed you could use instead. Heck, why scrape at all when you can download our cc-wiki data dump??
- Be considerate. Pulling data more than every 15 minutes is questionable. If you need something more timely than that ... why not ask permission first, and make your case as to why this is a benefit to the SO community and should be allowed? Our email is linked at the bottom of every single page on every SO family site. We don't bite... hard.
- Yes, you want an API. Now there is one! see http://stackapps.com for all the info you could possibly want, and more.