-2

I have looked everywhere, robots.txt and on the site. I want to crawl and index Stack Overflow.

What are the policies and procedures for doing this correctly?

Timeouts and then subsequent blacklistings need to be avoided.

2

1 Answer 1

21

You should not crawl Stack Overflow. Beyond wasting your time or exercising your crawling software writing skill, there is no need.

Every 3 months most of the public SE data is transferred in a convenient XML format to the Internet Archive and can be downloaded for your own use.

Every week (on Sunday, at 03:00 UTC) the Stack Exchange Data Explorer is refreshed with the latest production data. SEDE allows you to run any SQL query against the database schema so you can shape / filter data to your requirement. It does limit query runtime to two minutes and returns a maximum of 50,000 rows. It does come with an Export to CSV option.

Find on MSE the documentation for the schema: Database schema documentation for the public data dump and SEDE

Alternatively you can use the Stack API. It has various endpoints to gather (and even change) data, be it questions, answers, comments or users. The benefit of the API is its use of live data. The downside is that the API is throttled. It tells you in its response how long you have to wait (backoff) before you are allowed to make your next request. Additionally it only allows 10,000 calls per day but SE does make exceptions, if it is proven that it is needed and you make a good case for it.

Another option is to use one of the various RSS feeds: What other hidden or inobvious RSS feeds are available on Stack Exchange and its sites?. Those produce XML conforming to the Atom standard which can be parsed / transformed relatively easy with any mainstream toolstack.

Lastly there is an option to have a client that connects to one of the many websockets. Those deliver small bits of data but it might aid in deciding what to crawl. Websockets are pretty resilient. I've had sockets open for 6 to 8 weeks.

When you still decide to hit the SE network with a crawler make sure you play, at a minimum, by a few rules, like identify yourself and be considerate.

Then finally to answer your actual question the exact throttling is both a secret and dynamic. If you're successful in hitting the network often enough and transfer enough data of it, you might get an honorable mention.

0

Not the answer you're looking for? Browse other questions tagged .