34

My question is inspired by this one as well as by this article that appeared in The Atlantic a number of years ago. The OP in the linked question was asking about the best way to demonstrate your knowledge and talent to prospective employers (specifically in the context of Stack Overflow jobs). His observation was that knowledge, talent, and your resume may not always align perfectly (I've known relatively young engineers that are almost intimidatingly talented and more senior engineers that haven't done anything new in 20 years). This can be a problem because if prospective employers look only at someone's resume they can miss out on high-potential candidates or end up hiring someone who'll be disappointing in the long run (in spite of having a lot of experience).

I was thinking: our Stack Overflow profiles have a lot of information about us as developers - code samples, writing samples, questions, answers, etc. It seems like you could infer a lot about someone (e.g. communication skills, how they're likely to interact with coworkers, their technical knowledge, etc.) based purely on their Stack Overflow profile and posts.

I'm hoping that this isn't too broad of a question, but have there been any efforts to mine this data (either alone or as data points for a larger algorithm)? How predictive is it of your actual ability as a developer? How strongly do objective measures of how well you're "doing" on Stack Overflow (e.g. reputation, badges, etc.) correlate with objective measures of job performance and success?

Does selection bias (the fact that you can delete poorly-received questions and answers, for example, and the fact that you decide which questions you attempt to answer) affect the accuracy of these measurements?

In particular, has anyone tried to use this to make hiring or promotion decisions (analogous to what the article in The Atlantic that I linked to describes about data-driven hiring or what the Oakland Athletics did when they started using a sabermetric-driven approach to identify undervalued players)?

6

1 Answer 1

3

If you were intending to data mine StackOverflow, even for a pet project, this post and answer from a SO mod will help guide you in the right direction. In particular, the existence of a SO API seems pertinent.

Here is his answer:

We have scripts that check for unusual / abusive access patterns, and a daily "top n" traffic summary report of inordinate and anomalous usage.

We regularly block (IP range ban) unknown scrapers that do not identify themselves and/or have poor behavior patterns. These bans are permanent until someone emails us to make a case that they should be removed.

If you don't want to get blocked, here's how:

  1. Use GZIP requests. This is important! For example, one scraper used 120 megabytes of bandwidth in only 3,310 hits which is substantial. With basic gzip support (baked into HTTP since the 90s, and universally supported) it would have been 20 megabytes or less.

  2. Identify yourself. Add something useful to the user-agent (ideally, a link to an URL, or something informational) so we can see your bot as something other than "generic unknown anonymous scraper."

  3. Use the right formats. Don't scrape HTML when there is a JSON or RSS feed you could use instead. Heck, why scrape at all when you can download our cc-wiki data dump??

  4. Be considerate. Pulling data more than every 15 minutes is questionable. If you need something more timely than that ... why not ask permission first, and make your case as to why this is a benefit to the SO community and should be allowed? Our email is linked at the bottom of every single page on every SO family site. We don't bite... hard.

  5. Yes, you want an API. Now there is one! see http://stackapps.com for all the info you could possibly want, and more.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .