106

I saw this, and it completely blew my mind:

http://gkoberger.github.io/stacksort/

It takes a problem, and a set of unit tests, then scans Stack Overflow for answers and applies them the best it can, and repeats until the unit tests pass.

I can't help but think that this is a huge idea, writing an algorithm to probe the hive mind for a solution to a problem.

I was thinking of writing some software to scrape Stack Overflow to do this. Do the creators of the site approve their site to be used in this way and perhaps is there a way we can streamline this process for great justice?

12
  • 9
    Servy mentioned using the API for something like this. Good idea. The link you provided does utilize the API
    – Andy Mod
    Commented May 8, 2015 at 15:55
  • 45
    Obligatory XKCD reference
    – Makoto
    Commented May 8, 2015 at 16:09
  • 92
    Have a computer grab code from the internet and execute it... What could possibly go wrong? Commented May 8, 2015 at 20:24
  • 16
    Even on StackOverflow, where hopefully nobody's writing malicious answers (or at least not getting away with them for long)… I have used __import__('os').system('rm -rf /') (or equivalents for other languages/Windows/etc.) as an example every time I've had to explain why you want literal_eval instead of eval. Some of these have been upvoted pretty high. Can stacksort tell that the best way to "apply" the code from that question is to read the text and understand that the whole point is to make sure not to ever call that code?
    – abarnert
    Commented May 8, 2015 at 21:53
  • 18
    To those that didn't read the disclaim on the linked site: "Is it safe? Uh… it evals both user input and random code, unchecked, from an external site. This is what security-minded folks would refer to as Very Bad™. "
    – Andy Mod
    Commented May 9, 2015 at 4:02
  • "That being said, for what it is, it should be relatively harmless." I ran it a few times and nothing bad happened :-) Commented May 9, 2015 at 11:01
  • 1
    It does some basics checks before running the code to avoid xhr and some other things and hopefully any malicious stuff has already been moderated on SO :) github.com/gkoberger/stacksort/blob/master/js/script.js#L158 Commented May 9, 2015 at 15:11
  • Unless your algorithm does automatically post a new question to SO when none of the tested answers work, this seems to be totally fine :-)
    – Bergi
    Commented May 9, 2015 at 20:14
  • 5
    @JonasCz - I also ran it a few times and nothing bad ha*(&*(!^
    – DVK
    Commented May 9, 2015 at 20:54
  • I saw this/similar idea presented on a conference by prof. Langdon: cs.bham.ac.uk/~wbl/biblio/gp-html/WilliamBLangdon.html
    – BartoszKP
    Commented May 11, 2015 at 15:16
  • 1
    @Makoto That xkcd was the inspiration of the linked project and is linked at the top of its page. :)
    – reirab
    Commented May 11, 2015 at 15:44
  • Lol, reminded me of this tweet Commented May 12, 2015 at 4:07

1 Answer 1

101

Use the Stack Exchange API if you want to programmatically access content from the site. Don't query the site directly and try to scrape the content, it's inferior for everyone involved.

If you're using sufficiently large quantities of data, you may also just want to grab the whole data dump and use that instead.

1
  • 25
    By the way, let's pause for a moment and think about how nice SO is for giving the data dump, considering how many websites leech that into cheap ad outlets. It's really great that this kind of thing is available. Commented May 11, 2015 at 8:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .