65

I've seen a few questions that go something like this, usually accompanied by some curl code:

"I'm trying to steal/scrape content from somesite.com but they have some protection in place. How can I get around it?"

Are these types of questions allowed? Should I flag them? Are there any ethics guidelines that I could link to?

18
  • 38
    There is a class of perfectly legitimate questions that do ask how to scrape content from a website. Honestly though, we're not lawyers and we're not trying to be lawyers, so I doubt that there's much you can do to discourage this sort of question except downvote it if it's really terrible.
    – Makoto
    Jan 15, 2016 at 0:07
  • 9
    Sure, but then there are questions that make it blindingly obvious that their intention is far from legitimate. It's these that I ask about.
    – rjdown
    Jan 15, 2016 at 0:09
  • 16
    Are you any kind of law enforcement agency, or the original copyright holder of the site that the person is attempting to scrape? It's tough to say that we can/should enforce those provisos without being on that side. But, to the main point of the question, it can be safely closed as "too broad" since what they're asking to do is not simply achievable without a very long and very detailed explanation, which they ain't gonna get from us.
    – Makoto
    Jan 15, 2016 at 0:13
  • 14
    My own legal position is irrelevant, I think. The question is whether there are existing rules or guidelines, not whether there should be. If the answer is "no", please post it as an answer and I will accept it.
    – rjdown
    Jan 15, 2016 at 0:24
  • 2
    Do you ave any specific examples?
    – apaul
    Jan 15, 2016 at 0:28
  • I would prefer to avoid the "meta effect", but sure stackoverflow.com/q/34562745/1301076
    – rjdown
    Jan 15, 2016 at 0:30
  • @StephanBranczyk the question is not about what is considered copyright infringement, nor is it about mishandling. I don't think an example is needed for what I am asking, but there is one directly above your comment.
    – rjdown
    Jan 15, 2016 at 1:14
  • I take the approach that I do not wish to be an accessory to anything that I consider is possibly illegal or immoral if I do not benefit from it!
    – Ed Heal
    Jan 16, 2016 at 9:29
  • 4
    I'm no lawyer and am wildly speculating, but surely you have to republish content for copyright law to come into play? If simply downloading freely-available content were automatically infringement, wouldn't we be infringing every time we visited websites in ordinary browsers, unless the site owner had somehow granted us permission? In the case where the scraper is deliberately working around protections, maybe they'd be violating some countries' broadly-drafted laws against "bypassing security systems", but I doubt that web scraping alone can ever be a breach of copyright, per se.
    – Mark Amery
    Jan 16, 2016 at 20:07
  • 6
    @MarkAmery at the very least he is attempting to circumvent the terms and conditions of the site, specifically the "Your Use of the Site" section. It's also not much of a stretch to assume he'll be republishing and/or redistributing the material in some form. There would be no point otherwise, except for personal use, but that's more of a stretch. But (yet again), this question is not about the legality. Nor is it about that question in particular. It is about the ethics, and more to the point, the current rules of stack overflow.
    – rjdown
    Jan 16, 2016 at 20:30
  • 5
    @rjdown It's also not much of a stretch to assume he'll be republishing and/or redistributing the material in some form. There would be no point otherwise, except for personal use, but that's more of a stretch - he may be redistributing the data (or aggregating it for statistical/research/journalistic purposes) without copying the form in which it was presented nor reproducing any significant body of prose. Those are perfectly ordinary, typical use cases for web scrapers, and are perfectly legal and ethical. You cannot copyright raw knowledge, and nor should you be able to.
    – Mark Amery
    Jan 16, 2016 at 20:34
  • 3
    @rjdown my objection to the premise of your question is not just that I think you may be wrong about details of law; it's that you've assumed that a perfectly reasonable, innocuous activity is in some way shady. If Stack Overflow didn't allow questions about any software development practice that could in principle be used to do illegal or unethical things, we wouldn't be able to ask anything at all; you've asked this question with the assumption that web scraping is somehow dodgier than other programming activities, but not even tried to make the case for why.
    – Mark Amery
    Jan 16, 2016 at 20:38
  • 14
    "they have some protection in place. How can I get around it?" if that's not "dodgier" to you, then we share different ethics.
    – rjdown
    Jan 16, 2016 at 20:43
  • 1
    @rjdown if you want to do something about such questions 99% of them show zero amount of research - downvote on for that. Also they are generally way too broad... Those that are specific will not be able to provide [MCVE] :) Jan 17, 2016 at 0:04
  • 1
    I think the question is a bit inflammatory and based on too much assumptions, especially when you put what you extrapolated in quotes as a title.
    – thor
    Jan 17, 2016 at 21:29

4 Answers 4

61

The short answer is "No", SO doesn't enforce the copyrights of/for other entities unless they receive a proper DMCA Takedown Notice

There's a lot of gray area and wiggle room between "How do I do this nefarious thing?" and "How do I prevent others from doing this nefarious thing?"

In general having a specific post about a common exploit is good for everyone... If it becomes common knowledge, it becomes easier to anticipate and prevent.

Applying that to the example you mentioned in the comments... I'm sure someone at Apple will eventually see the post you're concerned about and do something to close the loophole.

Related:
Policy regarding questions related to unethical or "shady" practices

4
  • 4
    My question was more about ethics than legal enforcement, but the related link was helpful. Opinion and tolerance appears to change between network sites. Odd. Thanks.
    – rjdown
    Jan 15, 2016 at 1:18
  • 6
    Bear in mind that whilst there are moderator enforced policies, the choice to close/downvote/delete are in the hands of the user community. And it is well within your rights to downvote unethical questions/anwers.
    – Sobrique
    Jan 16, 2016 at 9:53
  • 1
    @rjdown Wouldn't a question about ethics be more appropriate on Philosophy SE? Jan 17, 2016 at 17:46
  • 7
    Not when it is about SO's rules.
    – rjdown
    Jan 17, 2016 at 17:47
5

I vote to close these sorts of questions, and usually downvote as well. Here's why:

Typical Question:

How do I scrape YouTube videos and turn them into MP3?

Not only does this violate the YouTube terms of service and encourages copyright violation, but it's simply too broad. The question isn't properly asked nor broken down.

Better Question:

How can I fetch the HTML of a page?

How can I parse the HTML of a page?

How can I search the DOM for elements referencing media resources?

How can I efficiently stream a large file from my HTTP client in (framework here) to disk?

How do I convert media from one format to another, stripping video tracks in the process?

Had someone asked these questions, not only are they totally appropriate for Stack Overflow, but they avoid the issue of copyright and terms of service entirely. The questions then become useful for their technical merit, and can be used as reference for any similar purposes.

Even if the person asking the question didn't know how to break the problem down, the fact that they wanted to scrape from YouTube is completely irrelevant to the problem being solved. They could have asked how to break down the problem of scraping HTML and saving the media at a minimum.

I downvote these bad questions because they're fairly useless, and the people asking them are lazy. Lazy people don't typically want to learn, they want a shortcut. I choose to spend my time on Stack Overflow helping people so that they can learn and help others. The best lesson to teach beginning programmers is how to ask good questions. Closing their bad questions and leaving a comment explaining why is an efficient way to teach this lesson.

2
  • Does a (clear) request for help to breach copyright/ToS play a part in your down/close vote? Or is it always purely due to a low quality question?
    – rjdown
    Jan 17, 2016 at 21:59
  • 1
    @rjdown While I personally think that making a business out of content theft is totally wrong, and I might add commentary and a downvote for that opinion, I wouldn't close vote because of it. I agree with the other comments that it isn't the responsibility of Stack Overflow to police what people do with answers to questions. However, if the question is asked correctly then I don't see how one could determine what the purpose, legitimate or not, of the code could be. Therefore, the low quality of the question and the copyright infringement are seemingly (always? usually?) linked.
    – Brad
    Jan 17, 2016 at 22:08
3

Given the subject matter of SO, I feel totally confident saying most regularly-participating members of this site are intelligent enough to know the difference between someone who is clearly asking for help to perform a specific data theft (as in your example) and someone who is asking about a technique which "could in principle" be used in an unethical way.

And the "discussing DVD encryption hacking can lead to better DVD encryption" argument from the related question Paul linked? Please. That's a secondary and unintended side effect of discussing DVD encryption hacking at very best, and such conversations will bring more DVD hackers than DVD encryption designers to SO by a factor of roughly 80 bazillion to 1.

Both of these arguments sound to me a lot like justifications for sticking our heads in the sand.

6
  • Agreed, There is a grey area but sometimes it is crystal clear.
    – rjdown
    Jan 17, 2016 at 2:26
  • 4
    "Cracking" DVD DRM is necessary to make a non-commercial player, and you need to have the physical media -- I think this is the least immoral illegal example.
    – Jeremy
    Jan 17, 2016 at 7:39
  • 2
    What is this concept that you refer to as "data theft"? Either you're illegally infringing somebody's intellectual property, or you're not; whatever broader category of behaviour you're trying to describe here that you believe to be immoral, I don't understand what it is, but I'd probably disagree vehemently with you if I did. The law of most countries respects patents and copyright, forbids bypassing security, and allows a limited ability to contract away your freedom to share knowledge, but beyond that, facts cannot be owned, and nor should they be allowed to be.
    – Mark Amery
    Jan 17, 2016 at 17:20
  • 5
    @rjdown the information sought by the scraper you objected to from the Stack Overflow question you linked to included or potentially included plenty that is in no way protected by copyright, like album ratings, album genres, numbers of preview songs for an album, or aggregate information about descriptions (like analyses of language usage). The idea that nobody but Apple has the right to programatically acquire such knowledge from the public iTunes store offends my moral sense more than any defence of mere piracy would.
    – Mark Amery
    Jan 17, 2016 at 17:29
  • 1
    Description text is likely licensed from a music authority, as is typical throughout the industry, or custom-built by Apple. Ratings are definitely owned by Apple. These are not "facts" - this is intellectual property. The website and its contents as a whole are also protected by copyright, and the terms and conditions for use clearly states that the use of scrapers is not allowed. Apple generously provide an API for a lot of information, but stop short of letting everyone copy EVERYTHING that they have spent a fortune on accumulating. Not everything is free. Sometimes you have to work for it.
    – rjdown
    Jan 17, 2016 at 17:42
  • 3
    @rjdown No, ratings are not intellectual property. If they were, valuable sites like metacritic would not be able to exist. That the site as a whole is protected by copyright is irrelevant; you cannot acquire ownership of a piece of knowledge simply by writing it down, you only own that representation, otherwise no journalist would ever be allowed to report on a story that another journalist had broken first. I have no desire to live in your 'ethical' world where I require a giant corporation's permission to investigate or reveal facts about its business.
    – Mark Amery
    Jan 17, 2016 at 17:52
1

It's just very difficult to differentiate between legitimate uses and illegitimate uses and as a general principle one should always give the benefit of the doubt until it's clear that a question is used to circumvent copyright.

For example, scraping content of websites is not per se illegal.

So we cannot just close all such questions.

However we can choose not to answer them in case we feel they might likely be used to break copyright.

In cases where it is extremely likely I would actually vote to close them (with a custom reason: is used for circumventing copyright or so...).

Maybe we could compile a list of examples/principles how one can recognize such a clear case.

2
  • A list would be great but really, the questions I'm talking about have stepped out of the grey area and there is no doubt.
    – rjdown
    Jan 17, 2016 at 21:47
  • @rjdown Then do not answer them and close vote. But I didn't know you meant absolutely clear questions because you didn't gave any examples. Anyway I fear we can only hope to catch the more stupid ones while the smart askers might get away by hiding their intentions. Jan 17, 2016 at 21:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .