16

Maybe it's just the tags I follow, but and tools that facilitate it seem to be very popular for questions these days. One category of question I see frequently is some form of:

I'm trying to scrape (some website) and my code was working fine for a while but now I'm getting (a 401/404/429 or some other HTTP error | an empty response | a timeout). Please help!

Nearly always it is the server and not the client that is responsible for this change in behavior, and nearly always the issue is that the server administrator doesn't want this kind of scraping and is trying to prevent it, using some kind of web application firewall or other bot-detecting mechanism, and serving errors or tarpitting the connection intentionally.

There are various techniques to evade server-side detection or to circumvent limitations, but these techniques are mostly not language-specific or tied to a particular website. This argues in favor of a question that can collect high-quality canonical answers about techniques rather than about implementations of those techniques in different languages. (For example, various kinds of back-off and request rate limiting, using a CSRF token from one request to build the subsequent request, etc.)

More importantly, I feel like such a question could also be a good place for a canonical answer explaining the importance of looking for alternatives to web-scraping, such as using a documented API when available. Many web-scraping questions are ultimately aimed at subverting the terms and conditions of the sites in question, and I feel like that should be mentioned as well, since there are legal and ethical implications of that. (I'm not saying that Stack Overflow should censor questions about User-Agent randomization, but just that we should point out that if you randomize your User-Agent with the sole intention of circumventing a rate-limiting mechanism, you're pretty clearly operating contrary to professional ethics.)

(On the flip side, there might also be a place for the "opposite" canonical question, about how to detect HTTP requests from web-scrapers or other bots and how to respond to them. Answers could describe things like request fingerprinting, rate-limiting and quota, and so forth. I say "might" in this case because relatively few people are actually implementing such things—most websites that are big enough to care will already rely on Cloudflare or Akamai or some other edge network to handle this stuff for them.)

In any case, I'm not aware of any existing canonical question. Is there one anyone can recommend? If not, I can create one, but I figured it would be better to ask the community first.

2
  • I'm not aware of any either. Considering the worries about scope mentioned in some of the answers below it'd probably be more useful to judge something already written up if you've got it. A really good answer goes a long way to salvage what's likely to be a pretty crappy question. Commented Jan 4, 2019 at 21:16
  • Many people define that sort of thing (rate-limiting, bypassing rate-limits, rotating or spoofing IPs etc.) as web-crawling not web-scraping proper. I suggest you ask instead "What's on-topic and off-topic to ask on SO under web-scraping? Which questions are web-crawler instead?")
    – smci
    Commented Jan 6, 2019 at 9:33

3 Answers 3

14

Leaving aside ToS considerations for the moment:

Almost all scraping (I've seen it spelled "scrapping" these days so many times I'm struggling to notice the difference and not chuckling about it as much as I use to) questions I've seen are regarding a misunderstanding what the source served DOM and the difference between how the browser JS modifies the DOM etc...

Almost all - unless they provide a DOM source in the question (and again they get confused with a browser generated DOM and not the originally served DOM) are too broad. One has to go an off-site resource (if even mentioned - because those with nefarious intents try to obfuscate that, which doesn't help at all).

If there's something giving a self-contained example that can be answered with "your XPath/CSS selector query is off", I might drop a hint (learn the developer tools available in your browser(s))
...otherwise, I generally close as too broad.

7
  • 2
    Yeah, the dynamic content issue is far more common from what I've seen. If there's a canonical for that I'd use it often. Commented Jan 4, 2019 at 20:33
  • @PaulCrovella I'm not sure how that could go further than "check your browser's development tools" and go from there... which could lead to "I don't understand that... or... I've got 'this' response - what does that mean" :(
    – Jon Clements Mod
    Commented Jan 4, 2019 at 20:36
  • 3
    @Paul scraping sites is a massive industry - one could argue their data is public anyway so it's fair game, but ToS of sites says please don't etc... I know I've had to scrape a site for a client because their supplier site couldn't work out how to expose their API or such... so they said fine, just do that until we work out what we need to do... it was harmless... gotten fairly proficient at it over the years though...
    – Jon Clements Mod
    Commented Jan 4, 2019 at 20:44
  • 4
    I'm still hopeful someone will come up with a mid-ground between "check yer dev tools" and a treatise on the world wide web. Or maybe they have and I haven't run across it.. dunno. It'd just be nice to give folk useful info without having to cater it specifically to their exact situation and (lack of) understanding, or turn them away completely. As far as ethics and all that go I can't say I'm totally comfortable with scraping, but there are other places handle those issues. Commented Jan 4, 2019 at 21:03
  • 3
    @PaulCrovella I've seen too many questions like this - that I suspect are just some guy trying t o support his family - doesn't know what he's doing and needs help... sadly I think there's so much lack of understanding that no one can help. I always give the benefit of doubt though and nudge and try to share experience...
    – Jon Clements Mod
    Commented Jan 4, 2019 at 21:27
  • A question like "My browser's dev tools show <thing>, but when I download/parse the page with wget/BeautifulSoup/Jsoup, it's not there. What gives?" would be useful. (Step 1: spoof the browser's user-agent string/referer/cookies/other headers. Step 2: it's dynamically loaded with JavaScript; either talk to the AJAX endpoints/parse the inline script tags, or use Selenium/PhantomJS/etc.) Commented Jan 5, 2019 at 8:48
  • Re the ethics of scraping, I'm sure there are people scraping their own company's sites because the person who maintained them quit and they're too scared to touch anything lest it break, or in larger companies, because no one from that department is willing to work with them, or whatever. We can ask people to respect other people's servers and data, but that's about it. Commented Jan 5, 2019 at 8:50
8

There are various techniques to evade server-side detection or to circumvent limitations, but these techniques are mostly not language-specific or tied to a particular website.

This makes trying to have a single question about the whole subject inherently Too Broad. Honestly even with some more specifics (like language being used) there's a good chance the question is too broad.

Trying to have a question covering all of the conceptual information relevant to scraping a site is still going to be too broad, even if you're trying to avoid the specifics of any given language or site. It's just too much.

Your "reverse" canonical, of how to avoid scrapers scraping your site, is also a really broad question. Too broad for the site, I'd say.

6
  • 1
    Canonical questions are often quite a bit more broad than what's typically accepted. I wouldn't dismiss them out of hand on that basis alone. The quality and usefulness of their answers tend to make up for it. Without seeing it written up I wouldn't judge it yet, and would probably leave it up to subject-matter experts in any case. Commented Jan 4, 2019 at 20:27
  • 1
    @PaulCrovella There aren't "Canonical" questions and "non-canonical" questions. All on topic questions are intended to be the canonical versions of that question, or be closed as a duplicate of the existing one. Questions are designed to be sufficiently narrow that it's reasonable to provide a complete answer for them in the QA format presented. This isn't just for certain questions, it's for all questions.
    – Servy
    Commented Jan 4, 2019 at 20:31
  • @PaulCrovella Finally, it's inherent to the nature of Too Broad questions that actually writing them up is either impossible or infeasible. You don't answer a question an then figure out if it's too broad or not. The question proposed would take an enormous amount of information to provide any sort of complete answer for. That makes the question too broad. You don't need to write an answer first to determine that.
    – Servy
    Commented Jan 4, 2019 at 20:31
  • In theory, sure. But in practice there are numerous question/answer sets that were crafted to be canonical, and they're great. Commented Jan 4, 2019 at 20:35
  • 1
    @PaulCrovella That someone is writing a question that they intend to post the answer to doesn't make that question immune to the quality standards of questions. They're just as important for such questions as for a question where the author doesn't know the answer. Self answering a question can be very useful, but it still needs to be reasonably scoped to be a good question. Since you're probably not looking to try to post an entire book as an answer, that generally means figuring out what more specific question you can ask to still provide the information you think is appropriate.
    – Servy
    Commented Jan 4, 2019 at 20:40
  • 2
    I hate to break it to you, but there are a lot of those "books" on the site already being used as canonicals, covering many more facets of an issue than is normally reasonably accepted. It's a matter of practicality and expediency for dealing with the deluge of only very slightly different special snowflake questions that show up every day. The ideal for this site doesn't always work at scale, and people have worked around that. SO docs was another attempt to address the issue, but we never got the ability to dup-hammer against docs so the utility was limited. Commented Jan 4, 2019 at 20:52
1

I doubt that a canonical for web scraping techniques could properly fit for the Stack Overflow policies.

That sounds more appropriate for SE Software Engineering, but still quite broad, if not asking for 3rd party resources (which is also an off-topic close reason over there).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .