9

Often times you find questions where the OP is trying to scrape a website and in that process is trying to use the internal API of that site, while doing so they face errors and ask a question about resolving the same.

Keeping aside the potential legal issues with these, which is touched upon in "Handling reverse-engineering related questions" these questions have various issues:

  • Might not be useful for future readers: These questions are very specific to a website and don't add much value for other readers.
  • No reasonable way to debug / answer: Given the question is about calling an internal API, one is most likely not going to find any documentation about the API and how to call it, any answerers will have to depend on trial and error to find a solution.
  • Not a programming question: Such questions are less about programming and more about reverse engineering how an API works.

Are these questions on-topic or should they be closed?

4
  • 8
    How exactly are the existing rules not sufficient? We don't need more layers of rules, don't overcomplicate things for yourself. If a scraping question does not have enough debugging details vote to close it as such. If it does... well then probably it's still pretty useless so put a quality vote as you see fit.
    – Gimby
    Commented Oct 11, 2023 at 11:50
  • 1
    @Gimby I personally do believe these should be closed but on the other hand I see these questions receiving other interactions such as edits, etc. as well without close votes. Some of them do get answered as well. So this question is me trying to figure out whether it is only me that has that opinion or does the community think something else on this. Commented Oct 11, 2023 at 12:24
  • Just like any other low quality question. Scraping questions are not special, so let's not pretend they are. Judge by what you see, not by what you personally dislike.
    – Gimby
    Commented Oct 11, 2023 at 13:51
  • related: At what point (if ever) is a fact-based question about the behaviour of closed-source software off-topic / not a good fit for SO? I also list there a bunch of related questions.
    – starball
    Commented Oct 11, 2023 at 21:30

2 Answers 2

17

Close these questions as lacking a minimal reproducible example.

These questions are no different than someone asking for help with debugging proprietary code and refusing to share it.

Alternatively, questions asking for reverse engineering an entire service or platform (like an API) are too broad, so those can be closed as needing focus, similar to asking how to build a "Facebook killer".

And, as always, even if you're not sure whether the question is close-worthy, if you think it's low quality or low effort, be sure to downvote it. If discerning viewers get there early enough, downvoting such questions can prevent the need for closing them in the first place.

4

While such questions are about programming — that it is against a website instead of a library doesn't really change that — and could in theory be answered to sufficient quality (that is, they admit a specific answer instead of just wanting wishy-washy opinions that go nowhere), the fact that they're unlikely to be useful to anyone else in the future is a strong black mark against them. I'd suspect that it would be very rare that such questions will reach the general quality threshold, especially in regards to utility to others. But if that is reached then it's hard to say that they are suitable for a hard prohibition except for potential reasons of legality.

The possible legality issues are definitely likely to be important. Stack Exchange have to care about that stuff because they'll get into bad trouble otherwise. But that would need actual legal advice; I'm not and never have been anyone's lawyer.

(I'm of the opinion that if something is delivered to clients at all, it's either something that should be documented as a public API, or it should follow a standard pattern such as an OpenID authentication flow. Exposing private endpoints to public clients is simply a recipe for trouble, and fools who do so bring the ensuing woes on their own heads.)

2
  • 6
    "StackExchange have to care about that stuff because they'll get into bad trouble otherwise." Section 230 indemnifies Stack Overflow or any other similar site from legal threats due to content posted by users. I recommend deleting this paragraph as it's incorrect and just defers to authority at the end anyway. The third paragraph is also irrelevant to the question and should be removed, in my opinion.
    – TylerH
    Commented Oct 11, 2023 at 16:05
  • 1
    "The possible legality issues are definitely likely to be important" that is for Stack Overflow to decide, not us. This has come up in the past when dealing with questions that ask how to write a virus. As much as you might hate the idea and want such questions to burn in hell, it's not an illegal question especially since the knowledge can be used to combat viruses as well.
    – Gimby
    Commented Oct 13, 2023 at 9:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .