5

I have recently come across several posts that seek help to avoid bot detection when running scripts on some website. I am avoiding posting examples here so as not to draw attention/meta effect to specific posts, but I can add some if it is appropriate.

At least some of these questions have enough detail to provide an answer (and there is actually something that can be done), but it feels like the wrong thing to do, providing such advice on Stack Overflow. That said, I cannot find any documentation that such requests should be removed and/or answers not provided. And, there are several fairly highly upvoted posts that discuss this topic. For instance, this one, this one, and this one..

Is avoiding bot detection an acceptable topic?

12
  • 3
    If you're uncomfortable providing an answer or advice don't, but we're not here to provide lessons on ethics, we're here to answer coding questions. Of course, there are always ethical reasons for asking such questions as well (wanting to know how to defeat some security, in order to improve the security of something, for example) Commented Jun 15, 2021 at 23:24
  • 5
    The real policy is here: "...you have to treat each post on its own merits."
    – Cody Gray Mod
    Commented Jun 15, 2021 at 23:26
  • No specific example if a question or a bot violates any TOS, but related: Should questions that violate API Terms of Service be flagged? Commented Jun 15, 2021 at 23:26
  • 6
    But, to answer your specific question, we don't police the intent of questions asked on Stack Overflow. There are legitimate reasons for doing most things, and then there are illegitimate reasons. Even when the asker may have nefarious intentions, getting the information out there may be to the broader benefit of the community. If the question meets our criteria for a good question, then it's fine. Otherwise, it's not, and you'll easily find one of the standard close reasons/flags that fits. As always, you can express your opinion about the question, subjective or not, by voting.
    – Cody Gray Mod
    Commented Jun 15, 2021 at 23:30
  • 7
    The assumption behind this kind of question always seems to be that discussions about how to circumvent some security system being public on the "normal" internet (like SO) helps the attacker more than it helps the defender. I am honestly not sure that's the case; while SE relies on some obscurity itself for some things like the Q ban algo, there are plenty of examples of attackers getting the info anyway from some darker corner of the internet while the defenders blissfully think they are running a secure service..
    – Baum mit Augen Mod
    Commented Jun 15, 2021 at 23:31
  • 5
    We get a lot of questions from people trying to "scrap" a website. But many of them are close-worthy because they are unclear, don't have an adequate MCVE, lack focus, etc. But sometimes I do wish that we had a close reason that addressed the ethical issue of code that defies the ToS of a site, and which unfairly strains the site's resources. Many of these scrapers just don't get the concept that what they're doing is unethical, and they only throttle their code in order to reduce the odds of being detected, not because it's the polite thing to do.
    – PM 2Ring
    Commented Jun 16, 2021 at 3:50
  • 1
    @PM2Ring that would be shot down immediately, because SO doesn't police the ToS violators. What it should police is bad security practices.
    – Braiam
    Commented Jun 16, 2021 at 12:53
  • 1
    @Braiam Sure, it's not SO's job to police ToS violations. But I would be happy if we did have an official policy that discouraged helping ToS violators. OTOH, I don't downvote such questions or their answers purely on ToS grounds. Bad security practices is a different issue, and I have no problem downvoting answers as not useful if they (for example) contain code that's vulnerable to SQL injection.
    – PM 2Ring
    Commented Jun 16, 2021 at 13:05
  • I find it interesting that there are questions about Selenium. Selenium is above else a testing framework, even though of course you can also use it for different more nefarious reasons since it is pretty good at what it does, why would you re-invent the wheel. I don't find the linked questions all that objectionable. Bot detection can actually make it hard to created automated tests using Selenium.
    – Gimby
    Commented Jun 16, 2021 at 14:10
  • 1
    @Gimby this is likely selection bias. I answer a lot on the selenium tag so that's where I found the examples.
    – C. Peck
    Commented Jun 16, 2021 at 14:13
  • @PM2Ring You, like me, might be a big fan of this answer by Ben Voigt. That is, "You should not do this because it violates the Terms of Service" is an answer to these types of questions.
    – Cody Gray Mod
    Commented Jun 16, 2021 at 16:06
  • @CodyGray That answer is ok, but it's not strictly true. A sufficiently advanced scraper may be able to operate for a long time. OTOH, once the scraped site figures out how to block the scraper it will be very hard to write a new scraper that avoids detection and retrieves data fast enough to be useful. We often get questions about that: "I've been using this script to scrape this site for a few month, but it suddenly stopped working. How do I fix it?"
    – PM 2Ring
    Commented Jun 16, 2021 at 17:42

0

Browse other questions tagged .