10

I came across this question this morning: https://stackoverflow.com/questions/29961395/proxy-issue-in-loading-website.

It seems likely that the asker is not scraping the site in question "legitimately" here, and for that reason my first instinct was to downvote and move on.

I didn't, though - I hopped over and reread a few sections of the FAQ and couldn't quickly find a reason why the question shouldn't be allowed. After all, in my opinion there's certainly value in explaining how to write a bot responsibly, for future visitors to the site.

So I put together an answer which I thought would help these future visitors, while the downvotes piled in on the question.

Soon after, my answer got a downvote, which wasn't entirely unexpected, but led me to question what I'd done a little.

I'm interested if others agree with my interpretation here. I'm not exactly a super-regular around here and so I don't have the grasp on what's ok and what isn't that many of you do.

Is the question OK? Is my answer OK? Should I have answered at all?

5
  • Thanks for metaizing this 'question', so allowing me to DCV it:) Commented Apr 30, 2015 at 10:41
  • Do you have anything to add on why you chose to closevote? I'm pretty sure the question meets all the criteria that it needs to be classed as on-topic here.
    – Hecksa
    Commented Apr 30, 2015 at 11:01
  • 2
    @Hecksa That question was closeworthy because the OP is working with specific code but does not show the code in the question, or tell us what error specifically they get. There can be any number of things that would cause a site to block a bot, without more information from the OP, the answers (as well intentioned as they may be) are going to be guesses or generalities. ("Blocked" is too general a symptom.)
    – Louis
    Commented Apr 30, 2015 at 12:09
  • @Hecksa: it's frustrating when a bad question like this gets a good answer like yours, since you lose the credit and the permanent record of your work. It is rather a risk for these situations though. Where a question has been placed on hold but not deleted, you can either fix it yourself and vote to reopen, or encourage the OP to do so, but the vast majority of 1-rep users are dump-n-run, and are unwilling to put in more than trivial amounts of effort.
    – halfer
    Commented Apr 30, 2015 at 15:12

3 Answers 3

15

Is the question OK?

Yes. I think the general idea behind the question is. Bots scrape sites all the time, and a properly written bot shouldn't have to have a noticeable impact on a site. (Ever heard of those tiny companies called "Google", or "Yahoo"?)
I like to assume good intentions. Maybe the user is unaware of the impact a scraper can have on a site. The only way this person can learn how to properly run a bot like that, is by asking questions.
That doesn't take away the fact that the question is mediocre, in that it's severely lacking content.

Is my answer OK?

Yes. It explains the hazards of running a bot like that, and suggests some things to keep in mind. (I'd suggest adding something about robots.txt, though)

Should I have answered at all?

An answer is okay considering the subject, although it may have been a waste of time, considering the quality of the question itself, which may be closed soon.

Also, keep in mind that one downvote really doesn't mean a lot. Just one person disagreed...

8
  • You are assuming good intentions. Make the poster does know about rate limiting, and what kind of load a bot can cause. :)
    – TZHX
    Commented Apr 30, 2015 at 8:41
  • 3
    If the poster knew about that, he'd have asked how to circumvent IP blocks specifically. Always assume good intentions if the intentions aren't clear. As far as we know, we should try to be welcoming, on the SE network.
    – Cerbrus
    Commented Apr 30, 2015 at 8:43
  • 2
    "IPs are blocked by website. How can I overcome that?" -- how is that not what the user is asking? I think the intentions are clear enough to base an opinion on -- clearly not everyone is going to have the same opinion, and that's a good thing.
    – TZHX
    Commented Apr 30, 2015 at 8:44
  • 2
    I tend to agree with the "assume good intentions", but even if I didn't, am I not writing an answer for everyone who might visit the answer in future, not just the individual asker? I'd like to be helpful to them, if not the actual asker.
    – Hecksa
    Commented Apr 30, 2015 at 8:46
  • I mean that he wouldn't have mentioned the scraping if he was aware of rate limiting / load caused by a bot, since he'd be aware that would be questionable.
    – Cerbrus
    Commented Apr 30, 2015 at 8:46
  • @Hecksa: You should indeed write a answer with future visitors in mind. All the more reason to explain why bots can be a problem, and how to write a proper bot.
    – Cerbrus
    Commented Apr 30, 2015 at 8:47
  • 1
    Questions like this can be just as helpful to the site owner being hit as they are to the user scraping the content. I'd say leave them open and provide them answers, and if possible, shoot the site owner an email with a link to the question. possibly even provide a way to prevent the method you are providing to overcome the blocking.
    – Kevin B
    Commented Apr 30, 2015 at 14:42
  • 1
    I tend to use a (slightly kinder) variant on Hanlon's razor when dealing with this sort of question: "Never attribute to malice what can be adequately explained by ignorance". Commented May 1, 2015 at 14:39
0

This started as a comment, but I'm an old man who rambles. Please forgive me.

Is the question OK?

No. I don't think it is. Aside from being of questionable legality, it (the user trying to get around the site owner blacklisting them) is unmistakably an immoral act. If the user has been enough of a nuisance for the developer of the victim website to block them, then they should stop what they're doing -- not try and continue the abuse.

Is my answer OK?

You both explain what might be the cause of the victim's actions, and then suggest ways to work around that. You later suggested the more correct course of action (contact the site owner and discuss it), I don't know how that times with the vote.

Also, a lot of people will downvote answers to bad questions as a way of trying to discourage people from answering bad questions.

Should I have answered at all?

No. That the question now has two answers has two effects:

  1. It prevents the user seeing what a stupid question they have asked and deleting it.
  2. It prevents the automated processes that exist to clean up crap deleting it.

Now we have to hope the question gets closed, and it comes across the eyes of enough 10k users to delete it.

TL;DR: Stack Overflow is meant to be a repository for quality information about the technical aspects of software development. It is not meant to aid script kiddies in their efforts of making the Internet a worse place.

7
  • You're assuming ill intentions. Maybe the poster doesn't know about rate limiting, or what kind of load a bot can cause.
    – Cerbrus
    Commented Apr 30, 2015 at 8:39
  • 2
    "An immoral act" of scraping that powers Google, Bing, DuckDuckGo, Yandex...
    – halfer
    Commented Apr 30, 2015 at 8:47
  • @halfer: How did I forget to mention those in my answer -.-
    – Cerbrus
    Commented Apr 30, 2015 at 8:49
  • 3
    @halfer Context is important. The immoral part is not the web scraping, it is trying to get around restrictions place on your use of the service by the owners of it.
    – TZHX
    Commented Apr 30, 2015 at 8:51
  • I agree about context. However, since we do not know why they are using proxies it is hard to say (and note that proxies come up in the web scraping tags quite often). See my comment on the question - I've asked why they feel their robot should be exempted from usual good-behaviour guidelines. My old answer here indicates why I think this is a complex area, in legal and ethical terms.
    – halfer
    Commented Apr 30, 2015 at 8:53
  • @Alex Did you misunderstand the word "questionable"?
    – TZHX
    Commented May 1, 2015 at 14:36
  • 1
    @Alex The scraper has been deliberately, consistently blocked by the site owner. Assisting them in getting around that, and potentially causing harm to the site owner, is not something I believe we should do. Legitimate scrapers do of cause exist, that is entirely irrelevant, because this user: a) is not one of them; and b) asked a question that is generally. I mentioned legality because it featured heavily in the comment thread on the question. I also said "Aside from" the legality part, I didn't think it was the most important point / at all important as to why the question was crap.
    – TZHX
    Commented May 1, 2015 at 14:51
0

I think you did the right thing. Always assuming good intentions might prove to be a bit naive sometimes but every tool might be used for wrongdoing and that doesn't mean it can't be learnt. About the downvote, there's always an individual that disagrees, it's kind of inevitable.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .