-3

I want to collect properties of deleted posts not disclosed by the SEDE, but available as 10k privilege. My method would be to generate a random sample of post Ids (by SEDE queries) and then using scripts to download the content (by logging in the site with my browser and then using my ordinary session cookie (acct=...) also in the script).

However, to my best knowledge, SE content rules at least dislike automatized site grabbing, so I do not want to do that.

However, I think, the purpose of these rules is to prevent (or to harden) data fishing or server overloads. My goal is to use the grabbed data in meta/chat discussions and I have absolutely no intent to grab anything without permission. But I have no way to prove it.

What can I do? Is it allowed or not? How far can I go?

Can I download 100 deleted question pages by a script? Can I do 10000? Or how much?

P.s. I never did such things until now and won't ever do if it is not allowed.

62
  • 3
    Scraping will get you hard rate limited real fast. The API is how you go about it. I also doubt SE is going to appreciate scraping. Not sure what they do about excessive and extensive scraping attempts (read: Not sure if employees get involved), but the automatic (hard) rate limit is going to slow you down substantially if you try. Commented Feb 8, 2022 at 14:54
  • 1
    @HereticMonkey Doesn't the data dump still exclude the exact same info about deleted posts as SEDE? Commented Feb 8, 2022 at 15:03
  • 2
    Technically you can scrape whatever you want and nobody can stop you. SE has implement rate limits, which prevent you from scraping this information quickly. An API is better but not all information is present there. I believe that as long as you do not abuse the resources and get rate-limited, it's ok to scrape the site.
    – Dharman Mod
    Commented Feb 8, 2022 at 15:08
  • 4
    Rate limits don't happen on the browser side. The server doesn't care if you're browsing or scraping.
    – Cerbrus
    Commented Feb 8, 2022 at 15:13
  • 4
    So that means you will use the same rate limits as mentioned in that article.
    – Dharman Mod
    Commented Feb 8, 2022 at 15:14
  • 1
    The server doesn't know that you are scraping the data. The only way to figure this out would be if your IP gets rate limited very often. That could mean that you are using some automation to visit SE pages. I doubt anyone would get suspended over that unless it would be some distributed DDoS attack.
    – Dharman Mod
    Commented Feb 8, 2022 at 15:16
  • 3
    It doesn't really matter IMO. That link is the information you have. Whatever else you get here is speculation. You'd have to go to the actual owners of the site with that question to get any kind of truthful answer.
    – Gimby
    Commented Feb 8, 2022 at 15:18
  • 1
    And here is an official response in this matter on Meta.SE meta.stackexchange.com/questions/443/…
    – Dharman Mod
    Commented Feb 8, 2022 at 15:19
  • 3
    They only block scrapers to protect themselves. If you're being very reasonable while scraping there's not going to be an issue. Scrape 100 posts an hour or so for a few days and you'll have several thousand without even raising an alarm. Commented Feb 8, 2022 at 15:35
  • 1
    @KevinB For everything. Here are my public SEDE queries, these help me a lot, but I need more.
    – peterh
    Commented Feb 8, 2022 at 15:42
  • 2
    The SE API is not that bad to use, and you can get a high daily quota by registering for a key Commented Feb 8, 2022 at 15:46
  • 4
    Then simply follow them. What's the problem? If you're hit by a rate limit, you will be limited. If you do it repeatedly or heavily enough, you'll lose access entirely.
    – Kevin B
    Commented Feb 8, 2022 at 15:55
  • 1
    I did read your question, and this has been said quite a few times now
    – Kevin B
    Commented Feb 8, 2022 at 15:56
  • 4
    SE blocks scrapers that don't follow the rate limits. Otherwise, they aren't being detected. #logic
    – Kevin B
    Commented Feb 8, 2022 at 15:57
  • 3
    Looks like OP overdid the scraping? Or why are they suspended until Jun 9?
    – Lino
    Commented Feb 9, 2022 at 10:22

1 Answer 1

-6

Summarizing the comments into an answer:

There are no clear limits, but doing this is very likely not the intended usage of the site.

7
  • 2
    I don't think it's your intention, but you have outright ignored much of the nuance of the conversation so far. Via your linked post (which is really old, it hasn't substantially changed or been updated since 2009, and thus may not be current), scrapers are blocked when they don't play by the rules. The Complete Rate-Limiting Guide, which has been linked several times now, documents all public rate limits for the site and API.
    – zcoop98
    Commented Feb 8, 2022 at 16:07
  • You seem to be looking for permission, someone to tell you "yes, go ahead". You're not going to get that, we simply don't have to authority to give you that. Scraping is a grey area. You'll probably be fine, as long as you play nice by the rules that have been pointed out to you. Otherwise, whether this works or doesn't, whether you get rate limited or not, is completely up to you.
    – zcoop98
    Commented Feb 8, 2022 at 16:08
  • Other pertinent links: Stack Exchange Network Acceptable Use Policy | Stack Exchange, Inc. API Terms of Use
    – zcoop98
    Commented Feb 8, 2022 at 16:13
  • @zcoop98 I wrote about 3 times in the comment-chat above: one thing what is technically possible, and another what is allowed by the rules. This question asks for, what is allowed.
    – peterh
    Commented Feb 8, 2022 at 16:14
  • @zcoop98 These look useful resources to create a better answer, or improve this one. I will do that soon.
    – peterh
    Commented Feb 8, 2022 at 16:15
  • As has been said multiple times already, there is no difference between scraping and browsing, as far as rate limiting repercussions are concerned. "What is allowed" is whatever the rate limits allow you to do... What more are you asking? Your answers are in the linked questions.
    – Cerbrus
    Commented Feb 8, 2022 at 16:24
  • @Cerbrus Yes, there is. Read the resources in the post and in the comments.
    – peterh
    Commented Feb 8, 2022 at 16:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .