Questions tagged [scraping]
The process of extracting information from the HTML source code of live websites. Typically used by third-party applications to interact with a website that doesn't expose an API.
22
questions
-9
votes
1
answer
219
views
Scraping Stack Overflow results using Beautiful Soup stopped by human verification challenge [duplicate]
I'm working on a project where I need to scrape Stack Overflow search results to retrieve relevant questions based on user input. I'm using Python with Beautiful Soup for web scraping. However, when I ...
66
votes
1
answer
3k
views
Will Stack Overflow disallow the GPT web crawler?
I do not know if Stack Overflow allows the ChatGPT Web Crawler to scrape information from this site so that it can use it to train its large language model, but OpenAI are reportedly encouraging sites ...
-3
votes
1
answer
215
views
I want automatized site grab to extend SEDE results, how far can I go?
I want to collect properties of deleted posts not disclosed by the SEDE, but available as 10k privilege. My method would be to generate a random sample of post Ids (by SEDE queries) and then using ...
4
votes
1
answer
399
views
Why is `www.xszz.org` hosting exact copies of SO questions and answers [duplicate]
I have seen numerous examples, but below is one example I have just seen today, I am sure there are many more.
Git: fatal: fsync error on 'sha1 file': Bad file descriptor
https://www.xszz.org/...
3
votes
0
answers
101
views
Could I report some websites that blatantly mirror Stack Overflow? [duplicate]
In the last two months I had found some websites that blatantly mirror or some of them blatantly crawl the questions and answers, and provide some translations.
Is this ok to report here?
For ...
2
votes
0
answers
151
views
Is https://stackoverrun.com/ an illegal page that takes info from stackoverflow.com in order to cheat users? [duplicate]
I found this question on an site that already was unknown for me. It seems it is kind of a phishing site
https://stackoverrun.com/es/q/7514014
The evidence is that the question is the same that ...
-2
votes
1
answer
1k
views
How can I crawl and index Stack Overflow? [duplicate]
I have looked everywhere, robots.txt and on the site. I want to crawl and index Stack Overflow.
What are the policies and procedures for doing this correctly?
Timeouts and then subsequent ...
3
votes
0
answers
77
views
See this beautifull scraper [duplicate]
I was wondering if some tag excerpt was plagiarised, so I entered it into Google's search engine. It turned out there was no result for the copied excerpt besides on two SO pages, and one hit on a ...
65
votes
1
answer
864
views
Received emailed survey
I received the following email:
Subject: Dear Stack Overflow user, we sincerely invite you to participate_in_an_anonymous_
survey_about_the_incentive_mechanisms_on_Stack_Overflow [...
60
votes
2
answers
4k
views
Can I scrape Stack Overflow job postings? [closed]
Am I allowed to scrape Stack Overflow job postings?
I love Stack Overflow and don't want to do anything that will get me in trouble.
10
votes
1
answer
392
views
Are there existing tools to help track a tag and gather statistics?
Lots of tags are monitored by the people/companies who helped build those products.
Are there existing tools to help gather basic statistics about a tag? Such as:
Number of questions asked per (day, ...
8
votes
0
answers
179
views
stackoverflow.com, automated cloning to other languages (chinese: helplib.com/qa)
I previously answered a question with a function from my personal JS library. And out of laziness used google to look it up (for work) the rather unique function name simpleRecusiveDeepEqual and its ...
3
votes
0
answers
164
views
Official SO policy on answering questions related to Web scraping [duplicate]
I would like to know what is the SO policy on answering questions related to the web scraping (crawling and similar procedure). As an SO user, I am always ready to answer questions relating to web ...
4
votes
0
answers
107
views
Another scraper: http://askproblem.com/ [duplicate]
I was searching google for a solution to my problem - and found a stackoverflow question about it - Application Cache Error event: Manifest fetch failed (4). Answers in that question didn't help me - ...
1
vote
0
answers
90
views
What is stacktheoverflow.com? [duplicate]
Recently I was looking for tips on how to do some programming things, so I used Google to help aid in my search. Several Stack Overflow questions appeared as a result of my input query, but as I ...
106
votes
1
answer
6k
views
Are sites that autonomously scrape Stack Overflow for answers to programming problems encouraged?
I saw this, and it completely blew my mind:
http://gkoberger.github.io/stacksort/
It takes a problem, and a set of unit tests, then scans Stack Overflow for answers and applies them the best it ...
2
votes
0
answers
98
views
Possible Stack Overflow phishing site - Searchalleasy.com [duplicate]
Searchalleasy.com seems to have cloned StackOverflow and is actually coming up higher in some Google searches (didn't even realize at first that it wasn't Stack Overflow).
Seems to be a very current (...
24
votes
2
answers
592
views
How prominent does attribution need to be?
While looking for an answer to a question I had, using this Google search, I found:
A StackOverflow question, right at the top of the results;
A scraped version of the question and similar questions ...
0
votes
0
answers
69
views
What is the site youranswers.info and why does it have copies of questions and answers on stackoverflow/superuser etc? [duplicate]
I noticed that the site http://youranswers.info had copies of questions, answers and comments of stackoverflow and superuser questions (it might probably have questions on other stackexchange sites as ...
2
votes
0
answers
84
views
2 scrapers found in the same search [duplicate]
When doing a following google search for a problem I'm having, I got 4 results.
The top 2 results were to stack overflow questions.
the next 2 results were strangely familiar. It turns out, they ...
3
votes
1
answer
296
views
Tiku (answers scraped from SO) comes before SO for me [duplicate]
If I run this search. I get these results:
Two from Django's Docs, Two from SO, and two from tiku, a site that's scraping SO questions.
AFAIK scraping is ok, and they do attribute back.
But is this ...
4
votes
0
answers
118
views
Can Quabr Scraping be Prevented? [duplicate]
I noticed Quabr.com had scraped one of my questions: http://quabr.com/27069548/is-tr2-going-to-be-released-in-c17
Does StackOverflow do anything to defend against scraping? Can anything be done to ...