Skip to main content

Questions tagged [scraping]

The process of extracting information from the HTML source code of live websites. Typically used by third-party applications to interact with a website that doesn't expose an API.

Filter by
Sorted by
Tagged with
-9 votes
1 answer
219 views

Scraping Stack Overflow results using Beautiful Soup stopped by human verification challenge [duplicate]

I'm working on a project where I need to scrape Stack Overflow search results to retrieve relevant questions based on user input. I'm using Python with Beautiful Soup for web scraping. However, when I ...
Shamanth M Hiremath's user avatar
66 votes
1 answer
3k views

Will Stack Overflow disallow the GPT web crawler?

I do not know if Stack Overflow allows the ChatGPT Web Crawler to scrape information from this site so that it can use it to train its large language model, but OpenAI are reportedly encouraging sites ...
user avatar
-3 votes
1 answer
215 views

I want automatized site grab to extend SEDE results, how far can I go?

I want to collect properties of deleted posts not disclosed by the SEDE, but available as 10k privilege. My method would be to generate a random sample of post Ids (by SEDE queries) and then using ...
peterh's user avatar
  • 1
4 votes
1 answer
399 views

Why is `www.xszz.org` hosting exact copies of SO questions and answers [duplicate]

I have seen numerous examples, but below is one example I have just seen today, I am sure there are many more. Git: fatal: fsync error on 'sha1 file': Bad file descriptor https://www.xszz.org/...
Clay's user avatar
  • 2,686
3 votes
0 answers
101 views

Could I report some websites that blatantly mirror Stack Overflow? [duplicate]

In the last two months I had found some websites that blatantly mirror or some of them blatantly crawl the questions and answers, and provide some translations. Is this ok to report here? For ...
Eriawan Kusumawardhono's user avatar
2 votes
0 answers
151 views

Is https://stackoverrun.com/ an illegal page that takes info from stackoverflow.com in order to cheat users? [duplicate]

I found this question on an site that already was unknown for me. It seems it is kind of a phishing site https://stackoverrun.com/es/q/7514014 The evidence is that the question is the same that ...
Alejandro Teixeira Muñoz's user avatar
-2 votes
1 answer
1k views

How can I crawl and index Stack Overflow? [duplicate]

I have looked everywhere, robots.txt and on the site. I want to crawl and index Stack Overflow. What are the policies and procedures for doing this correctly? Timeouts and then subsequent ...
Rusty Nail's user avatar
  • 2,702
3 votes
0 answers
77 views

See this beautifull scraper [duplicate]

I was wondering if some tag excerpt was plagiarised, so I entered it into Google's search engine. It turned out there was no result for the copied excerpt besides on two SO pages, and one hit on a ...
Luuklag's user avatar
  • 3,916
65 votes
1 answer
864 views

Received emailed survey

I received the following email: Subject: Dear Stack Overflow user, we sincerely invite you to participate_in_an_anonymous_    survey_about_the_incentive_mechanisms_on_Stack_Overflow [...
ChrisInEdmonton's user avatar
60 votes
2 answers
4k views

Can I scrape Stack Overflow job postings? [closed]

Am I allowed to scrape Stack Overflow job postings? I love Stack Overflow and don't want to do anything that will get me in trouble.
Frank Harb's user avatar
10 votes
1 answer
392 views

Are there existing tools to help track a tag and gather statistics?

Lots of tags are monitored by the people/companies who helped build those products. Are there existing tools to help gather basic statistics about a tag? Such as: Number of questions asked per (day, ...
Shawn Tabrizi's user avatar
8 votes
0 answers
179 views

stackoverflow.com, automated cloning to other languages (chinese: helplib.com/qa)

I previously answered a question with a function from my personal JS library. And out of laziness used google to look it up (for work) the rather unique function name simpleRecusiveDeepEqual and its ...
PicoCreator's user avatar
  • 10.1k
3 votes
0 answers
164 views

Official SO policy on answering questions related to Web scraping [duplicate]

I would like to know what is the SO policy on answering questions related to the web scraping (crawling and similar procedure). As an SO user, I am always ready to answer questions relating to web ...
user227710's user avatar
  • 3,184
4 votes
0 answers
107 views

Another scraper: http://askproblem.com/ [duplicate]

I was searching google for a solution to my problem - and found a stackoverflow question about it - Application Cache Error event: Manifest fetch failed (4). Answers in that question didn't help me - ...
Aleks G's user avatar
  • 57.2k
1 vote
0 answers
90 views

What is stacktheoverflow.com? [duplicate]

Recently I was looking for tips on how to do some programming things, so I used Google to help aid in my search. Several Stack Overflow questions appeared as a result of my input query, but as I ...
TNT's user avatar
  • 2,920
106 votes
1 answer
6k views

Are sites that autonomously scrape Stack Overflow for answers to programming problems encouraged?

I saw this, and it completely blew my mind: http://gkoberger.github.io/stacksort/ It takes a problem, and a set of unit tests, then scans Stack Overflow for answers and applies them the best it ...
Eric Leschinski's user avatar
2 votes
0 answers
98 views

Possible Stack Overflow phishing site - Searchalleasy.com [duplicate]

Searchalleasy.com seems to have cloned StackOverflow and is actually coming up higher in some Google searches (didn't even realize at first that it wasn't Stack Overflow). Seems to be a very current (...
Arkaine55's user avatar
  • 548
24 votes
2 answers
592 views

How prominent does attribution need to be?

While looking for an answer to a question I had, using this Google search, I found: A StackOverflow question, right at the top of the results; A scraped version of the question and similar questions ...
ArtOfCode's user avatar
  • 5,702
0 votes
0 answers
69 views

What is the site youranswers.info and why does it have copies of questions and answers on stackoverflow/superuser etc? [duplicate]

I noticed that the site http://youranswers.info had copies of questions, answers and comments of stackoverflow and superuser questions (it might probably have questions on other stackexchange sites as ...
slayernoah's user avatar
  • 4,462
2 votes
0 answers
84 views

2 scrapers found in the same search [duplicate]

When doing a following google search for a problem I'm having, I got 4 results. The top 2 results were to stack overflow questions. the next 2 results were strangely familiar. It turns out, they ...
Sam I am says Reinstate Monica's user avatar
3 votes
1 answer
296 views

Tiku (answers scraped from SO) comes before SO for me [duplicate]

If I run this search. I get these results: Two from Django's Docs, Two from SO, and two from tiku, a site that's scraping SO questions. AFAIK scraping is ok, and they do attribute back. But is this ...
AncientSwordRage's user avatar
4 votes
0 answers
118 views

Can Quabr Scraping be Prevented? [duplicate]

I noticed Quabr.com had scraped one of my questions: http://quabr.com/27069548/is-tr2-going-to-be-released-in-c17 Does StackOverflow do anything to defend against scraping? Can anything be done to ...
Jonathan Mee's user avatar
  • 38.7k