-13

As of at least September 12, 2023, it was possible to use pd.read_html to easily read Markdown tables used to create minimal reproducible examples.

However, as of this posting, it no longer seems possible to read reproducible data tables directly from Stack Overflow.

Why were changes made to Stack Overflow/Stack Exchange which break accessing markdown tables for reproducible data, or is it an (un)intended side effect of changes made on Cloudflare?

This change, after many years of working, breaks existing answers, which relied on reading the table from the question.

The point of an easy to use minimal reproducible example is to make it simple for non-expert users of SO to copy and execute the code in the question, and in the answer. Running the code in the answer was easy, scraping the body of a question with the API, is not easy for casual users trying to run the given code.

For example, the following table might be posted in a question about how to plot a dataframe.

timestamp cnt
2015-01-04 9234
2015-01-05 20372
2015-01-06 20613
2015-01-07 21064
2015-01-08 15601
2016-12-27 10842
2016-12-28 12428
2016-12-29 14052
2016-12-30 11566
2016-12-31 11424

And the following code, which can easily be copied and executed by anyone with the same question, would be posted in the answer.

import pandas as pd

# load the data from the markdown table in the OP, into a dataframe
df = pd.read_html('https://meta.stackoverflow.com/q/426625/7758804')[0]

# plot bars
ax = df.plot(kind='bar', x='timestamp')

Now results in HTTPError: HTTP Error 403: Forbidden:

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[15], line 1
----> 1 df = pd.read_html('https://meta.stackoverflow.com/q/426625/7758804')[0]

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\html.py:1245, in read_html(io, match, flavor, header, index_col, skiprows, attrs, parse_dates, thousands, encoding, decimal, converters, na_values, keep_default_na, displayed_only, extract_links, dtype_backend, storage_options)
   1229 if isinstance(io, str) and not any(
   1230     [
   1231         is_file_like(io),
   (...)
   1235     ]
   1236 ):
   1237     warnings.warn(
   1238         "Passing literal html to 'read_html' is deprecated and "
   1239         "will be removed in a future version. To read from a "
   (...)
   1242         stacklevel=find_stack_level(),
   1243     )
-> 1245 return _parse(
   1246     flavor=flavor,
   1247     io=io,
   1248     match=match,
   1249     header=header,
   1250     index_col=index_col,
   1251     skiprows=skiprows,
   1252     parse_dates=parse_dates,
   1253     thousands=thousands,
   1254     attrs=attrs,
   1255     encoding=encoding,
   1256     decimal=decimal,
   1257     converters=converters,
   1258     na_values=na_values,
   1259     keep_default_na=keep_default_na,
   1260     displayed_only=displayed_only,
   1261     extract_links=extract_links,
   1262     dtype_backend=dtype_backend,
   1263     storage_options=storage_options,
   1264 )

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\html.py:988, in _parse(flavor, io, match, attrs, encoding, displayed_only, extract_links, storage_options, **kwargs)
    977 p = parser(
    978     io,
    979     compiled_match,
   (...)
    984     storage_options,
    985 )
    987 try:
--> 988     tables = p.parse_tables()
    989 except ValueError as caught:
    990     # if `io` is an io-like object, check if it's seekable
    991     # and try to rewind it before trying the next parser
    992     if hasattr(io, "seekable") and io.seekable():

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\html.py:248, in _HtmlFrameParser.parse_tables(self)
    240 def parse_tables(self):
    241     """
    242     Parse and return all tables from the DOM.
    243
   (...)
    246     list of parsed (header, body, footer) tuples from tables.
    247     """
--> 248     tables = self._parse_tables(self._build_doc(), self.match, self.attrs)
    249     return (self._parse_thead_tbody_tfoot(table) for table in tables)

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\html.py:811, in _LxmlFrameParser._build_doc(self)
    809             pass
    810     else:
--> 811         raise e
    812 else:
    813     if not hasattr(r, "text_content"):

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\html.py:790, in _LxmlFrameParser._build_doc(self)
    788 try:
    789     if is_url(self.io):
--> 790         with get_handle(
    791             self.io, "r", storage_options=self.storage_options
    792         ) as f:
    793             r = parse(f.handle, parser=parser)
    794     else:
    795         # try to parse the input in the simplest way

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\common.py:718, in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
    715     codecs.lookup_error(errors)
    717 # open URLs
--> 718 ioargs = _get_filepath_or_buffer(
    719     path_or_buf,
    720     encoding=encoding,
    721     compression=compression,
    722     mode=mode,
    723     storage_options=storage_options,
    724 )
    726 handle = ioargs.filepath_or_buffer
    727 handles: list[BaseBuffer]

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\common.py:372, in _get_filepath_or_buffer(filepath_or_buffer, encoding, compression, mode, storage_options)
    370 # assuming storage_options is to be interpreted as headers
    371 req_info = urllib.request.Request(filepath_or_buffer, headers=storage_options)
--> 372 with urlopen(req_info) as req:
    373     content_encoding = req.headers.get("Content-Encoding", None)
    374     if content_encoding == "gzip":
    375         # Override compression based on Content-Encoding header

File ~\anaconda3\envs\py11\Lib\site-packages\pandas\io\common.py:274, in urlopen(*args, **kwargs)
    268 """
    269 Lazy-import wrapper for stdlib urlopen, as that imports a big chunk of
    270 the stdlib.
    271 """
    272 import urllib.request
--> 274 return urllib.request.urlopen(*args, **kwargs)

File ~\anaconda3\envs\py11\Lib\urllib\request.py:216, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    214 else:
    215     opener = _opener
--> 216 return opener.open(url, data, timeout)

File ~\anaconda3\envs\py11\Lib\urllib\request.py:525, in OpenerDirector.open(self, fullurl, data, timeout)
    523 for processor in self.process_response.get(protocol, []):
    524     meth = getattr(processor, meth_name)
--> 525     response = meth(req, response)
    527 return response

File ~\anaconda3\envs\py11\Lib\urllib\request.py:634, in HTTPErrorProcessor.http_response(self, request, response)
    631 # According to RFC 2616, "2xx" code indicates that the client's
    632 # request was successfully received, understood, and accepted.
    633 if not (200 <= code < 300):
--> 634     response = self.parent.error(
    635         'http', request, response, code, msg, hdrs)
    637 return response

File ~\anaconda3\envs\py11\Lib\urllib\request.py:563, in OpenerDirector.error(self, proto, *args)
    561 if http_err:
    562     args = (dict, 'default', 'http_error_default') + orig_args
--> 563     return self._call_chain(*args)

File ~\anaconda3\envs\py11\Lib\urllib\request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
    494 for handler in handlers:
    495     func = getattr(handler, meth_name)
--> 496     result = func(*args)
    497     if result is not None:
    498         return result

File ~\anaconda3\envs\py11\Lib\urllib\request.py:643, in HTTPDefaultErrorHandler.http_error_default(self, req, fp, code, msg, hdrs)
    642 def http_error_default(self, req, fp, code, msg, hdrs):
--> 643     raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: HTTP Error 403: Forbidden
7
  • 15
    Honestly, just scraping a page breaking doesn't seem like a bug. Instead, use the API to get the body of a question.
    – Erik A
    Commented Sep 26, 2023 at 18:03
  • 14
    Stack Overflow never supported this, what you were depending on was basically web scraping. If a site decides to actively prevent such scraping, reporting that as a bug doesn't seem sensible to me. See: Every site requesting verification that I'm human Commented Sep 26, 2023 at 18:11
  • 6
    If the error were something to do with actually interpreting the data, then you might have an interesting point to make about what the page is actually outputting - although it would probably not be accepted as a bug as long as it renders properly in most browsers. But HTTPError: HTTP Error 403: Forbidden is extremely explicit. This is Stack Overflow saying that it does not want to provide the requested information to your program. That is not a bug unless you can find something in the site documentation that supports a claim that it's supposed to be provided in this specific way. Commented Sep 26, 2023 at 18:27
  • 3
    They recently changed to Cloudflare, which caused some other issues. I'm not sure if the error here is intentional or not.
    – Laurel
    Commented Sep 26, 2023 at 18:34
  • 2
    Businesses switch to Cloudflare because they want its services and features. Anti-scraping is probably one of the most famous of those - surely you've seen one of those "Please enable Javascript and cookies to continue; we need to make sure your connection isn't malicious" sorts of screens before? It is extremely unlikely that this would be considered undesired. Commented Sep 26, 2023 at 18:44
  • Query to get links of posts broken due to this: data.stackexchange.com/stackoverflow/query/1787848/… (Note: This only finds posts that have the pandas tag on them, tried with the Python tag as well but that gives a timeout) Commented Sep 27, 2023 at 10:53
  • I think Perl's LWP has the same problem. A workaround is to call out to the system's Wget (or similar). Something like my $qxString = "wget -O - \"$URL\" "; my $content = qx/$qxString/; (the URL must be quoted, at least if it contains "?" (GET parameters)). And capture standard output (or through a temporary file). Commented Nov 16, 2023 at 23:32

1 Answer 1

15

Why were changes made to Stack Overflow/Stack Exchange which break accessing markdown tables for reproducible data?

Because you are not supposed to be able to do this now and it was never an advertised site feature. Sites publish APIs for a reason: so that a) you don't have to do this, and b) in exchange, you don't do this but instead use the server's interface in a way that is less demanding of their resources.

Generally speaking, if you don't see a blog article or announcement about it or anything like that, then there's no publicly available information about it. Maybe the site staff would see this question and be interested in answering; but what good does it do you? Setting the question this way comes across as a rant and disagreement with the change being made; but you haven't made a good case for rolling it back (and the above is my explanation of why it shouldn't matter to you).

But "Q. Why did site X stop allowing me to do Y? A. Because the people who operate site X do not wish for you to do Y, and they've now gotten around to doing something about it" is really as far as this ever goes.

Cloudflare's anti-bot protection is demonstrably one of its best-known features. (Consider for example that their own self-description page has a big orange help button for DDOS-attack emergencies.) I find it extraordinarily unlikely that the effect is unintended or that SE Inc. would find it undesirable. Again, if they wanted people to scrape the site, they wouldn't have provided an API.

(As a meta note, if you really want an answer about this sort of thing directly from staff, consider directly using the support email rather than posting here.)

This change...breaks existing answers, which relied on reading the table from the question.

Someone who writes code in an answer that includes something to web-scrape the same page to get data from the question is very clearly not properly answering the question. Code in questions is not meant to be self-contained, but exemplary. Answers like what you describe are hard-coded to the data in the question, but in a roundabout and hard-to-understand way that adds huge amounts of clutter.

If you find an answer like that, please edit out extraneous stuff like that, or downvote it if it isn't a clear edit. The answer is supposed to focus on the question being asked - not on setup tasks and boilerplate. If someone shows some example data to clarify the format of an input Dataframe, and then asks how to do a certain manipulation with the data, answers should assume the input is already readily available.

The only question that such web-scraping code answers is the question of how to write a Stack Overflow web scraper - which makes the question "Needs More Focus" if OP actually asked, and is completely superfluous otherwise.

The point of an easy to use minimal reproducible example is to make it simple for non-expert users of SO to copy and execute the code in the question, and in the answer. Running the code in the answer was easy, scraping the body of a question with the API, is not easy for casual users trying to run the given code.

Web scraping is not easier than using a proper API, nor copying and pasting for this situation.

If you are just trying to reproduce a Pandas example from a single question - since you will have to do some manual interpretation anyway, i.e. parse the question text as a human to figure out what to put in which file and how to run it all - this is a much easier way to get what you need:

  1. Copy and paste the contents of the table, as they appear rendered in your browser, to a plain text file.

  2. Optionally give it a .csv or .tsv extension.

  3. Use pandas.read_csv to read it.

If the example code includes a demonstration of reading code, it probably shouldn't, unless the question is actually about using Pandas I/O. In the latter case, the question needs to contain data in a compatible format to provide a properly reproducible example - in this case, the Markdown table is supplementary and for reference only.

3
  • The comment by @Laurel Commented Sep 26, 2023 at 18:41
  • 1
    "Someone who writes code in an answer that includes something to web-scrape the same page to get data from the question is very clearly not properly answering the question." I don't agree... I answer a small Tag about Web-Automation (and Data-Extraction is a "sub-set" of Web-Automation), and that's a technique I also sometimes use, exactly for the Answer/Solution to be self-contained, usually about specific Content that turns out to be "difficult" to extract (with accents or special chars or emoji's or "strange" formatting etc...) and/or for further data manipulation on the extracted data.
    – chivracq
    Commented Sep 26, 2023 at 19:37
  • 10
    @chivracq I'd disagree on using this sort of technique for questions related to Web-Automation as well. The structure of the HTML of a page can change without notice, so your answer can break quite easily. Given we expect the OP to provide a minimal sample of the HTML they are working with for scraping questions I'd expect answers to simply use that same sample. Commented Sep 27, 2023 at 5:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .