26

Reading the "Join Stack Overflow in Standing Up for Net Neutrality" blog post, I saw this statement:

Downloading the entire archive of questions and answers on Stack Overflow would only use about as much data as watching 10 hours of Netflix.

Immediately, I thought that couldn't be right. A quick Google search shows that Netflix can use anywhere from 3 GB to 70 GB every ten hours, but it usually uses 10. link

A quick look on the Newest Questions page on Stack Overflow shows that the most recent question is #45067396. That's 45 million questions.

wget-ing a question with three answers (which seems to be about the average) shows that each page is about 125k bytes (but some are closer to 100k, some are over 5m).

So, 125,000 bytes * 45,067,396 questions = 5,633,424,500,000 bytes = 5,633 GB. This isn't even including Meta nor any of the help pages or user profiles.

However, as mentioned by Servy, if you were to download only the Markdown of the post, it would be a dramatically smaller amount of data. Each post (Rendered Markdown only) is about 1500 bytes (ranging from about 1000 to 5000). 45,067,396 questions * 3 answers per questions * 1500 bytes per post = 202,803,282,000 = 202 GB.

Did I miscalculate, or did the author of the blog post just make a highly exaggerated statement?

Edit - I miscalculated. The author of the post was more or less right (see Oded's comment).

9
  • 5
    The amount of data in the HTML file of a question is dramatically more than the amount of information in the actual question and answer alone, by several orders of magnitude. If you're just looking at the markdown for the question and answer (and title and tags and such) you're not looking at anywhere near as much data per question.
    – Servy
    Commented Jul 12, 2017 at 21:09
  • @Servy Hm, that's true. I'll check up on the actual size and edit it into the question, one sec.
    – MD XF
    Commented Jul 12, 2017 at 21:10
  • I've pinged the author about whether or not it was a 'highly exaggerated statement'.
    – Taryn StaffMod
    Commented Jul 12, 2017 at 21:12
  • 14
    The gzipped archive of all posts on Stack Overflow (in XML format) on archive.org - the stackoverflow.com-Posts.7z file is 10.5Gb. I'd expect that to expand quite a lot.
    – Oded StaffMod
    Commented Jul 12, 2017 at 21:13
  • SO has 14m questions and 22m answers, not 45m questions and 135m answers, as you used in your numbers. Next, when looking at the actual size of the archive file 10.5 GB, and dividing by that number of posts, I got an average number of bytes per post at just over 300, not 1500 (granted that's compressed, so that's helping).
    – Servy
    Commented Jul 12, 2017 at 21:21
  • 1
    The PHP questions are the size of ‘Terrace House' :) Commented Jul 12, 2017 at 21:24
  • 1
    @Servy - don't forget - that's compressed size
    – Oded StaffMod
    Commented Jul 12, 2017 at 21:26
  • @Oded I did mention that in my comment, but of course the quote under scrutiny here is considering how much data the ISP would need to have to deal with passing through their network, so the compressed size is what's relevant here, not the uncompressed size, because people aren't downloading an uncompressed archive.
    – Servy
    Commented Jul 12, 2017 at 21:27
  • @Servy - That's fair. We do try to send everything out as gzipped.
    – Oded StaffMod
    Commented Jul 12, 2017 at 21:29

1 Answer 1

51

So, I've been hoping someone would ask about that...

A few years back, we got an email from a student at a university in Cuba. They have (or had, at the time) extremely limited Internet access - so they'd built their own local network, within which they'd constructed a mirror of Stack Overflow built from our publicly-available data dump. I remembered this, because it's pretty cool and illustrates the utility of publishing these data-dumps, and also because it's sorta tragic that students at a major educational institution would have to do something like that.

There but for the grace of God go I.

Anyway... As Oded noted, the compressed size of the most recent Stack Overflow dump is pretty close to what one might expect to use watching 10 hours of Netflix - the 10 GB estimated in your link. If I had to give up my cable Internet connection and rely on mobile data, I wouldn't be watching much Netflix - but I could still set up a mirror of Stack Overflow.

1
  • 12
    Of course, when you uncompress the (7zip) files from the public dump, parse the XML to put it into a DB of some sort - it's not even worth trying with at least 300gb of space.
    – Jon Clements Mod
    Commented Jul 12, 2017 at 22:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .