22

I've been working on a C# wrapper for extracting and playing with the Stack Exchange Data Dump, with mixed results. Specifically, here are the results:

Successful: 135 
Index out of Range Exception: 5 
Failed to Read: 215

I'm using the SharpCompress library to read the 7z files, and it appears for the 215 it's unable to read, it's because they don't conform to the normal 7z signature:

enter image description here

Instead, they don't have anything for the first 0x005FFFF0 Address bytes, but at address 0x00600000, they have this:

enter image description here

Which doesn't conform to the signature for 7z files.

Because of this, SharpCompress is unable to decompress them.

Since some SE data dump zip files are affected, but not all; I thought this was a valid question to post here:

What is going on with the data dump that some files conform to the specification, but not all?

Here are just some of the files that don't conform (see the github issue for all, since there are quite a few):

academia.stackexchange.com.7z android.stackexchange.com.7z
anime.stackexchange.com.7z apple.stackexchange.com.7z askubuntu.com.7z
aviation.stackexchange.com.7z bicycles.stackexchange.com.7z
biology.stackexchange.com.7z bitcoin.stackexchange.com.7z
blender.stackexchange.com.7z boardgames.stackexchange.com.7z
bricks.stackexchange.com.7z buddhism.stackexchange.com.7z
chemistry.stackexchange.com.7z chess.stackexchange.com.7z
christianity.stackexchange.com.7z codegolf.stackexchange.com.7z
codereview.stackexchange.com.7z cogsci.stackexchange.com.7z
cooking.stackexchange.com.7z crypto.stackexchange.com.7z
cs.stackexchange.com.7z cstheory.stackexchange.com.7z
dba.stackexchange.com.7z diy.stackexchange.com.7z

Here's the complete list

13
  • 5
    Posted here instead of general SE meta for two reasons: 1) More programmers frequent this site; 2) Be jealous of your own content, you know? Mar 25, 2016 at 2:22
  • 2
    Got a direct link to one of these problem files (preferably a small one)? Tried downloading archive.org/download/stackexchange/diy.stackexchange.com.7z and is seemed normal. Mar 25, 2016 at 2:43
  • I assume you downloaded these from the torrent... Are you sure it was finished?
    – Shog9
    Mar 25, 2016 at 2:47
  • @shog9 Yes; was using uTorrent, and apparently sometimes torrents have padding issues? torrentfreak.com/bitcomet-pollutes-bittorrent-with-junk-data Mar 25, 2016 at 2:59
  • @AlexanderO'Mara Here's the smallest 7z file that fits the description: github.com/gortok/SoDataDumpExtractor/blob/master/7z/… Mar 25, 2016 at 3:07
  • 2
    That file you linked to has a section of null bytes in the middle. archive.org/download/stackexchange/chess.stackexchange.com.7z is fine. Sounds like that block of the torrent was not downloaded, or perhaps not flushed to disk. Or maybe it's just uTorrent being the Internet Explorer of torrent clients. Mar 25, 2016 at 3:11
  • @AlexanderO'Mara You're right. I'd spot checked some of the files and they all displayed the null bytes at the beginning; I didn't think to check this particular file and it has the null bytes in the middle. Sounds like the torrent client was being bad. Of course, it did say everything was downloaded; soooo. Mar 25, 2016 at 3:14
  • 3
    Did a quick check using uTorrent, can't reproduce.
    – Shog9
    Mar 25, 2016 at 4:13
  • 7
    Your BitTorrent client should have an option to "recheck files" or something like that. That should be able to reliably detect if you have had a download error, and cause those pieces to be redownloaded. But FYI if you're trying to download a non-latest dump, the Internet Archive's mis-configured web seeds will send you incorrect data, and your torrent can be corrupted by that. (Probably not with null bytes like this, though.)
    – Jeremy
    Mar 25, 2016 at 4:33
  • Kinda ironic that a moderators post is put on hold as off topic. (sorry George :P) Mar 25, 2016 at 10:38
  • @BhargavRao Was it re-opened?
    – AStopher
    Mar 26, 2016 at 18:10
  • @cybermonkey Yep meta.stackoverflow.com/posts/319705/revisions Mar 26, 2016 at 18:16

1 Answer 1

12

Instead of using uTorrent the second time, I used Deluge. I was able to use the code provided at the link to decompress all of the files successfully; it took around 5 hours and 210GB of disk space to do so.

As it stands, this appears to be a one-off issue related to the initial download; not any problem related to how Stack Exchange zips/transmits the data to archive.org.

1
  • If I am only interested in the posts.7z file am I supposed to download a particular padding file? Using utorrent but after reading this I guess I should switch to deluge.
    – Sharun
    Mar 18, 2017 at 13:44

Not the answer you're looking for? Browse other questions tagged .