I've been working on a C# wrapper for extracting and playing with the Stack Exchange Data Dump, with mixed results. Specifically, here are the results:
Successful: 135
Index out of Range Exception: 5
Failed to Read: 215
I'm using the SharpCompress library to read the 7z files, and it appears for the 215 it's unable to read, it's because they don't conform to the normal 7z signature:
Instead, they don't have anything for the first 0x005FFFF0 Address bytes, but at address 0x00600000, they have this:
Which doesn't conform to the signature for 7z files.
Because of this, SharpCompress is unable to decompress them.
Since some SE data dump zip files are affected, but not all; I thought this was a valid question to post here:
What is going on with the data dump that some files conform to the specification, but not all?
Here are just some of the files that don't conform (see the github issue for all, since there are quite a few):
academia.stackexchange.com.7z android.stackexchange.com.7z
anime.stackexchange.com.7z apple.stackexchange.com.7z askubuntu.com.7z
aviation.stackexchange.com.7z bicycles.stackexchange.com.7z
biology.stackexchange.com.7z bitcoin.stackexchange.com.7z
blender.stackexchange.com.7z boardgames.stackexchange.com.7z
bricks.stackexchange.com.7z buddhism.stackexchange.com.7z
chemistry.stackexchange.com.7z chess.stackexchange.com.7z
christianity.stackexchange.com.7z codegolf.stackexchange.com.7z
codereview.stackexchange.com.7z cogsci.stackexchange.com.7z
cooking.stackexchange.com.7z crypto.stackexchange.com.7z
cs.stackexchange.com.7z cstheory.stackexchange.com.7z
dba.stackexchange.com.7z diy.stackexchange.com.7z
uTorrent
, and apparently sometimes torrents have padding issues? torrentfreak.com/bitcomet-pollutes-bittorrent-with-junk-data