-9

I always face hard time to extract SO post content from the CSV file returned by the Stack Exchange Data Explorer. CSV file is comma-separated, but the structure is easily defeated by the unstructured textual content of SO post. It contains commas, quotes everything. Did anybody face this issue? If yes, please share. Using data dump is a choice, but that is a lot of work IMO.

5

1 Answer 1

4

I fail to see the problem.

Yes, titles and content contain comma's and quotes. But (1) they are quoted in full (so the comma's are not a problem), and (2) double quotes inside content are doubled (and so your CSV parser needs to handle that).

  1. I ran your query, which created a 50,001-line CSV (59.5MB)
  2. I replaced all even number of "" with nothing, then
  3. removed all (now) single-quoted "..." strings.

I ended up with 1 line, the CSV header, so there are no encoding errors in the CSV text. It is possible the few entries that contain hard returns inside quoted text threw off the CSV viewer that you use.

Use some other software to view your file with. The built-in CSV viewer on my lowly iPad, for example, has no problems at all and interprets the CSV correctly, and so does OSX's QuickLook Preview.

7
  • Well, that is only 1000 rows. Can you try the same with 50000 rows and report? Jun 1, 2015 at 22:23
  • I don't believe the number of rows make a difference in how each single row is stored. It is possible that you encountered a case where the quoting got messed up due to the content, but without any further evidence it's more likely your CSV reading tool could not handle the formatting in general.
    – Jongware
    Jun 1, 2015 at 22:31
  • CSV-based text content output is just a bad idea, probably XML is a preferable option. But DE does not support that. Jun 1, 2015 at 22:35
  • 4
    @MasudRahman: I was sure the "XML solves everything" fad died some time ago... The only advantage using XML would have is considerably bigger files. Jun 2, 2015 at 0:46
  • @Jongware Something is going on with this data. Even Java Scanner cannot read the whole file, and terminates unexpectedly. Jun 2, 2015 at 6:02
  • 1
    @Masud, That's probably a problem with Java not handling large text files correctly. I'd ask this on SO. Jun 2, 2015 at 7:59
  • For laffs, I wrote a simple C program to parse the data file according to these simple rules. It successfully parsed 50,000 lines of data and about the only thing worth mentioning is that each data line ends with a CR/LF pair, but 'internal' newlines are LF only. time tells me reading and 'parsing' this >50Mb file took only 0.506s of user time.
    – Jongware
    Jun 4, 2015 at 22:49

Not the answer you're looking for? Browse other questions tagged .