I want to write a question asking why a data set composed of hundreds of csv files, which is 1.99 Gb on HD, becomes 3.8 Gb when loaded in memory (R). I can provide the full code (it's not much, basically a read_list
function and a main which calls it), but how can I create a reproducible example?
I cannot share the original data on a shared drive for IP reasons, and I don't know how to generate a similar data set without wasting weeks. Each file has the same headers, but widely different number of rows (from 1 to thousands). Also, some columns which are empty in a file are not empty in another and vice versa. The reason why I want to ask this question is because I wonder if the excess memory usage is due to some mistake on my part in writing and using the read_list
function.
Ps I specify that the language I'm using is R. I wrote it above, but probably it was too cryptic.