13

Let me start by saying that I am aware that several similar-sounding questions have been asked, but I have reviewed the answers supplied in all of those I could find and none of them appear to me to apply to my particular case.

I have a very large file in a format that I can not read or work with (Stata .dta). I have conversion software to get it into R (haven) But the file is too big to put in my RAM (about 30 GB). So I want to post a question about some way of breaking in into pieces and loading it piecewise into R, and thence into a database.

However, I do not know how to make a reproducible example for this. It seems like a would need a copy of Stata to subset the original file. I could read a set of lines and output the lines, maybe, but I don't think this preserves the header information regarding, e.g., factor levels.

What should one do in such a case? If I can somehow break off a piece of more reasonable size, is there a way to upload it so that someone who wants to help can get at it? Is a dropbox link acceptable in this very specific set of circumstances? Should I try to find some arbitrary small Stata file, with no guarantee that the data is similar to my data?

5
  • 11
    Well in your case, the Example seems all found, as well as enough Minimalistic, probably very Reproduceable, and very Complete in its sole theoric explanation you gave here. Congratulations, you've got a good theoretical question with an MCVE, no need for this data.
    – Kaiido
    Commented May 5, 2018 at 9:56
  • 4
    Pretty important to focus on the specific problem you want to solve. Right now you can't seem to decide to get somebody else to test the code for you since you can't do it yourself anymore or ask for help on how to break up a large file. Not having enough RAM is not a problem, it is an opportunity. That's how programmers get their boss to approve a nice shiny new machine. Commented May 5, 2018 at 11:58
  • To be fair, it looks like 2 different questions: how to split the file and how to read it chunk by chunk in R. I'd focus on the first question, and once with the answer to the first, go to the second part. In that case I would say you have a MCVE, as other people knowing this file format should be able to answer the first question, and with that, people knowing R will be able to answer the 2nd one. Separation of Concerns helps!
    – Adonis
    Commented May 7, 2018 at 9:06
  • 3
    Is it possible to write a simple generator for the big file? (You can consider that a form of compression). An alternative for trying out answers is to use a much smaller input, but limit the process's resident set size and/or virtual address space (on Linux, that would be ulimit -m and ulimit -v respectively) to get the same effect. Commented May 7, 2018 at 10:40
  • There is a tosql plugin for stata, maybe you can convert it to an sql database which makes it easier to parse in batches? Commented May 7, 2018 at 11:07

3 Answers 3

20

A MCVE is a useful concept that can help enormously especially for debugging questions; it is not an absolute requirement that every question has one.

The Help Center specifically states:

Not all questions benefit from including code. But if your problem is with code you've written, you should include some. (...) Include just enough code to allow others to reproduce the problem.

To me it looks like your question just happens to be one of those questions where MCVE is not a good (or even viable) approach. But that does not mean that it isn't allowed on SO, or that it isn't answerable.

1
  • 2
    Normally I don't have problems with people misinterpreting the MCVE thing, but I think quite a few high rep users could learn you don't need an MCVE for every question Even if a question has an example, it need not be MCVE if it isn't about debugging. I've only run into this twice, though thankfully not recently. Its funny when I have no problems answering these types of questions, but mister 20k gets really mad that my non debugging question that asks about how one would do something and requires an MCVE to understand anything... I wouldn't be asking the question if I could amigo.
    – Krupip
    Commented May 7, 2018 at 15:45
7

This is why I have a personal problem with MCVE. It fosters a scenario in which the problem you wish to solve becomes unaskable because you cannot fulfill our criteria when it would otherwise be perfectly on-topic.

Let's really get to the spirit of what it is you're asking instead.

I have a large file I need to work with, but I can't fit the entirety of it into memory. I'm working in R. How would I approach this?

Your initial thought is right; breaking it into pieces that can fit into memory is probably the right approach. You're merely now looking for the right way to go about doing that.

If I were you, I'd phrase the question almost how you already have it here.

I have a very large file in a format that I can not read or work with (Stata .dta). I have conversion software to get it into R (haven), but the file is too big to put in my RAM (about 30 GB). Is there a specific approach I can take to break my data file up so that it can fit in memory through R?

If someone wanted to reproduce this exact scenario on your behalf - complete with a large data set - there's a very good chance that they haven't had to solve this problem before, and that their answer would be a literal guess. My impression is that the experts before you who have actually addressed this problem would be able to understand it and answer it for you.

If it gets closed before then...direct them to this Meta question. It'd be useful to understand why they felt like this question needed to have more in order for them to answer.

17
  • 13
    My DV for this: "This is why I have a personal problem with MCVE." Although I pretty much agree with the rest of your answer, I strongly disagree with this initial sentence. Some edge cases like discussed here don't invalidate the whole concept and usefulness of MCVEs in the common case (or, if you think so, you'd have to explain why, IMHO)
    – user2371524
    Commented May 5, 2018 at 9:33
  • Nobody has a problem with the concept. But is must not be considered a dogma. Often a verbal descrition will do. Often an image. So, would you aggree if the first sentende were stated as This is why I have a personal problem with MCVE as a dogma. ?
    – TaW
    Commented May 5, 2018 at 9:41
  • 2
    @TaW yes, because it never was meant as such. MCVE is required by site rules only for questions asking why some code doesn't work as expected. (which is according to my experience the vast majority of on-topic questions here ... but sure there are on-topic questions about different things)
    – user2371524
    Commented May 5, 2018 at 9:44
  • 2
    IMM the main goal of a so called MCVE is that OP takes the firsts steps into the debugging process: clears out everything that is not needed from the issue. If the problem needs a 3TB file in order to be reproduceable, then stating it is enough, of course there is no need to add said file, as long as OP is sure that the provided code is an MCVE: if I pass such file to the script, then I get the same output.
    – Kaiido
    Commented May 5, 2018 at 9:50
  • @Kaiido In fact no, mcve clearly include an exemple of input and output if this is needed to be "complete". We can't believe OP that the input is "ok". I have already see tons of questions where the problem was in the input.
    – Stargateur
    Commented May 5, 2018 at 10:19
  • @Stargateur that doesn't conflict with my point: an mcve's main goal is that OP takes the first steps in debugging, so that volunteers don't have to dig in there. If the input is the broken part, then enough "no-repro" comments will probably lead to the typo I'm sorry made you lost tons of time.
    – Kaiido
    Commented May 5, 2018 at 10:24
  • "It fosters a scenario in which the problem you wish to solve becomes unaskable because you cannot fulfill our criteria." I mean… yeah. That's kind of exactly what a close reason is for. That same sentence can be dropped in to describe the fallout of library recommendations, of "write-the-whole-program-for-me-broad" questions, and so on and so forth. The fact that we have criteria that make some questions unaskable on this site is absolutely status-bydesign. Commented May 5, 2018 at 10:54
  • 1
    @NathanTuggy: You misunderstand my point. Do you mean to tell me that an expert in R cannot answer a question about breaking up a substantially large data set and have it read piecewise into their application without having a sufficiently large data set for them to demonstrate on? It's this narrow thinking of why MCVE is needed on everything which is causing this very real friction.
    – Makoto
    Commented May 5, 2018 at 16:26
  • 1
    @FelixPalmen: Did you misinterpret my answer as a complete dismissal of MCVE? I'm actually quite a vocal supporter of its intended spirit; show us what you've done and where you're stuck. This is simply put, one case in which drumming up dummy data is not useful to demonstrate anything. I'd also doubt the expertise of anyone versed in R that can't solve this problem without first having the OP's 30GB data set to operate on.
    – Makoto
    Commented May 5, 2018 at 16:28
  • @Makoto: Sure, but your phrasing was (and is) quite confusingly over-broad. I'd agree that in this case a traditional MCVE is probably unhelpful. But as stated, it sounds like you're objecting to all cases in which lack of MCVE is the only problem we have with a question and therefore it is closed. But you aren't; you're objecting to cases in which MCVEs aren't relevant. Commented May 5, 2018 at 17:41
  • @NathanTuggy: Yep. And there a more subtle point to be had here. Instead of focusing on the actual question the OP wants to write, they're preoccupied with formulating the ideal MCVE. Good questions beget good examples where necessary. If we've got OPs too focused on an MCVE, we've permanently harmed their predisposition on asking questions here.
    – Makoto
    Commented May 5, 2018 at 17:47
  • 3
    There is nothing confusing about Makoto's answer. The MCVE close reason is being applied - nay, misapplied - dogmatically, on questions where an MCVE isn't even absolutely essential to getting them answered. Makoto isn't taking issue with the fact that not all questions are answerable or on-topic. They're taking issue with the fact that a perfectly answerable and reasonably scoped question becomes not just because you don't see a block of code you can copy and paste wholesale into your IDE.
    – BoltClock Mod
    Commented May 6, 2018 at 4:05
  • 3
    I sympathize with the fact that most askers are unreliable narrators and simply fail to present their problem exactly as it is from the get-go, either through a bad description, providing a code snippet that ends up failing to illustrate the problem, or even circumstances out of their control. Ask for an MCVE then. Otherwise, if the problem is presented clearly, with code that illustrates the problem to a reasonable degree, I'd trust an expert to be able to answer that question effectively (or tell if the information presented is insufficient to do so).
    – BoltClock Mod
    Commented May 6, 2018 at 4:13
  • 3
    probably 90% of the time, if you force someone to think critically about what they are doing enough to create an MCVE, they will solve their problem themselves in the process. Suggesting they use a step-debugger in the process moves the possible success rate very close to 100% in most cases. Just because there is even a 10% edge cases that might not be able to MCVE'd does not mean it still should not be the first requirement. It just means they need to put work in to explain it in even more detail to make it answerable.
    – user177800
    Commented May 6, 2018 at 17:55
  • 1
    @feelingunwelcome: Again, can you demonstrate that an expert in R would not be able to at least understand this scenario without an MCVE? I genuinely don't disagree with using them, I just don't think that in this case, we need to be preaching it as much as we do. MCVE is not a cure-all for questions or problem solving. We shouldn't be treating it like it is.
    – Makoto
    Commented May 6, 2018 at 17:58
-6

Actually, I misunderstand your question:

How to create a reproducible example when converting a very large file I am unable to manipulate directly?

Misleading me. My first answer was just focus about "How to create a reproducible example when converting a very large file", my brain just skip the rest because this is impossible. You can't ask how to convert a file to something without be able to manipulate directly. So, I just somehow ignore the end of your sentence.

Actually, I don't see any problem to just ask "a question about some way of breaking in into pieces and loading it piecewise into R" without any MCVE.

2
  • 1
    Actually, I don't see a need to delete downvoted posts on meta, as votes here aren't really about quality but more about agreement / disagreement :) So thanks for undeleting!
    – user2371524
    Commented May 5, 2018 at 10:14
  • 1
    @FelixPalmen I deleted because I totally miss understand the question...
    – Stargateur
    Commented May 5, 2018 at 10:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .