2

I am asking about the following tags:

My problem is that I have a small query that works on small data sets, but when I use it on 2.5 GB of data, I get an error in Fuseki. I have already asked the official support but got no reply.

How can I ask that here? No one will download 2.5 GB of data.

Update

The null pointer exception is happening on the tool I use, Fuseki, not on my code, which is just a SparQL query.

11
  • 1
    just to not be confused, i just asked a quesiotn about null pointer exception in fuseki, but that is not the same thing for this question, this question is about an error in fuseki when the data is too big
    – Ania David
    Commented May 18, 2016 at 11:30
  • 2
    If it's not to do with the data itself but just its size, there's no need to upload the data. I'd just explain the situation and quote the error.
    – Pekka
    Commented May 18, 2016 at 11:33
  • 1
    Null pointer on large data sets? Are you sure there's not a row with missing data?
    – Cerbrus
    Commented May 18, 2016 at 11:34
  • @Cerbrus yes i am sure because sometimes the query works good, but sometimes the error happens, it is like 10% works and 90% no
    – Ania David
    Commented May 18, 2016 at 12:12
  • Generating gigabytes of test data is very simple, a for-loop never gets tired. Very high odds that you then discover that your bug disappears. In case it needs to be said: nobody wants to test your data. Commented May 18, 2016 at 12:14
  • So, what makes you think it's not the contents of the data, but the amount of data?
    – Cerbrus
    Commented May 18, 2016 at 12:14
  • @Cerbrus in sparql, there is no wrong in content, you are querying a graph, if the query matches the graph, you get result, if not, you get nothing. my null pointer exception is in engine , not in my code. my code is just a query, and i am querying using fuseki, and the null pointer exception is on fuseki, which is the tool that takes my query and search the graph and give result
    – Ania David
    Commented May 18, 2016 at 12:22
  • 3
    [sigh] what happens if you run your query on 1.25GB of data, ie. the first half? No error ever, reduced eror rate, or same error rate? What happens if you run your query on the second half of the data? Why are you not trying to get more debug info? We cannot magically deduce fixes for intermittent system bugs from some blog text! Commented May 18, 2016 at 12:31
  • 1
    Why is this even on meta? Commented May 18, 2016 at 12:34
  • 1
    I'm voting to close this question as off-topic because this appears tob e a "How to debug" question in disguise.
    – Cerbrus
    Commented May 18, 2016 at 13:01
  • The MCVE is the guide line, but it’s not mandatory for asking a question. If you post a working code that fails only on large data sets, you are already better than a lot of askers. Experienced developers might be able to recognize certain kinds of scalability problems in your code even without actually reproducing the error then. Of course, an example which reproduces the problem out of the box would be even better, but you can’t have that in every case. Otherwise, we had to ban almost all of the multi-threading related questions…
    – Holger
    Commented May 18, 2016 at 14:14

2 Answers 2

13

When I run into a situation where the data size is the problem, there is usually a way to write a piece of code that generates a representative data set. Such code could be included in a question posted on SO to provide readers with the information they need about the data set. If truly the problem is triggered by size, then what the data set contains exactly should not matter. This exercise actually constitutes a good way to double check that your hypothesis about size being the trigger. If you discover that you cannot reproduce the problem by creating a new data set of the same size, then there is more to it than size. It may even be the case that size is not part of the problem at all.

Some problems are just not a good fit for being asked on SO. If it turns out that your case is one where you truly need gigabytes of data that cannot be simply generated, then your problem is not a good fit for SO.

6
  • i can't repreduce the data by a scrpt, the data is rdf graph, where i have too many classes and object properties and inference as well, but i can upload it
    – Ania David
    Commented May 18, 2016 at 12:13
  • 6
    @AniaDavid: Read the answer again. Specifically: "If it turns out that your case is one where you truly need gigabytes of data that cannot be simply generated, then your problem is not a good fit for SO."
    – Cerbrus
    Commented May 18, 2016 at 12:15
  • 1
    @Cerbrus i don't like that :) here is the only place i am getting help :)
    – Ania David
    Commented May 18, 2016 at 12:23
  • 5
    If you have data that reliably fails then can you chop it into two pieces and run again on each piece? Repeat on the failing half until you get to something small enough for you to analyse or to add to a question.
    – AdrianHHH
    Commented May 18, 2016 at 12:27
  • @AdrianHHH yeah, I just suggested that. Debugging 101:( Commented May 18, 2016 at 12:33
  • @AniaDavid read AndrianHHH comment above.
    – Braiam
    Commented May 18, 2016 at 12:45
1

How do you know for sure that your query can't be optimized or improved to handle that kind of error?

I'd start with posting the query, explain that it's not working on large data sets and go from there. Never assume that you already know what the problem is.

1
  • well my query is too big, i know that, and it could be the problem, but look if the query is the problem, the engine should not throw an exception. the engine (which is fuseki), should at least give me a time out error. imagine sql, if you don't know sparql, if your query is really really bad, sql server will not crash.
    – Ania David
    Commented May 18, 2016 at 12:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .