38

Take this question for example. It's a clear question on the performances of random number generators, with a clear answer that helped me. And yet the most upvoted comment (more than both the question and the answer) is this:

Build a benchmark test and find out. If you care about performance, you measure it. If you don't measure, this means you don't really care.

I mean sure, they could have, but I thought one of the big purposes of SO was to save time? For him/her, testing would have been faster than asking a question, but if you consider the time of all the people asking the same question, isn't it a good thing to have answers like these on SO?

Many of the most upvoted questions on this site (like this one) are ones whose answers could be found in the official documents, should we tell them to just read the documents as well (and that if you don't, you don't really care)? Not to mention that the answer contained additional information I could not get by simply running a benchmark test (like how rand is discouraged). So why are questions about performance discouraged?

14
  • 3
    The accepted answer seems to boil down to "it depends on what you need" and the other to "this method of getting poor randomness is faster than a proper one". Do these really answer the question? They seem to be mostly useful for answering another interesting question that wasn't actually asked there. Commented Jul 15, 2022 at 7:25
  • 36
    It is not so much that questions about performance are discouraged, it is more that people royally fail at asking them in a way where someone might have a chance of answering something that is factual and useful. Luckily the person answering the question you linked to could kind of smell that it was an X/Y problem. The question they answered is "should you use rand() in situations where high performance matters". Very clear and measurable arguments have been given why you should not.
    – Gimby
    Commented Jul 15, 2022 at 8:50
  • 1
    See this: meta.stackoverflow.com/a/412904/584518
    – Lundin
    Commented Jul 15, 2022 at 9:29
  • 19
    ' thought one of the big purposes of SO was to save time?', OK, whose time would be saved, and whose time wasted? Commented Jul 15, 2022 at 12:10
  • 4
    Questions like that tend not to be useful. For more details: Which is faster? Commented Jul 15, 2022 at 14:21
  • 2
    What makes you think the linked question was poorly received? It had a score of 4 (which is perfectly respectable) and 2 upvoted answers. Would that all of my questions got such a reception...
    – TylerH
    Commented Jul 15, 2022 at 19:33
  • I see this comment more like: do more research first. After all, measuring the time of something is only the first step in performance optimization. A question like: "this piece of code solves problem X and I measured the time but it's too slow, how can I improve the speed" should be ontopic. Commented Jul 15, 2022 at 20:53
  • 1
    Is it faster to compute 5+1 or 4+1+1? OK, you may think that, but I have some hardware with some very particular optimisations, optimisations which have been acquired over a very long experience of optimising. Do you feel lucky today? Commented Jul 16, 2022 at 0:10
  • 4
    I have 2 horses. Which is faster?
    – QHarr
    Commented Jul 16, 2022 at 0:38
  • 3
    Some performance questions get a very positive response, e.g. Why does C++ code for testing the Collatz conjecture run faster than hand-written assembly? / Why does this delay-loop start to run faster after several iterations with no sleep?, and SO's highest-voted Q&A, Why is processing a sorted array faster than processing an unsorted array? . It all depends how you ask, and what kind of performance question it is. Commented Jul 16, 2022 at 1:56
  • 7
    (Also many optimization questions get good answers, especially in tags like simd avx. But performance questions about explaining interesting performance effects get upvoted if they come with a good benchmark.) But "which is faster" without lots of context is usually not a good question. Commented Jul 16, 2022 at 1:58
  • 1
    Pretty sure it just depends who is on at the time, some people seem to want to genuinely help, and some people just get a kicks at talking down to others. So oi la la, doesn't follow general rules, time for those types of people to get off. But that's just my 2 cents, based on the types of comments I get while trying to give an answer to help out OP (from other members!) OP is usually happy and wants to discuss further
    – Natio2
    Commented Jul 16, 2022 at 5:37
  • 5
    In my experience you just need to be in the right neighborhood. If you're in a situation where you already know you've beaten the compiler and have succumbed to typing asm instructions to put your AVX registers to better use in a critical number-crunching code path, you will see good and meaningful answers from people like @PeterCordes in the [asm] tag et al. But if you're talking about micro-optimizing some javascript variable assignments that run exactly once, then what are you even doing, and also frankly, at that point it's not like anyone even knows what's going on in the CPU anyway.
    – dialer
    Commented Jul 16, 2022 at 6:05
  • 4
    @PeterCordes it's interesting that all those questions want to know "why" between two things one is faster than the other. It's like good questions already know which is more efficient but want to know the underlying reasons for that. Not just "my program slow, make it go brrr"
    – Braiam
    Commented Jul 16, 2022 at 11:17

3 Answers 3

56

While that comment could be phrased more constructively, I think it's better interpreted as a hint as to the correct approach rather than discouraging the question. While there's a good answer with some general information, the only reliable way to determine if something has suitable performance is to use it in your program and see.

Raymond Chen at Microsoft touches on this in one of my all-time favorite blog posts: The wrong way of benchmarking the most efficient integer comparison function. It refers to a Stack Overflow question in which people have written a lot of very clever methods to squeeze as much performance out of an integer comparison function as possible, and someone has written a thorough benchmark comparing them all and determining which one's the fastest. The problem: it's all completely wrong, and the naive approach is faster than all of them, because it's the one the compiler knows how to optimize down to a single instruction in any likely calling code.

Discussing the performance of generating a random number in isolation has similar problems. Only in context is it possible to tell if something is adequate or not, and even to know how long it will take (CPU pipelining is another related factor).

6
  • Good point, didn't think of it from that perspective.
    – ssamtkwon
    Commented Jul 15, 2022 at 7:31
  • 5
    The same deal happened in Java land. The JVM got more optimised over time to the point where you have to be a true master if you want to beat it. But people kept treating their code as if they were still programming for Java 1.2. This sparked articles named write dumb code.
    – Gimby
    Commented Jul 15, 2022 at 7:54
  • 6
  • 7
    I find that good performance questions are usually of the form: "I have this specific piece of code in this program [insert mcve], why does doing X cause worse performance than doing Y?" I personally love those kind of questions, specially when it comes down to complex low-level CPU-specific architectural optimizations. Commented Jul 15, 2022 at 16:33
  • I upvoted but I still think you went too easy on the phrasing of the comment. It is not "better interpreted as a hint" and if it's trying not to discourage the question (or others like it) then I think it has failed in that respect. A good point is pretty easily lost when delivered that way.
    – briantist
    Commented Jul 16, 2022 at 21:15
  • The bothersome thing about that blog post is in most constructions, you take the address of the comparison function and pass is to something else. The compiler doesn't get to optimize it out.
    – Joshua
    Commented Jul 18, 2022 at 2:31
27

Donald Knuth answered this question nearly half a century ago:

Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.

The performance of a random number generator implementation is almost certainly going to be irrelevant to the overall performance of your application*. In other words, asking about performance of an RNG is an effectively useless question; you might as well ask how many angels can dance on the head of a pin. These are microbenchmarks, and they are the root of all evil to which Knuth refers: the canonical example in programming of missing the forest for the trees.

But in the context of Stack Overflow, performance-type questions are heavily discouraged for the simple reason that, in the words of one of my favourite Age of Empires 2 game casters, It Depends:

  • A particular implementation will be fastest when run on one particular hardware configuration; but when you change hardware, a different implementation is faster. (This is also why answers that benchmark different implementations are fundamentally useless wastes of bytes. I don't care how fast it is on your hardware, it's gonna be running on mine.)
  • A particular implementation will have been fastest in the past, and highly-upvoted answers will suggest to use it; but since those answers were posted a newer, faster implementation has been developed.
  • A particular implementation was fastest in the past, but hardware optimisations have made another implementation(s) faster.
  • A particular implementation will be fastest but difficult to use, while another will be slower but far easier to use.

All of the above, and far more, combine to ensure that there cannot ever be a single objective answer to "which is fastest?". And as Stack Overflow questions are intended to generate objective answers, such questions are therefore inherently off-topic.

* Unless you're writing an application that needs to generate lots of random numbers, like cryptography. But that is a small minority of users, and those smart enough to be writing crypto properly are also smart enough to benchmark it properly on their hardware.

6
  • 6
    And a fifth: Your boss likely ain't paying for fastest. They're paying for fast enough, and fast enough is a LOT easier to reach. Commented Jul 15, 2022 at 18:59
  • 1
    "It Depends" And cache issues also factor in. If an RNG algorithm has a lot of state, then calling it frequently in a hot loop might be OK, but calling it once every now and then could cause a lot of cache misses. How much that matters changes with hardware. Commented Jul 15, 2022 at 19:58
  • 2
    Depends on the question. Sure, sometimes there's no clear answer due to the reasons you mentioned. But sometimes there is, especially if the question is dealing with algorithms and data structures, and is looking for speed-ups unrelated to microoptimizations. Commented Jul 16, 2022 at 4:19
  • 1
    @CertainPerformance The problem is that 99% of questions about performance are not of the "good" type that you mention.
    – Ian Kemp
    Commented Jul 16, 2022 at 20:06
  • 5
    I love the full quote of Knuth's famous statement, there's nuance to it that's lost in the shortened version that gets bandied about. The problem with SO is that it's often impossible to know if someone is asking about the 97% or the 3%. As for the speed of RNG being a non-issue, I know I have an answer on SO somewhere that was criticized for being slow because its run time was dominated by random number calculation. Commented Jul 17, 2022 at 5:06
  • @MarkRansom Reminds me of a problem I had, caused by someone using the wrong method of java.security.SecureRandom (generateSeed instead of nextBytes), which caused applications to block or slow down significantly in VMs due to lack of entropy Commented Jul 17, 2022 at 8:53
3

That comment may be snarky but it has as point: performance only means something if you measure something, and often doing that relatively to something else.

Let me give an example of a good performance question: OpenMP basic temperature modelling with parallel processing taking longer than serial code

Here the observation is that more cores gives lower performance (on an operation that should speed up) and the explanation lies in substandard workings of a particular compiler.

My pov: performance questions are legit if quantifiable.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .