8

Very often we get questions about performance differences between similar ways to express the same thing, such as these two asked within 15 minutes of each other today:

What is more efficient for a PHP variable, $timelimit 10*60; or $limit 600;?

Which is more efficient: string concatenation or substitution?

These are practically always about differences that in practice have negligible performance difference. It's misleading for anyone to think that they matter, and it's a waste of time (IMHO) to answer them.

My usual reaction to these questions is to post a comment with a link to a website that explains Donald Knuth's quote premature optimization is the root of all evil. But I feel like this isn't sufficient -- people will still waste effort benchmarking it just to show how negligible it is (someone did that in the first question above, his result was 16ns per assignment).

I feel like questions about performance should be based on actual code that's experiencing performance problems -- for instance, we often get good questions about complex SQL queries that the poster needs help optimizing. Theoretical or curiosity questions like the above should be closable in some way. If you agree, what close reason would you suggest?

18
  • 5
    Another thing that matters, beside the "premature optimization" one, is that micro-benchmarks are often misleading or wrong. I would tag it microbenchmark and then ignore it, as I concurr that there's seldom anything useful there, but I don't know what else to do with it. Sep 4, 2015 at 0:52
  • Yeah, I tried his benchmark repeatedly, and a few times it reported that the "faster" version was slightly slower. But it was mostly consistent with his result (although on my machine the difference is 20ns).
    – Barmar
    Sep 4, 2015 at 0:56
  • 3
    I have issues with those questions as written; doesn't matter that they talk about 'negligible differences'. Sep 4, 2015 at 0:56
  • 10
    "Which is faster, making a coffee or taking the bus?" "Well, son, it depends on a number of factors..."
    – BoltClock
    Sep 4, 2015 at 2:47
  • 6
    ^ Besides comparisons where the difference is negligible, there's also the comparisons that make absolutely no sense whatsoever.
    – BoltClock
    Sep 4, 2015 at 2:49
  • Haven't noticed too many of those, but I guess they're similar.
    – Barmar
    Sep 4, 2015 at 2:51
  • @BoltClock: Or open-ended comparisons: meta.stackoverflow.com/questions/291558/… Sep 4, 2015 at 3:03
  • 1
    It is rather regretful that these questions are so taboo, program perf is a very basic measure of meeting the project goals. Pretty big project disaster when it just can't get the job done by a factor of 2 or more, always discovered at the worst possible moment. Unfortunately the wrong people ask them. Or it just doesn't matter anymore because machines are fast enough anyway. Hmm. Yet we hear from programmers constantly that know how to multiply 16 nanoseconds to slow every day. Sep 4, 2015 at 22:08
  • I'm not talking about factors of 2, I'm talking about factors of 0.001%.
    – Barmar
    Sep 4, 2015 at 22:11
  • 5
    Well, you don't like the [performance] tag. Add it to your ignored tags, when you run into a question that's missing it then just add it and you'll never have to see it again. Sep 4, 2015 at 22:40
  • I don't think most of the questions have that tag. Someone added [microbenchmark] to the two questions I linked to.
    – Barmar
    Sep 4, 2015 at 22:49
  • 1
    It's also worth noting that what is considered negligible in one context might not be in another. Premature optimization is bad. Optimization itself isn't. In some cases, performance differences of a few orders of magnitude might not matter at all (e.g. who cares if something takes 2 ns or 2 ms in handling a button click?) In other situations, a difference of a percent or less can be quite substantial (e.g. frequently-run HPC programs.) What's negligible and what isn't is a matter of context.
    – reirab
    Sep 6, 2015 at 21:52
  • 2
    @reirab That's precisely why a good question about performance should be in the context of a specific program. If you have a program whose performance is not acceptable, it's perfectly reasonable to ask for help in optimizing it. My issue is with these generic "which is better, X or Y?" questions that are not related to an actual bottleneck.
    – Barmar
    Sep 6, 2015 at 22:27
  • Folks ask about completely theoretical perf worries without profiling or benching, e.g., will X minor syntax change affect my speed. I'm not sure those are uniformly pointless q's (like, they're a chance to educate asker about compiler optimizations and "build first then profile") but they aren't "good" in the sense that the asker did what they could to figure it out themselves before coming to SO, and (usually) the answers aren't likely to help others.
    – twotwotwo
    Sep 6, 2015 at 22:48
  • I've gotten in the habit of commenting something like "life's too short to spend too much time sweating hypothetical performance issues; build what you want then bench to look for problems" on questions like this, if applicable followed by any intuition that leads me to think it won't be their problem. Mixed record: some askers satisfied to some extent by that, many not. :)
    – twotwotwo
    Sep 6, 2015 at 22:53

5 Answers 5

18

Part of the issue I have with questions that say "X is faster than Y, should I use X or Y?" is that these questions generally do not contain any benchmarks. They make a supposition without facts to back it up.

Any time a question says, "x is faster than y", or "Is X faster than y?" should be providing the benchmarks to prove that assertion.

Here's what a benchmark-centric question should contain:

  • The actual code used to run the benchmark
  • The time differences over the number of trials
  • The expected results vs the actual results
  • A question that doesn't 'assume facts not in evidence'.

The problem with both of those questions is that they contain none of these things. They read like this:

I think X is faster than Y, but I don't have the time or inclination to actually benchmark this. Since X is faster than Y, should I use X or Y?

At best it represents a question whose answer will be someone who actually constructed a benchmark; and in general it just becomes a cesspool of non-information, surrounded by navel-gazing.

9
  • 7
    at my first glance your answer seems have a paradox: if users pay the effort to prove "x is faster then y" then why they still need to ask "is x faster than y"?
    – ggrr
    Sep 4, 2015 at 1:59
  • 6
    Which is faster, constructing a benchmark or inspecting the navel?
    – BoltClock
    Sep 4, 2015 at 2:55
  • 3
    @BoltClock I vote for the benchmark, you do not know how deep you can dig and how stuck that lint can get.
    – Gimby
    Sep 4, 2015 at 9:21
  • 1
    @amuse That's exactly my point. Sep 4, 2015 at 13:46
  • 6
    meh... xkcd.com/1445
    – Braiam
    Sep 4, 2015 at 19:53
  • 1
    "...what a benchmark-centric question should contain" ... Umm you might want to add: The system that the benchmark was run on (and maybe the input data that was used, if it is not artificial and/or provided with the code)
    – Marco13
    Sep 5, 2015 at 0:21
  • Great point. Feel free to edit it in if you get to it before I'm in front of a PC. Sep 5, 2015 at 2:02
  • I do agree, but I feel that this answer misses the question: Most of the problematic posts don't establish which one is (presumed to be) faster, they just ask "Should I use X or Y - I want the faster one?". They basically ask for a benchmark, and the solution would be to close as "looking for resources" or answer with "it doesn't matter, speed is not what you should be looking for".
    – Bergi
    Sep 6, 2015 at 21:45
  • still not get the main idea, do you feel that kind of questions are problematic? or the questions are not originally problematic, but just to despise users that not paying effort before asking?
    – ggrr
    Sep 7, 2015 at 2:58
16

Playing devils advocate here: There is a difference between a question about performance and a badly asked question that's actually about the best way to do something. To that end, the opinion that premature optimization is the root of all evil, and that optimization question are, by extension, also evil, is just that, an opinion. And as is so rightly pointed out many times a day on Stack Overflow, opinion-based entries be they question, answers or editorial actions, are not appropriate.

Yes, most of the questions of this type are uneedful, but unless you can link to a duplicate then a summary closing of a question is not helpful to the community as the noob and those noobs that follow are then left in the dark. We are all here to learn and become better programmers. If we go around indicating that people are idiots (even just by implication by aggressive question closing) for asking questions then that's not very constructive is it?!

Lots of new programmers don't know or don't understand the idea of premature optimization or why it's bad. What's more, as several people have mentioned already, such questions are actually trying to understand the best way to achieve something, but currently have no other way of being able to explain better. It is important to note that they have no other way, because they don't know enough... That's why they've come to Stack Overflow in the first place: to learn more on a particular point. If they can be pointed to documentation or the Stack Overflow guidelines or have the difference between performance and what they actually want pointed out to them in a succinct manner, then that would be better than just closing questions with prejudice.

When a person doesn't know enough to properly know what to ask it can be very frustrating to come to such a storehouse of knowledge as Stack Overflow and be turned away precisely because they don't know enough!

4
  • Yes, it is very useful to have these kinds of question, especially if the user is then encouraged to benchmark it himself and answer his own question.
    – Toothbrush
    Sep 4, 2015 at 17:31
  • Dupe is not the only close reason. Sep 4, 2015 at 17:50
  • 3
    One of the problems I see is the naive assumption that the "best" way to do something is the way with the best performance. While there are some contexts where you have to worry about every byte or microsecond, for most applications that is not the most important consideration. If they ask for the best performance when they really want to know the best way, it's kind of an XY problem. And for most of the things they ask about, there isn't really a best way -- they're just different styles that all have their proponents.
    – Barmar
    Sep 4, 2015 at 19:45
  • A good example here that I came across myself recently is: Is it better to use Count() or Length() on VB.NET arrays? Being a VB.NET noob I turned to the internet. MSDN, as ever was about as helpful as standing on my head. But how to ask which is 'best' on Stack Overflow when 'best' could be opinion based? Luckily someone had already asked the same question. Furthermore the answer was actually performance related! (Count() actually counts each item, Length() simply returns a stored variable).
    – Toby
    Sep 10, 2015 at 17:49
11

First of all:

java - Why is processing a sorted array faster than an unsorted array?

This is a performance/optimization question with a quite impressive score. More importantly, its answer is mind blowing: branch prediction explained in lay man terms!

Secondly, the performance of either solutions differ by an order of magnitude, which is not to be scoffed at.

Thus my belief that performance related question have their place on Stack Overflow.


I would also like to take the opportunity to remind everyone that Knuth's quote is:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.


Are the questions you linked bad?

Yes, they certainly do seem so.

It does not mean, however, that by extension any question about performance is necessarily bad. Let's judge each question on its own merits.

2
  • True, although there is a profound difference: the PHP question about 600 vs 60*10 is asking "which is more efficient". The java question you like is asking "why is it more efficient" which is a theorethical and way more interesting question.
    – nico
    Sep 6, 2015 at 13:09
  • 1
    @nico: true, however, given that benchmarks are worthless without analysis, a good answer will anyway delve into the reason and area of applicability of the result. Sep 6, 2015 at 13:12
8

I wholeheartedly agree with this idea, but I want to call some nuances out here.

First, we already do a pretty reasonable job of handling the silly benchmarking questions; at least in tags that I frequent. Normally, if I do see something that seems a bit awkward to benchmark, I usually write a comment to the effect of:

It's tough to say which is faster, since we don't know your workload. Have you run this through a profiler of some kind that truly identifies this as a hot spot?

Without information from the profiler to back the question up, and without a reduced enough sample so that others may encounter and attempt their own solutions at it, it's just too broad. Further, there would be no value in trying to dissect something unless it was a bonafide pain point.

Second, depending on the question itself, there may be a more suitable site for it. I've remarked that I don't mind seeing those sorts of questions go to Code Review, but they've got to be damn good; that is, they have to be complete and fully on-topic. They can also fit on Stack Overflow too, if they're also just as complete. The database optimization questions I'd expect would have more of a home on DBA.SE than anything else.

4
  • 1
    Questions of this type are almost never about any actual code. The poster seems to be just curious, or he's starting to write code and wants to know the best way to write it in general. So there's no workload, and no code to send to CR.SE.
    – Barmar
    Sep 4, 2015 at 1:20
  • In that case, by all means, drop the anvil. There's no value in those questions at all.
    – Makoto
    Sep 4, 2015 at 1:21
  • 4
    I'd reason that the anvil can be dropped because these are people who are asking the wrong questions. They want to know what is "better" and can only come up with a performance comparison to give a body to what "better" means, completely ignoring such trivialities as code readability and maintainability, functional correctness and re-use potential. And to be honest, when I was a complete novice I spent way too much time worrying about performance too. It was one of the few things my mind could grasp, lacking working experience.
    – Gimby
    Sep 4, 2015 at 11:55
  • Agreed. From my perspective, allowing these questions to stay open perpetuates the myth that wringing every bit of performance out of the code is an important skill.
    – Barmar
    Sep 4, 2015 at 19:50
3

As someone who is at the sort of programming knowledge level to be asking such a question...

I occasionally poke around in assorted languages simply because I enjoy doing so, because I want to achieve some result etc.

So I'll google a problem, and realise that there are a couple of different approaches to solving it. At which point I'll google a bit more and hopefully get some idea of which to use.

If I can't find that directly I'll likely use the one that is documented the most (i.e. gets the most - useful - google results)

If at that point I posted on SO to figure it out, I'd be kind of annoyed if that question got closed.

Maybe at the point of writing the question people are thinking about the best way to ask, performance is an obvious thing to ask, as it is less subjective than 'best'. 'Best' will lead to an opinion, and is likely to get the question closed, performance is concrete and so it seems the better question.

Maybe the best response is to answer the question slightly differently and instead of answering the speed question, you give some tips about microbenchmarking and an overview of what x and y are.

That way they go away thinking they got some help, rather than 'what a bunch of...'

8
  • SO is supposed to be for "practical" programming questions. Usually the answer to this microoptimization questions is "it doesn't matter", so they're not really practical. But I get your point: at the time you're asking, you don't know this.
    – Barmar
    Sep 5, 2015 at 0:31
  • @Barmar my concern is that maybe the question they've asked, isn't actually the question they want an answer for, that they've twisted the question from what's the difference between x and y, to which is quicker, because that feel more like an SO question. Maybe clarifying in comments what they actually want to know, guiding them towards the question they want answering, rather than sending them away because its a stupid question! (to paraphrase a little ;) )
    – Michael B
    Sep 5, 2015 at 0:42
  • 1
    Maybe that's occasionally possible, but it seems to me that in most cases they really think these performance differences matter. Most SO questioners seem to be self-taught, and they don't know enough about how things work to be able to estimate what's likely to matter and what doesn't. E.g. they don't understand parsing vs. execution -- I've seen people who thought that longer variable names could slow things down.
    – Barmar
    Sep 5, 2015 at 23:41
  • @Barmar I think in the first instance we should give the benefit of the doubt, give them a guiding hand towards light. if they start complaining at that point that they want to save the 2ns on their app. then we can wander off and mutter about them. (and it did take a moment's thinking to figure the parsing vs execution thing :P - but at my level of (mis)knowledge, SO questions are my primary source of getting on the right track)
    – Michael B
    Sep 5, 2015 at 23:47
  • @Barmar: Wait, why do you think that SO is for "practical" programming questions? There can be good questions that are anything but practical, deep dives into the lowering to assembly or implementation tricks or standards interpretation/ambiguities are rarely immediately practical, but they are about programming or programming languages, can be answered objectively and are of interest to some. Sep 6, 2015 at 12:53
  • @MatthieuM. It's in stackoverflow.com/help/on-topic: "a practical, answerable problem that is unique to software development"
    – Barmar
    Sep 6, 2015 at 22:29
  • @Barmar: Interesting, I would say that all the questions on language specifications are not practical (but theoretical); but then I guess that since it's a rule of thumb ("generally covers"), it's all good to stray from it occasionally. Sep 7, 2015 at 6:14
  • 1
    @MatthieuM. That's true of some questions, like "why is <expression that no sane programmer would ever write> undefined behavior?"
    – Barmar
    Sep 8, 2015 at 3:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .