I am very much against these answers in the majority of cases.
My most frequent tags revolve around JavaScript and I can confidently say that I have not seen a single answer that collects measurements that is worth the bits it takes up in a database. Moreover, I expect almost all* such answers to be completely useless.
*because I do not want to simply say "all" and have somebody bring up a single example to prove the opposite. Perhaps the only example on the site.
Where the code runs makes a difference
This may sound harsh but for JavaScript things change all the time. What ran in 99ms today may run in 15ms tomorrow. Or 150ms the day after. All due to different updates to the engine prioritising or de-prioritising different operations. So, any measurement taken over a year ago is most likely not relevant at all today.
However, there is even more to it - I mentioned updates to an engine. But there are multiple JavaScript engines out there. What takes 99ms in Firefox, might take 15ms in Chrome or vice versa. In the same point in time. And an update to either or both can change these measurements drastically.
Benchmarks can be done wrong
Moving on, there are even more reasons why the measurements might be off. One really big one is that people who take them do not know what they are doing. I have seen many benchmarks that do not even measure the same thing in two solutions. One of the measurements might take into account irrelevant operations like traversing arrays instead of doing operations. Or one might take into account putting together a dataset. Or when comparing two solutions that mutate data, then the second one tested might be working with the already mutated dataset.
Benchmarks may look correct but are not measuring real performance
Next problem. Here is some required reading: microbenchmarks fairy tale and Which is faster?. The moral of reading these is that performance is hard to measure. A couple of lines of code and some sample dataset may have vastly different performance characteristics when ran individually versus when ran as a part of a real-life application against real-life data. The numbers you get measuring the solutions in the Q&A might be completely off.
One very common problem around this area in recent years are JIT compiler. Measuring the performance of two solutions without proper warm-up of the engine can give misleading results for the first solution tested. This is because JIT optimisations will not start running immediately but after some usage of the code.
Again, JIT optimisations can very well differ in a real-life application using real-life data, so even if the benchmark included a warm-up, the results might still be misleading.
Benchmarks might omit relevant differences between solutions, focusing on just speed of performance
This is very important to keep in mind. Different solutions often have more to them than just different numbers. For example in JavaScript to check if someObject
has a key-value for foo
one can test
someObject.foo !== undefined
which returns false
if the property exists but the value of it is undefined
or if the property does not exist at all
someObject.hasOwnProperty("foo")
which checks if the object has non-inherited a property called foo
. In particular, it would return false
if the property exists (even if it has a non-undefined
value) but is inherited.
"foo" in someObject
which checks if foo
exists at all in the object, even checking the inheritance.
There is an overlap between these three when checking an object without any inherited properties has or does not have a property. And we either do not care if the value is undefined
or it is not expect it to be that. Comparing the speed of the three throws all this aside to produce a set of numbers.
Similarly, often solutions do have relevant differences and it is most important to pick the one that fits your requirements. Rather than the fastest one.
The difference often does not matter
To come to this point we have to go through a lot of "ifs" - if the benchmarks are current, representative, accurate, if they even compare the correct things. Even if all of these pass, the numbers can still end up useless. Let's take three options for checking object properties above. Somebody did test them and the results were
=== undefined
: 2 787 485 operations per second.
hasOwnProperty
: 2 673 798 operations per second.
in
: 2 344 174 operations per second.
Which one is the winner in this case? Clearly...none of them. Yes, clearly one has the highest number yet the worst performance is still 426.589494 nanoseconds or 0.000426589494 milliseconds per operation. IF you used the worst one and decided to check a million records, the code would still run in well under half a second. That is still a huge "if" - hence the bold and the capitals. If you are not performing a million operations, then the difference would not matter.
Very often I see performance measurements for operations where the number should mean that the whole benchmark was done in vein. Yet it was posted and upvoted very likely misleading others that it provides useful information.
Why are benchmarks a bad fit for Stack Overflow
So far, what I said was broadly applicable to benchmarks. Somewhat geared toward JavaScript as that is where most my experience is, however, the points above are equally valid on-site and off-site. This point is specific to Stack Overflow.
Let's assume the benchmark do pass all the ifs above - they are representative, accurate, useful, etc. The problem is that they have to be maintained. And they very often are not. An answer from 2015 with an update from 2016 might still be missing the comparison with answers that were posted since. Or any updates made to previous answers.
It might not even be possible for somebody else to update the answer without entirely rewriting it, as any editor likely cannot run the benchmarks with the same parameters as the original author of the answer. They would be running the code on different machines, after all. And some answers even omit relevant setup, so any editor would have to either guess or entirely re-create the benchmark suite.
Any and all benchmark answers become unreliable the day they are posted. What if the author never comes back? What if the author is diligent in updating it but how long would they be doing that?
Simply put, all benchmarking answers have to have ongoing maintenance to stay relevant. Which I would argue does not fit in the Q&A model. Yes, other answers can also become out of date. Yet, any benchmark answer is by its very nature reliant on all other answers and has to be constantly updated to match. Add to that any of the possible changes in environments where the code would run or differences in how the code might behave under different circumstances and I would argue these answers are ill equipped to provide on-going useful knowledge.
What can we do about these answers?
Absolutely nothing right now. Even if the answer is grossly misleading and completely wrong, if it has high score count, we cannot do anything about it. Nor are users apparently interested in the comparison being correct. The votes are coming in from the comparison being there, despite its correctness.
What should happen instead is benchmarking answers should be removed. And if the speed of a solution is a factor, then edit all other existing answers adding a benchmark for the solution they provide.
This also relies on establishing very clear and objective guidelines on how things can and should be benchmarked. Perhaps that can be set up in the question, thus any answerer can do it themselves, or anybody can edit the results in.
I can already foresee one of the very first objections would be missing out on reputation. To which, I could point out the goal of Stack Overflow is to be a repository of knowledge. Not a reputation mining farm.
Yes, the current system is not well equipped to handle answers with contributions. That is a much deeper flaw that runs throughout the Q&A format set up for the site. "Meta answers" that comment on other answers are a symptom of it. Yet, I believe this is the right thing to do. As opposed to produce answers that are either irrelevant at the time of post, or are left to lose relevancy as time progresses and especially as new answers are posted.