-6

Prompted by this question: How should we handle answers that don't answer the question, but evaluate the performance of other answers?

The issue with the above Meta question is that it used as its basis a Stack Overflow question that explicitly asks for the most efficient implementation, thus somewhat undermining the former's premise.

But after looking at the underlying question, I think the problem is with that question itself - specifically the bit where it requires efficiency - for many reasons, some of which are outlined by @VLAZ in their excellent answer:

  • Where the code runs makes a difference
  • Benchmarks can be done wrong
  • Benchmarks may look correct but are not measuring real performance
  • Benchmarks might omit relevant differences between solutions, focusing on just speed of performance
  • The difference often does not matter (because the input dataset simply isn't large enough for it to matter)

In short, I believe that questions that asks for efficient solutions are almost always misguided (in my experience they are often asked by people who don't understand the least bit about efficiency). And because such questions invite the legitimate problem of benchmark "answers" identified by the linked Meta question, I think it is valid to judge such questions as implicitly too broad.

Therefore, I propose that such questions should be extremely strictly curated. Unless an asker can demonstrate a legitimate need for efficiency (I would argue this would necessitate providing extensive prior research as well as very specific constraints), any mentions of that term should be removed from such questions, which will implicitly discourage and/or invalidate benchmark "answers".

4
  • 5
    This seems very situational. For example, the Python tag often has CPython+numpy questions that regularly deal with large data amounts and provides stable performance characteristics. Would you allow for any technologies as implicitly having a legitimate need for efficiency? Commented Apr 3, 2023 at 8:15
  • '... or, how to prevent benchmark "answers"' - I don't think that presence/absence of the word "effective" in the question post correlates with the probability to get "benchmark answer" for that question. Even if a question post contains the word "effective" word, no one prevents to post an answer which ignores this aspect. And if a question post doesn't contain the word "effective", but has two answers with different approaches, then it is acceptable to post an answer which compares these approaches.
    – Tsyvarev
    Commented Apr 3, 2023 at 9:54
  • I concur with MisterMiyagi, in R it even happens a community wiki ends done to compare answer efficiency like here stackoverflow.com/a/47435067/3627607 and a wide bunch of question are about alternative ways than the OP because their way ends up taking too long.
    – Tensibai
    Commented Apr 3, 2023 at 9:54
  • 2
    This meta-Q seems to run into the exact same issue as the other one: Citing an example that doesn't support the perceived scenario. The CPythpn+numpy scenario is sufficiently stable and well-defined for "efficient" to be actually meaningful. I've just now – 5 years later! - rerun the benchmark on a certainly different setup and the quantitative results were equivalent. In that respect I would even say the benchmark answer is the most useful of the bunch. Commented Apr 3, 2023 at 13:11

1 Answer 1

11

I think this would needlessly damage areas in which efficiency is sufficiently defined and benchmarks are meaningful. Critically, it would prevent experts from giving advice whether efficiency considerations and benchmarks are meaningful.


Practically all reasons listed against such questions are areas of expert knowledge: an expert knows whether benchmarks are meaningful, an expert knows which features are relevant, an expert knows what impacts real performance, and so on and on. As such, many questions need a bit of leeway in what is the metric and what are the features under consideration because that is exactly what answers should supply.

While there are enough areas where a benchmark would not be meaningful, and efficiency would be ill-defined, this is not a general feature of "efficiency questions" and "benchmark answers". Many tags and technologies do imply stable environments and interesting orders of magnitude. Requiring to motivate what is already given in these areas would be counter productive.

As such, I think it is valid to judge each such question on its own merits.

9
  • 1
    My takeaway: "don't do anything, even if no expert would agree the benchmark is meaningful, because the non-expert will swamp it with upvotes anyway". Which feels a little bit lackluster.
    – VLAZ
    Commented Apr 3, 2023 at 15:04
  • 3
    @VLAZ Honestly, I've had it with the constant misrepresentations that people apply to this topic. Did I say "don't do anything" anywhere? No? Then please consider to re-read what I actually said before commenting again. If you feel I did, please spell it out what parts made you think so. Commented Apr 3, 2023 at 15:14
  • So... how should we handle answers with benchmarks which are unhelfpful? "judge each such question on its own merits." then what? After it is judged how do we handle this? Also "Did I say "don't do anything" anywhere? No?" I mean - yes. You did not propose doing any action. Well except "judge" but we do that already anyway, which is why I ignored it.
    – VLAZ
    Commented Apr 3, 2023 at 15:17
  • 3
    @VLAZ Downvote unhelpful questions, close vote unclear questions, edit salvageable questions, ... Like seriously, am I stuttering or something? In what way is "judging" equal to not doing anything, or even explicitly refraining from doing anything as you seem to imply? Commented Apr 3, 2023 at 15:20
  • 2
    Right - so it's "do nothing". Because we are unable to handle unhelpful benchmarks. And those appear in on-topic questions. So, we can't CV the question, we can downvote the benchmark and, so what? It's not changing the triple digit number anyway. Even if it's double digit, it matters little. We certainly cannot delete it. A flag for removal will almost assuredly be refused. Thus your proposal really is "don't do anything". I'd consider myself an expert and can tell you that benchmarks exist that are wildly off. And my downvote mean jack all. Maybe made the score +100 / -5.
    – VLAZ
    Commented Apr 3, 2023 at 15:23
  • 2
    @VLAZ How is that related to benchmarking answers and efficiency questions in anyway? I've wasted a truckload of rep and time on Q&As where it had absolutely no practical impact at all. Hardly any of that was benchmarks. If you feel curation is a blunt sword in general, then complain about that. Commented Apr 3, 2023 at 15:28
  • 2
    "even explicitly refraining from doing anything as you seem to imply?" For clarity, I didn't imply that we as users should do no actions. But that no actual resolution would follow. We judge something as "bad and misleading" and we as users can click some buttons but we cannot actually affect such answers in a meaningful way. Downvotes work until a post has maybe 5 votes total. After that, downvotes are lost in the deluge of upvotes and no longer meaningful.
    – VLAZ
    Commented Apr 3, 2023 at 15:28
  • 1
    @VLAZ No, you have implied that I had implied we as users should do no action. This freely reinterpreting what others said is exactly why I am critical of allowing "experts" to hit the nuke button. Commented Apr 3, 2023 at 15:38
  • 1
    Someone on the internet decided to upvote it so it must stick around forever, regardless of how awful or damaging it may be. This is supposed to be a knowledgebase of useful questions and answers, after all
    – Kevin B
    Commented Apr 3, 2023 at 15:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .