25

In regards to this question Mysterious StringReader performance, should it really be closed as "This question is not reproducible or was caused by typos"?

Disclaimer : I answered this question, and am not debating whether the question should be reopened.

I can see the argument that it was the mistake in the OP's benchmarking methodology that was the issue, and if the benchmark was just run in release mode it would have garnered more intuitive results.

However, I wonder if the question was worded differently, maybe with a more descriptive and concise title, would it be a little more useful to Stack Overflow? Or perhaps it could be made into a better signpost?

My reasoning is, benchmarking is not the easiest thing to achieve, and using the right tools in the right way is paramount to meaningful Empirical Evidence. Closing as not reproducible or a typo seems a little well... simplistic (though it might be valid, I'm not sure ¯\_(ツ)_/¯).

So, is this a legitimate question, is it something that might benefit from being edited, should it be a duplicate, or is it just noise?

7
  • 5
    I don't use C#, but for Java I think the constructive thing to do would be to close as a dupe of How do I write a correct micro-benchmark in Java?, and add a comment explaining the specific mistake they had made. Here I'm assume that the mistake is one that is addressed in the canonical dupe target. I wasn't able to find a similar dupe target for C#, but that may just be because I'm not familiar with that tag at all.
    – kaya3
    Dec 6, 2021 at 3:55
  • 19
    IMM the most important part of this close reason is "[it] was resolved in a way less likely to help future readers". So do you think someone in the future facing the same issue will be able to find this Q/A through search engines? I am not a SME, but from the look of it, I guess all it can do is attract people that will actually face problems with StringReader and these may get frustrated to find an answer unrelated to their issue. So you could edit the question to be more about the real issue, but then again, future readers would have to know that their issue was with the benchmarking tool.
    – Kaiido
    Dec 6, 2021 at 4:59
  • 6
    @Kaiido agreed, either way "Mysterious StringReader performance" is not the most appropriate or SO friendly title. The actual question should be 'Spurious results benchmarking with stopwatch in debug mode against (seemingly) the same code`
    – TheGeneral
    Dec 6, 2021 at 5:05
  • 1
    Indeed, it is probably best to avoid any kind of terminology that means in some form or another strange, weird, mysterious, etc. It just means you don't understand what you're seeing yet and thus haven't done the right tests yet. Posting a question in that "It's weird!" phase of your mental being is too soon.
    – Gimby
    Dec 6, 2021 at 10:20
  • 5
    Suggestion: Instead of voting to close for some reason, create a canonical "How to Benchmark" Q&A, and then use that are a dupe target to close the Q's against. Saves use from regurgitating the same info, and lets people with gold badges close with a single vote saving others their close votes. Dec 6, 2021 at 13:15
  • 1
    @NathanOliver: There's already Idiomatic way of performance evaluation? which is not language-specific, recommends using a benchmark framework for whatever language, and points out some pitfalls of micro-benchmarking on modern CPUs. Dec 6, 2021 at 18:36
  • 1
    "...if the question was worded differently, maybe with a more descriptive and concise title, would it be a little more useful to Stack Overflow?" Only if there were a real effect, i.e. if under relevant conditions (release mode, overhead of benchmark subtracted), there would still be a significant effect on speed. Then the causes could be analyzed. However, in the current form it's unclear if there is even such an effect. Dec 6, 2021 at 18:39

4 Answers 4

25

It's just noise.

If I had a nickel for every C# benchmarking question where the asker failed to run their app in Release mode, I'd be a rich man.

If I had a nickel for every C# benchmarking question where the asker failed to use BenchmarkDotNet, the de facto benchmarking library for .NET, I'd be doubly wealthy.

If I had a nickel for every C# benchmarking question where the asker failed to spend five seconds Googling "how to benchmark in .net", which turns up BenchmarkDotNet as the first result, I'd be thrice as rich.

Technically, closing that question as a typo is incorrect, but let's be honest - the question shows zero research effort and very minimal understanding, and we no longer have those close reasons because they're "not nice", and the likelihood of that question ever being useful to anyone else is somewhere between zero and negative infinity, so... any close reason in a storm, I guess. (Note that I did not cast any close votes on said question; I'm merely guessing what the close voters there did.)

What would be more correct is to close that question (and all future similar ones) as a duplicate of a canonical "how do I benchmark my .NET software correctly" question. The latter doesn't exist yet, but you're welcome to ask and self-answer it with a link to BenchmarkDotNet.

19
  • 12
    But it's not getting closed "as a typo". That's one of the reasons for picking that option; the more general version of that close reason is "[it] was resolved in a way less likely to help future readers". e.g. wrong benchmark methodology, not a real effect of the sort described by the question title. Still, as suggested by you and in comments under this meta question, closing as a dup of a benchmarking-methodology Q&A would be appropriate. (Such as Idiomatic way of performance evaluation? or a language-specific one.) Dec 6, 2021 at 15:50
  • 1
    "Technically, closing that question as a typo is incorrect..." Or even technically correct, because in release mode the results are not reproducible, or are they? Dec 6, 2021 at 18:37
  • 29
    This answer is sad... When there's an issue that's on-topic, not 1 + 1 basic, and so incredibly common people close vote it out of sheer annoyance of seeing the subject being brought up yet again without any other good reason, but no canonical exists, no existing post is fit to be turned into a canonical, and no-one is willing to write one, we've kind of failed as a community there.
    – Erik A
    Dec 6, 2021 at 21:08
  • 9
    "let's be honest - the question shows zero research effort and very minimal understanding, and we no longer have those close reasons because they're "not nice", and the likelihood of that question ever being useful to anyone else is somewhere between zero and negative infinity" -- I subscribe to every single word, this is basically what I feel voting to close stuff every day. OTOH though it feels bad to not be able to give OP any meaningful direction (dupe links, better close reason, etc) :\ Dec 7, 2021 at 0:38
  • 11
    @ErikA Have we failed as a community? Or... has our community constantly been failed by the people at Stack Exchange Inc. who were supposed to do what is best for us, to the point that the community is fracturing at the seams?
    – Ian Kemp
    Dec 7, 2021 at 7:57
  • @ErikA "...no-one is willing to write one..." That seems to be at the heart of it. The questions are written by amateurs while the answers are written by experts, but amateurs aren't good at writing good questions. What would be the way out? Should we start writing our own questions? Should questions from people below X rep just serve as suggestions of what could maybe be good questions? Or should we start rewriting existing questions? What would you have done differently here? Dec 7, 2021 at 9:00
  • 4
    @Trilarion We should certainly start writing our own questions, if no good one has been asked yet. Unfortunately, I'm not a subject matter expert here, but for the tags I follow, there were 2 of these repetitive basic questions I didn't have a good dupe for, and I've written canonicals on both of them to make the question just broad enough to make the answer cover all common cases, and made the answers community wikis. One was fairly broad so I checked with meta first, which is common (search)
    – Erik A
    Dec 7, 2021 at 9:09
  • 2
    @Ian This one is on us, imo. Nothing is stopping us from writing a canonical, or improving a question to make it good enough to function as one (unfortunately, I don't know C# well enough to write one). That we got disappointed by the company again and again and don't feel the passion we used to to make this place excellent may certainly play a role, but we can't blame them for everything, imo.
    – Erik A
    Dec 7, 2021 at 9:23
  • 5
    @ErikA Your notion that the community must eternally give and be happy about it, and any reluctance to give is a massive failure by that community's members, is exactly the mindset that Stack Exchange Inc. has exploited to build itself into a multibillion-dollar company off the backs of our labour, while giving almost nothing back in return. Would you do free work for a billionaire who doesn't pay taxes? If not, why do you do free work for Stack Exchange Inc.?
    – Ian Kemp
    Dec 7, 2021 at 13:10
  • 7
    @Ian It seems you misunderstood me, you do not need to do anything, but we should not have community members that are so tired of these questions they close them for the wrong reason (leaving the person asking it very confused) but unwilling to invest in creating a canonical. If you want to do absolutely nothing, that's understandable and fine with me, but it's not fine if we go about closing questions for the wrong reasons because we're not willing to do the work required.
    – Erik A
    Dec 7, 2021 at 13:26
  • 3
    There is a shared responsibility, I think. We as a community can't take the full burden; we've been asking for better tools to help people write better initial questions, including finding potential duplicates, but the push to "welcome" question askers has pushed that to the back seat. However, we (experts in our fields) need to be willing to write the canonicals that are somehow to be found by those who seek them. I do feel like the community has been asked, repeatedly, to shoulder a lot of the burden of curating the collection with less help from the company than is received. Dec 7, 2021 at 13:54
  • "[...] because we're not willing to do the work required." I understand this sentence as meaning that there is work required by either party in this situation @ErikA. If that's the case, could you define what is the work required of the asker of the question on main and how they met that work? Evidently, there is a sous-entendre here that I don't believe they've done that required work to start with, but I would be interested to understand how you believe so. Dec 7, 2021 at 18:10
  • 2
    @Félix The work required I mean there is solely on the close voter, not the asker. If you close vote, you must make sure to close vote for the right reason. Nothing more, nothing less. If you think the right reason is because it's asked so frequently that there should be a canonical, that means either finding that canonical, or if it doesn't exist, creating it. If you're not willing to do that, you can either do nothing, or simply downvote if you think it's poorly researched. Both are fine, just never close vote for the wrong reason.
    – Erik A
    Dec 7, 2021 at 18:34
  • 1
    @Trilarion: no one is willing to write one? Well, it's not so hard to get the answer right, but the question is certainly "too broad", "needs details", or "is opinion based". If posted as a regular question, it would likely be closed within seconds. Would such a Q&A help for closing as duplicate? stackoverflow.com/questions/70260105/… (currently deleted, 10k only, so you should quickly answer a few questions in order to see it) Dec 7, 2021 at 19:59
  • 1
    @Trilarion: indeed, this felt more like writing a blog post than a Stack Overflow answer. I don't care about that, but let's see what the community thinks about it. Luckily I don't need to care about rep any more. Dec 8, 2021 at 19:18
12

So, I thought id throw my hat into the ring here.

I have refrained from making comments thus far or modifying the question until this has progressed through the fiery crucible of meta discussion for which all Stack Overflow foot-soldiers truly share the same rank regardless of what tag, nerd, or angst levels they come from. Additionally, my opinion must be tarred with the obligatory "Its my answer, and confirmation bias" brush

To be honest, I am no angel, and have been suspended multiple times for being “UnKInD”, and downvoted and closed my fair share of the most atrocious questions known to Developer-Kind. Notwithstanding, there is a level at which some questions get closed in a way that might not be optimal to the resource we have all helped to create.

I am not saying the question should be reopened, or that being closed as a typo is unjust, just that, there had to have been something more optimal that could have been done to line the pockets of our grandmasters and overloads, *cough* I mean, better the Q&A resource we all use?

There were some better than average qualities about this question:

  1. It wasn’t just some kid’ling doing homework and asking for “Gimme Codez” and “Gimme free lunchez”
  2. The OP had (at least) a basic understanding of the domain, and wrote a coherent question (which is unusual)
  3. The problem is very common at the intermediate level, and it’s a shame there are no wiki style canonical questions that can be pointed to in C# (or at least I could find)
  4. Benchmarking questions should be fostered. There are a lot of suspect ways kids (and professionals) are trying to ascertain real world metrics on performance, and as such its done wrong time and time again.
  5. The OP had gone the effort of making a reproducible example and fiddle to produce the problem (extremely unusual)

There were a lot of suspect qualities about the question:

  1. The title was not future proof and could have been better in hindsight, though could have be made more appropriate by tag experts (after the fact).
  2. The OP was making the classic mistake of trying to perform performance benchmarks in a non-optimized environment, using far from optimal tools.

It brings us to some interesting points regardless…

If a user has the ability to code and to lift code from the framework, to ask a question in a way that passes just about every official Stack Overflow Guideline, and was after knowledge of why the results were not (rightly) as they expected, then we have a surface area for what Stack Overflow is about…

Furthermore, by deduction it can be safely assumed the OP had obviously used this site enough to ask a decent question and capable of researching their own problems… after all they had come this far. Then there must be a gap in knowledge that a Q&A site like this should have been able to fill. Marking this as a typo and claiming it as noise, has no benefit to future users..

So, in my honest opinion (it is what it is).

  • The question should have been edited (even after the fact) with an appropriate title
  • At worst, not closed
  • At best, used as a signpost to a canonical well written benchmarking duplicate (that seemingly doesn't exist).

So C# Jedi, call to arms, create canonical wiki benchmarking duplicate.

2
  • 4
    Nice answer, it's a shame how few people on Meta are willing to express it or support it. I know for a fact that there are many who hold similar viewpoints, perhaps even most of the SO answerers, but as a rule we don't participate in these discussions on Meta. I think that's in part because Meta is not seen as relevant to the goal of helping people. Dec 8, 2021 at 13:12
  • The problem is the question got into the weeds with the whole pre-explaining of results. If you removed all of the speed assumptions, noted the compile types, and aimed at "how do I benchmark StringReader implementations in c#?", and included their code and some of the text, then you would have a question which the current set of close reasons wouldn't close. It is going to require a significant departure from the OP's original question though, and that means you are now solving a different problem. Is that problem really the issue the OP faced? Perhaps, but who really knows aside from them.
    – Travis J
    Dec 9, 2021 at 8:39
1

The question was closed because the close voters decided that the problem was not reproducible, in other words, the question states that their version is slower, but upon inspection, it was faster.

That has nothing to do with typos.

This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.

But it is right there in the close reason! Yeah, that's the real problem here. The close reasons themselves are outdated, poorly grouped, and ridiculously broad (more on that if you are interested). If we had a standardized definition for what we allow as close reasons, this one would surely be deleted.

Benchmarking here is tangential, and should be asked in a separate question.

6
  • 4
    The problem was reproducible, though, you can run the code in the question and get the same results, and the answer was likely to help others, as Ians answer states, confusion around how to benchmark C# code is common and many users that have unexpected results when benchmarking can be helped by the answer that explains how to do it properly. Benchmarking wasn't tangential, it was the entire issue. It's really common the asker doesn't fully understand what the issue is, else they'd have a much easier time researching and resolving it.
    – Erik A
    Dec 7, 2021 at 20:22
  • @ErikA - The problem wasn't reproducible, though. You cannot run the code and come up with the claimed result. I am not sure why you would look at this and think that. In general, benchmarking questions are actually on topic here, when they are properly composed. This one fails in several key areas. First off, "to my surprise my code is over twice as slow as the .NET StringReader"; this is unreproducible... it is a tenet, literally the core premise of the problem being observed, and it is unreproducible.
    – Travis J
    Dec 7, 2021 at 20:32
  • I must have missed something, can anybody please explain? was the secondary part here, literally playing into "this one was resolved in a way less likely to help future readers.". The close reason itself is the problem here, as it addressed too many edge cases and ends up confusing people. If there was one individual aspect here to address perhaps it could have been fixed, but the actual answer would have been, in a sentence, "you used debug instead of release". Benchmarking questions should hopefully be more sophisticated than "you forgot to turn it on right at first".
    – Travis J
    Dec 7, 2021 at 20:32
  • 4
    I don't really get how you can claim it's not reproducible. There's a fiddle. You can click on it, and you get the same numbers. That's reproducible to me. You can argue that the question hasn't explicitly stated that debug mode was used, but the fiddle solves that, since for performance it explicitly refers to numbers produced with the fiddle. We need to stay within reason here, if someone has shared all relevant code, doesn't know it's a configuration problem, but has created an online runnable example that demonstrates the problem, that should be enough.
    – Erik A
    Dec 7, 2021 at 22:36
  • 1
    "The close reasons themselves are outdated, poorly grouped, and ridiculously broad" - wish I could upvote the post multiple times just for that sentence alone.
    – l4mpi
    Dec 8, 2021 at 16:28
  • @ErikA - You can run their example code, but that code isn't the problem they encountered. They encountered the situation where they observed their code being outperformed and posted a question about it. That original problem, and situation, are not reproducible, because the premise of the question was incorrect itself.
    – Travis J
    Dec 8, 2021 at 20:27
-2

There is one very constructive way for users to improve closed benchmark questions on SO: Cast a delete vote to delete the question. This ensures that the question is removed from view and helps improve SO by taking out the trash.

Benchmark questions are almost always off-topic due to scoping issues (too broad) or other problems (e.g. requiring specific hardware or large datasets). Furthermore, the vast majority of such questions is not useful, because benchmarking a specific piece of code only helps the OP and thus does not help in building a knowledge repository (this was called too localized in ye olde times).

There might be room for a few canonical questions related to benchmarking, if they can be appropriately scoped. There might also be questions related to benchmarking that are useful and on-topic (e.g. questions arising while writing a benchmark library). However, most benchmark questions on SO should be closed and deleted, which includes the linked post.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .