42

Some time ago, the top-voted, accepted answer to a Stack Overflow question was featured on The Daily WTF (in the meantime, another answer became top-voted). Since then, I have been wondering how prevalent these kind of answers that are intrinsically flawed yet either accepted or top-voted are, to see if there would be need for a feature to flag them.

First, let us define what we are looking for here. This is not entirely dissimilar, but still different from the obsolete answer matter: I am wondering about answers that could have been known as invalid at the time of posting, not answers that simply did not age well, the two cases probably warrant different handling (e.g. an obsolete answer might still be useful and simply need a reminder of the year in which it was posted, to allow visitors to make an informed decision).

If we think about what makes voting and accepting on answers work on SO, it is that the asker accepts or the visitor upvotes the answer because it works for them, and who am I to question their judgment in the general case? Ah, but what if the answer appears to work, enough so for them to accept/upvote it, but it doesn't actually work, in the sense that will blow up later in a way that is hard for them to realize right now if they are not informed of the issue? I thought long and hard about it, and concluded that I am interested in answers that feature one or more of these issues:

  1. race conditions
  2. security vulnerabilities
  3. technology-independent time bombs (e.g. suggesting printf("Year is 19%d", localtime(now).tm_year); in 1996)
  4. localization bombs*

All four work in many instances except some very important ones that are hard to test for (or in the third case, hard to remember to test for). I may have missed some, though I don't think there could be too many. Note that case 3 does not include the problem of suggesting obsolescent techniques, like gethostbyname() for instance (which does not play well with IPv6, for one), which are more context-dependent and are thus better handled in the context of obsolete answers.

Also, it is not a problem if an answer is fundamentally sound but does happen to feature, say, a security vulnerability (which, if it needs to be reminded, is one of the fundamental use cases SO is meant to address): this can already be handled with editing, either directly or with edit suggestions. The issue here are answers that, like the one that prompted this research, are so fundamentally flawed that fixing them would require deleting them or changing them into a completely different answer altogether, neither of which is ethical for an editor to do.

Answers with these criteria are not, in my humble opinion, correctly handled by the SE engine currently: as we've seen editing is a non-starter, and simply drowning them by downvoting them and upvoting the correct answer did not work in the case of the answer that prompted all this (until it was featured in TDWTF, at which point it was corrected at least on the voting side); in fact, between the main two answers of that question the incorrect one was actually posted last, so the incorrect one did not even have a head start (and this also means this is not related to the fastest gun in the west issue). I've been thinking of a system of votes-to-challenge, similar to votes to close, at which point the asker, if it was accepted, and the upvoters would be messaged to inform them of the issue and ask them to potentially reconsider their acceptance/upvote in the light of this new information. But before I can suggest this I need to figure out if this phenomenon is actually significant.

The Research

I've been working on and off this subject for some time, and took advantage of today's holiday to work on it some more. Apparently people here believe in data, and though it is not necessarily easy to express it as such, I have been trying. I have run a query on SEDE on the SO dataset trying to get a list of potentially problematic answers I could then check by hand, on the assumption that the answer in question would have received a comment telling about one of these issues.

Before I list my findings, it is important to note this should be considered a tiny tiny sampling that likely vastly under-evaluates the problem. For one, the original answer that prompted all this would not even have been found with these criteria at the time, since while it had a comment telling about the issue that comment did not contain any of the words I searched for (and searching for other words yielded a lot more noise). Second, there are probably an order of magnitude more such problematic answers to which no comment warning about the issue was left (while if there was dedicated UI to challenge such an answer it would have been more likely for them to be challenged as such). Third, even in its current form the query generated a lot of noise (comments about stack traces, braces, rubyracer, disgrace, comments that answer @LightnessRacesinOrbit, etc.), such that I stopped after about half of the 413 returned rows, which are themselves only a sampling of the whole SO dataset (a.id > 20000000 and a.score > 10). Nevertheless, I did find this:

Admittedly, you do find the same issue (disabling SSL cert checks) multiple times (in fact, in one case not listed above it was even specifically asked for); also, I cannot vouch for the fact these are all actual races or security vulnerabilities, as I can't tell for sure, only that the commenter made a credible case.

I think I have now worked on it enough and have enough data to at least launch the discussion here. What do you think? Should this matter be explored further? What should be done next?

*that is, issues that show up when either:

  1. processing non-ASCII text (e.g. carelessly using byte arrays for storing text, not paying attention to encoding in a protocol, but also more subtle issues such as comparing strings by doing a character by character comparison, even with Unicode, which would result in café with é directly encoded as U+00E9 being considered different from café where it is encoded as U+0065 followed by U+0301: a combining acute accent; the same issue exists for e.g. Hangul syllables, used in Korea), or
  2. running under a specific locale or some specific locales (e.g. using has_suffix(uppercase_string(filename), ".AVI") which could fail under a Turkish locale because uppercase_string("foo.avi") could well give "FOO.AVİ" (note the dot), but also using date or number formatting functions for Internet protocols without specifying a "C" or "POSIX" locale, which results in the user locale being used)
5
  • 5
    excellent question. Just curious what is unclear about the question that someone wants to close it? May 2, 2015 at 11:00
  • 1
    Do you intend to include cases where the OP is doing two things very wrong, but only asking about one of them, and the answer fixes only that one and leaves the other very wrong thing in its sample code? There are an awful lot of those… and I'm not sure they're a bad thing. (I personally always try to point out and correct other mistakes, especially important ones like races and security vulnerabilities, or things that may not show up in development testing but will be hard to debug after deployment, but I don't think that's necessary to a good answer, and I've upvoted plenty of that don't.)
    – abarnert
    May 3, 2015 at 21:58
  • @abarnert As far as I can tell from your description, I'm not including this case, as long as it's reasonable to edit the wrong part of the answer while leaving the essence of the answer intact. May 4, 2015 at 8:49
  • To make it more concrete: Imagine the question is asking how to get the output back from system, but their example is also including unvalidated and unescaped user input in that system call. An answer will probably involve pretty complete sample code (using popen or the subprocess module or whatever's appropriate for the language), which will then be bad code (because it's still passing unsafe input to the shell), but may get upvoted anyway, because it's a good answer. I think that's a separate problem to what you're asking, but I wanted to make sure you do too.
    – abarnert
    May 4, 2015 at 8:57
  • It seems it's more or less the same problem as here and here.
    – Bruno
    May 29, 2015 at 21:19

3 Answers 3

12

These have to be handled on an individual basis, unfortunately.

If you find one:

  1. Prepare to be mocked mercilessly if the answer actually displays a greater depth of programming knowledge than your complaint.
  2. Leave a comment pointing out the problem.
  3. Bring it up in a related SO chatroom.
  4. Watch it downvoted to oblivion. This often gets the answer author's attention in a way that a comment will not. Especially in case of massive ego.
  5. If you were foresighted enough to post a really-correct answer before step #3, collect upvotes.
  6. Lather, rinse, repeat
11
  • 2
    One would hope that #4 only happens when #5 has also happened. An answer with a documented technical flaw that's otherwise correct apart from that flaw is still far better than no answer at all. Maybe editing/suggesting an edit in addition to leaving a comment is a better approach?
    – aroth
    May 3, 2015 at 6:28
  • 11
    Watch it downvoted to oblivion. how many years should a person spend on watching it? With old heavily upvoted question, I am not sure I will live long enough to see the number of downvotes to decrease at least in half. May 3, 2015 at 7:14
  • 3
    I agree with Ben's observation. Ironically, he and I were even participants in a question that is relevant to this very discussion. He and I both pointed out a fundamental flaw in the OP's design choice, but two other answerers insisted on providing implementations with exactly that flaw, one of which got accepted. But based on informal observation, my experience is that this happens rarely and would be hard or impossible to automate detection anyway. May 3, 2015 at 7:20
  • @aroth : Just a reminder that we are in a context where the answer is fundamentally flawed in a way that no ethical editing can fix it. May 3, 2015 at 9:15
  • @PierreLebeaupin - Fair enough. I missed that in all the text. I mostly just saw the example cases (race conditions, security flaws, etc.), which I'd generally say are not fundamental flaws that cannot be fixed with an edit. To me a fundamental flaw means "this can't be used to solve the problem at all, because it's completely incorrect", which is a different class of things than "this may solve the problem, but it will introduce other, potentially worse problems if used as-is".
    – aroth
    May 3, 2015 at 9:43
  • @BenVoigt Interesting suggestion: more or less, using chat as a cabal. I'll try it the next time I see one of those; unfortunately I am not competent enough in the subjects discussed to do so in the answers I linked to (even SQL: I know MySQL and SQLite, but not SQL Server, and there does not appear to be a solution for the problem in standard SQL) May 3, 2015 at 9:54
  • @aroth: 5 is not required for 4. Frequently there will be some correct answer already posted by someone other that the one leading the witch hunt.
    – Ben Voigt
    May 3, 2015 at 12:31
  • @peter indeed, I'd given up on that question before you got involved with it. Good to see a correct approach that solves the XY problem the poster was having. Although I'm not sure why you didn't use a collection initializer, your code is much longer than it needs to be.
    – Ben Voigt
    May 3, 2015 at 12:33
  • 1
    @BenVoigt: "I'm not sure why you didn't use a collection initializer" -- thanks for the input. But I'm not sure what you mean; I do use collection init where I can, but for KeyValuePair<BloodType, int>[], the compiler doesn't know how to init KeyValuePair<TKey, TValue>, so while the array itself can (and does) use the collection init syntax, I have to explicitly init each kvp object. There are some concessions to clarity in the implementation, but as far as collection init, I don't see a way to make it much more concise. Please feel free to add a comment there though if you see a way. May 3, 2015 at 16:07
  • @PeterDuniho: If the OP's design has a fundamental flaw, an answer that showed how to do what the OP asked, but also explains why it's a fundamentally flawed idea, can still be good. The implementation of the OP's design will probably both be a useful explanation of how to implement such things, and also code that can be used to demonstrate the design flaw rather than just explaining it abstractly. And it might be ethical to edit the answer into such a thing by just adding the explanation.
    – abarnert
    May 3, 2015 at 22:02
  • @PeterDuniho: I can't find a great example off-hand of that kind of answer, but for a mediocre one, here's one I wrote for a user who wanted to know how to use eval to parse CSV files in Python. (If Chris Wesseling's answer hadn't already existed, I would have also included something like that up at the top, instead of just referencing his.)
    – abarnert
    May 3, 2015 at 22:06
4

We have mechanics (the various close reasons, historical lock, etc) for labeling questions that, however highly upvoted they may be, are considered poor. There's no equivalent for an answer; they can be deleted, but that's not considered the appropriate thing to do with an answer that is merely wrong.

I suggest a new mechanic, similar to closevotes, for answers. Let's call it "wrongvotes". If an answer gets wrongvoted, that would have two effects: a big obvious warning message would be plastered across the top of the answer, and it would no longer be pinned to the top of the answer list even if it was the accepted answer. Wrongvoting requires you to specify a reason, which goes into the warning message.

Made-up example (with a real flaw):

This answer is fundamentally flawed. Some other program might create the file in between when you check whether the file exists, and when you create it. Instead, just go ahead and create the file, using O_EXCL, and the open call will fail if the file already exists.

To check whether a file exists, before creating it, use stat or access.

We could fake this mechanic by editing such warnings into answers; that might be a reasonable way to test it out.

4
  • 3
    The thing is - who can cast the wrong votes? What happens if the one who cast it is actually wrong? Who can take down the warning message? Editing the warning banner seems like a good solution, but I'm afraid that the author doesn't like it and will fight to remove it.
    – nhahtdh
    May 4, 2015 at 5:14
  • I'm afraid @nhahtdh may be right. In the past week, I've seen 3 answers with exactly this problem, and in each case, when I just wrote a comment explaining the problem, the answerer got angry and defensive. If I actually edited this into the answers, I can't imagine they'd take it better. Of course if someone's hurting other people by being an idiot, that bothers me more than someone being angry, but given how easy it is to just roll back an edit or flag it, I don't know that it would stand that often.
    – abarnert
    May 4, 2015 at 9:00
  • 1
    I think such a banner is the best way to handle this (as well as a notification to both asker and answerer), as for "who", I would suppose it would be a new privilege (based on reputation) and that whoever can put a banner also has the privilege to remove it. I would also suggests that such a banner be placed immediately (first suggestion) and then become "disputed" (gather votes of its own, with a high "down/up" ratio after at minima 5 votes removing it). Of course, it also means that such banners would likely require a dedicated moderation queue (like close votes) to attract attention. May 4, 2015 at 12:42
  • Historical locks are probably the worst of all mechanisms on this site, especially when it applies to questions like those mentioned here: you can no longer vote, comment or edit to point out this sort of problems in these answers.
    – Bruno
    May 29, 2015 at 21:24
1

The problem for these bad answers is that the voting system does not always correct them, by up voting the correct answer and down voting the incorrect answer.

How can that be? I suggest it can happen only when a horde of inexperts can out vote the handful of experts. We want to prevent that happening. I suggest giving the vote of an expert more weight than the vote of a beginner. That is, when an expert casts an up vote, have it increase the score by +2 or -2 rather than +1 or -1.

But how can we tell who the experts are? We could make it a simple privilege, acquired when your reputation exceeds a particular threshold (10000?).

I don't think you would need to demonstrate that the purported expert was an expert on the precise subject of the question for this to work. The flip side of the Dunning Kreuger effect is that someone who is an expert on something closely related (all the questions on SO are closely related in that respect) knows the areas in which they are not experts.

1
  • 3
    That's a tricky one, you'd need to be an "expert" by having gained enough reputation on that tag. This can also cause problems by privileging users who are more "vocal" (i.e. they produce a lot of answers with similar flawed advice). The problem is more fundamentally that there are too many developers out there who will upvote the first or easiest solution they come across, whether it's the correct way to solve their problem or not.
    – Bruno
    May 29, 2015 at 21:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .