Some time ago, the top-voted, accepted answer to a Stack Overflow question was featured on The Daily WTF (in the meantime, another answer became top-voted). Since then, I have been wondering how prevalent these kind of answers that are intrinsically flawed yet either accepted or top-voted are, to see if there would be need for a feature to flag them.
First, let us define what we are looking for here. This is not entirely dissimilar, but still different from the obsolete answer matter: I am wondering about answers that could have been known as invalid at the time of posting, not answers that simply did not age well, the two cases probably warrant different handling (e.g. an obsolete answer might still be useful and simply need a reminder of the year in which it was posted, to allow visitors to make an informed decision).
If we think about what makes voting and accepting on answers work on SO, it is that the asker accepts or the visitor upvotes the answer because it works for them, and who am I to question their judgment in the general case? Ah, but what if the answer appears to work, enough so for them to accept/upvote it, but it doesn't actually work, in the sense that will blow up later in a way that is hard for them to realize right now if they are not informed of the issue? I thought long and hard about it, and concluded that I am interested in answers that feature one or more of these issues:
- race conditions
- security vulnerabilities
- technology-independent time bombs (e.g. suggesting
printf("Year is 19%d", localtime(now).tm_year);
in 1996) - localization bombs*
All four work in many instances except some very important ones that are hard to test for (or in the third case, hard to remember to test for). I may have missed some, though I don't think there could be too many. Note that case 3 does not include the problem of suggesting obsolescent techniques, like gethostbyname()
for instance (which does not play well with IPv6, for one), which are more context-dependent and are thus better handled in the context of obsolete answers.
Also, it is not a problem if an answer is fundamentally sound but does happen to feature, say, a security vulnerability (which, if it needs to be reminded, is one of the fundamental use cases SO is meant to address): this can already be handled with editing, either directly or with edit suggestions. The issue here are answers that, like the one that prompted this research, are so fundamentally flawed that fixing them would require deleting them or changing them into a completely different answer altogether, neither of which is ethical for an editor to do.
Answers with these criteria are not, in my humble opinion, correctly handled by the SE engine currently: as we've seen editing is a non-starter, and simply drowning them by downvoting them and upvoting the correct answer did not work in the case of the answer that prompted all this (until it was featured in TDWTF, at which point it was corrected at least on the voting side); in fact, between the main two answers of that question the incorrect one was actually posted last, so the incorrect one did not even have a head start (and this also means this is not related to the fastest gun in the west issue). I've been thinking of a system of votes-to-challenge, similar to votes to close, at which point the asker, if it was accepted, and the upvoters would be messaged to inform them of the issue and ask them to potentially reconsider their acceptance/upvote in the light of this new information. But before I can suggest this I need to figure out if this phenomenon is actually significant.
The Research
I've been working on and off this subject for some time, and took advantage of today's holiday to work on it some more. Apparently people here believe in data, and though it is not necessarily easy to express it as such, I have been trying. I have run a query on SEDE on the SO dataset trying to get a list of potentially problematic answers I could then check by hand, on the assumption that the answer in question would have received a comment telling about one of these issues.
Before I list my findings, it is important to note this should be considered a tiny tiny sampling that likely vastly under-evaluates the problem. For one, the original answer that prompted all this would not even have been found with these criteria at the time, since while it had a comment telling about the issue that comment did not contain any of the words I searched for (and searching for other words yielded a lot more noise). Second, there are probably an order of magnitude more such problematic answers to which no comment warning about the issue was left (while if there was dedicated UI to challenge such an answer it would have been more likely for them to be challenged as such). Third, even in its current form the query generated a lot of noise (comments about stack traces, braces, rubyracer, disgrace, comments that answer @LightnessRacesinOrbit, etc.), such that I stopped after about half of the 413 returned rows, which are themselves only a sampling of the whole SO dataset (a.id > 20000000 and a.score > 10
). Nevertheless, I did find this:
- How to get connect to core database in sitecore (found from a previous version of the query which did not filter by votes)
- Unable to verify leaf signature
- SQL Server Insert if not exist
- error in send email using Mandrill (php)
- How to solve error “Missing
secret_key_base
for 'production' environment” on Heroku (Rails 4.1) - Qt Creator, ptrace: Operation not permitted. What is the permanent solution?
- To open “Eclipse” you need to install the legacy Java SE 6 runtime
- Node.js EACCES error when listening on http 80 port (permission denied)
Admittedly, you do find the same issue (disabling SSL cert checks) multiple times (in fact, in one case not listed above it was even specifically asked for); also, I cannot vouch for the fact these are all actual races or security vulnerabilities, as I can't tell for sure, only that the commenter made a credible case.
I think I have now worked on it enough and have enough data to at least launch the discussion here. What do you think? Should this matter be explored further? What should be done next?
*that is, issues that show up when either:
- processing non-ASCII text (e.g. carelessly using byte arrays for storing text, not paying attention to encoding in a protocol, but also more subtle issues such as comparing strings by doing a character by character comparison, even with Unicode, which would result in
café
withé
directly encoded as U+00E9 being considered different fromcafé
where it is encoded as U+0065 followed by U+0301: a combining acute accent; the same issue exists for e.g. Hangul syllables, used in Korea), or - running under a specific locale or some specific locales (e.g. using
has_suffix(uppercase_string(filename), ".AVI")
which could fail under a Turkish locale becauseuppercase_string("foo.avi")
could well give"FOO.AVİ"
(note the dot), but also using date or number formatting functions for Internet protocols without specifying a "C" or "POSIX" locale, which results in the user locale being used)
system
, but their example is also including unvalidated and unescaped user input in thatsystem
call. An answer will probably involve pretty complete sample code (usingpopen
or thesubprocess
module or whatever's appropriate for the language), which will then be bad code (because it's still passing unsafe input to the shell), but may get upvoted anyway, because it's a good answer. I think that's a separate problem to what you're asking, but I wanted to make sure you do too.