The idea that Stack Overflow would maintain automated tools that test code and check it is, quite frankly, absurd.
The top 10 tags are:
- JavaScript
- Python
- Java
- C#
- PHP
- Android
- HTML
- jQuery
- C++
- CSS
Each of these has a variety of versions available. And a variety of tools to test the code for what is good or bad practices. The code can also run on a variety of platforms - different operating systems, or perhaps different browsers for the web languages.
Just because I am more familiar with the subject matter: JavaScript has 11 "versions" out. These are the various ECMAScript specifications: ES1, ES2, ES3, ES5, ES6 (a.k.a. ES2015), ES2016, ES2017, ES2018, ES2019, ES2020, ES2021. That means that code should be able to be checked against 11 different things. Plus one more every year as the new specs come out.
Furthermore, the versions do not actually help. There is a lot of code that does not conform to a version because it conforms to a proposal. Features for the ECMAScript spec start out as proposals and then after being ratified are added to the main specification. There are environments and tools that do allow you to use a feature from a proposal which is not officially ratified yet (but presumably on track to be, eventually).
Aside from the problem of "what is valid code, even" there is an additional problem to figuring out "what is invalid code, even". There are three major linting tools for JavaScript code: JSLint, JSHint, ESLint - which one should the automated workflows use? Moreover, once they pick one what configuration do they use with it? Very simple code example if (a == null) return;
- this is valid (not syntactically wrong) code but is it correct? JSLint says "no" because it wants you to use ===
instead. ESLint has the eqeqeq rule to determine whether you need ==
or ===
. Usually it is recommended that you use strict equality with ===
however, that can introduce a bug with the code here. This is a real thing I have done - I installed a linter, it told me to change ==
to ===
and it broke some functionality. Because loose equality was intended when the code was written.
This is just a very, very simple example of just one thing. For one language. And that language has 11 different versions out there. It is ludicrous to expect this can all work uniformly for all JavaScript code past and present.
There are other languages to consider as well. HTML and CSS in particular are not exactly "runnable" in the same sense as JavaScript, Python, or Java are. They can be analysed via static checking tools but not really checked if they "work". How do you verify <input type="text" />
does what it is supposed to do? It is valid HTML but what if the question asked for how to make a button - the correct code would be <input type="submit" />
or <button></button>
. Or it might even be <a class="btn"></a>
. Which one is it? Are any "unsafe"? How do you automatically check for it? CSS is even more elusive without some HTML elements to style - a CSS rule could be valid but "harmful". Like turning all text green on a green background instead of only some of the text green on non-green background.
All of the above assumes you even have code that is complete in some way. Here is some code from an answer of mine (for the record, it is Java):
return LongStream.iterate(0,
nextYear -> ((nextYear + amount) * (100 + interest)) / 100
)
.skip(years)
.findFirst()
.getAsLong();
This is the entire code block. No tool I am aware of will be able to run this. Neither do I know of an automated process that can even make it runnable since amount
, interest
, and years
are missing. In this case it is at least somewhat easy to assume they are integers (other types do not much sense for the code), however, a lot of stuff is still missing. It would also be, to my knowledge, impossible to verify if the code is correct even if it was somehow converted to be runnable. As a side note: Java has 21 major versions out - that partial code should somehow be able to be automatically ran in all of them.
It is time to talk about the automated running process. As shown, there is a lot to consider. But somehow the automated process has to be able to not only load the code and run it thus proving it is syntactically OK. You also expect the automated process to test the code:
However you can use logic to find commonly used built in function names in the Answer's code to test those functions by themselves.
No, bluntly you cannot. Sure, it is possible in theory but to my knowledge there is no way to generically create tests out of source code. Here is a very simple example in JavaScript:
function a(x, y) {
return b(y, x);
}
function b(x, y) {
return x * y;
}
a
calls b
and b
uses the arguments that are passed into it. To generate a test, you need to:
- Find that
a
and b
are functions.
- They expect two arguments.
- To call them with two arguments.
- The arguments are of the correct type.
The above example is exceptionally simple and there are tools that can statically analyse the code and can determine that the arguments should be numbers, which is correct. However, if *
was +
then it static analysis cannot reveal what x
and y
should be in b
and consequently in a
because +
could apply to numbers or strings. Or a mix. Is a("world", "hello")
the correct invocation or a(2, "4")
? Or is one of these a bug? Maybe both. You expect a lot out of automation that it cannot deliver. The problem just gets larger and larger with each language, versions, environment, etc. That has to be considered.