-19

When we ask users to add their code as a minimal reproducible example via Stack Snippets and tell us what they expect the code to do, the snippet is quite helpful but we have no way to verify if an answer actually solves the problem. That's why I think, there should be something like tests or simplier an expected result.

Options

Tests

Some code playgrounds like codesandbox even run complete testing suites like Jest. Might be more complicated to implement but is more flexible.

Expected result

I contributed to foundeo/cfdocs a while ago and they have tests that check the last line against what is supplied via the examples.result key. Pretty simple, could work in simple use cases. Data structures like array and objects have to serialized but that's okay.

Feedback collected from comments

Pros Cons
Answers can be sorted, compared and weighted better Stack snippets only apply to a minority of languages
Answer posters can check their code before posting Ineffective/restricted output
Multiple possible solutions
Additional time needed to write the tests

Prospects

Maybe there could be some simple benchmarks, too. That'd add another criterium to sort answers. Some questions already have answers doing that manually so they could really benefit from it.

14
  • 1
    "but we have no way to verify if an answer actually solves the problem." No? Can't you just log the result and see if it's correct? And even if this were to be introduced, who's gonna write these tests?
    – Cerbrus
    Dec 17, 2021 at 9:01
  • @Cerbrus Well, not everybody uses a stack snippet in their answer and even if they do maybe they don't log anything to the console. Of course, we can do that manually but that's additional effort for each answer.
    – shaedrich
    Dec 17, 2021 at 9:07
  • @Cerbrus Ideally, the question poster should also add the tests since they need to know what they expect. Even when someone is not familiar with testing per se, these tests would be very simple most of the time. And if the question poster doesn't provide them, the person who posts the answer can do it themselves then.
    – shaedrich
    Dec 17, 2021 at 9:10
  • 5
    How many Q&A would actually benefit from this? Leaving aside that snippets don't apply to most languages, what fraction of questions are reasonably testable? For example, what if the requested output is just inefficient/restrictive? What about question "tests" that miss important edge cases, or when there is no single solution an answers have to pick a case to solve? Most importantly, what happens if an answer does not "solve" the question – does it get marked somehow, or even not published? Dec 17, 2021 at 9:26
  • @MisterMiyagi Good point about the language! Thanks for your constructive feedback by the way. Since I thought about implementing the tests to the stack snippets, they'd be copied from the question to the answer and could be changed (unsure if that's better). If an answer does not "solve" the question wouldn't it be better to post a comment instead? Your point about different solutions is also valid but that collides a bit with requesting a precise expected result of the question poster already.
    – shaedrich
    Dec 17, 2021 at 9:31
  • @MisterMiyagi However, your point about the languages is more an argument against stack snippets than against tests for them. Yet stack snippets got implement. So this point is not a con per se, it's more a note on the possible prioritization.
    – shaedrich
    Dec 17, 2021 at 9:54
  • 2
    Stack snippets are useful for the JS / CSS and HTML tags, which are some of the largest tags on the site, so it's not weird that it got implemented. These tests, however, are only useful for a small portion of those questions.
    – Cerbrus
    Dec 17, 2021 at 9:58
  • @Cerbrus You're right. It's comparably easy to test JS compared to HTML and CSS.
    – shaedrich
    Dec 17, 2021 at 10:06
  • And even then, only a portion of JS questions are testable, assuming tests are even useful for a question... Questions should be small and contained, I wouldn't generally expect significant output
    – Cerbrus
    Dec 17, 2021 at 10:07
  • Wasn't there a recent meta question about this same thing? Dec 21, 2021 at 17:27
  • "we have no way to verify if an answer actually solves the problem" What do you mean? Just click "run" on the Stack Snippet and you can see for yourself if it solves OP's problem. That's the entire point of Stack Snippets.
    – TylerH
    Dec 21, 2021 at 17:31
  • That's manual verification based on subjective observation. That works to a certain extent but is not that precise.
    – shaedrich
    Dec 21, 2021 at 19:16
  • It is exactly as precise as it needs to be.
    – Kevin B
    Dec 21, 2021 at 19:18
  • If you say so. "as far as I can tell" it is then.
    – shaedrich
    Dec 21, 2021 at 19:19

1 Answer 1

5

A question author is free to write their test cases as code in their question. Some actually do that. Then an answer can show they are covered by making a snippet and including the test cases. Even if an answer doesn't do that, any reader can copy the code and the tests and try them out.

The test cases can be as simple as:

foo(4, 0); // expect false
foo(4, 1); // expect false
foo(4, 2); // expect true

Do we really need anything more formal than that?

On one hand, it would help because it is additional clarity for the question.

On the other hand, I feel it slips too much into "write this code for me" territory. To clarify: Stack Overflow is about writing code and many questions are reasonably expected to be answered with code in some form. The problem is that Stack Overflow is not only about writing code. Visitors on the website should not expect copy-pasteable production-ready code. Answers can outline a technique, and they can include code that demonstrates it, but not every question should just have a full implementation as an answer.

While there is code on Stack Overflow, it is not a website where you just shop for code. Find which one works and grab it. It's much more important to understand what the solution is. Runnable snippets help in that regard because you can immediately see a result but there are many languages that do not have runnable snippets. Just from the top 5, there are: Python, Java, C#, and PHP. There is no way to verify the code in those* and yet they are still useful.

So, I do not think we should mandate tests be included with questions. Especially since not even all questions benefit from this.

I would say that if a question asker is diligent, they would add such tests to clarify what they are after when it makes sense to have it. And tests would not be included if that would not make sense. Questions answerers can then also use the tests, if they intend to provide a clear and useful answer.


* There is a separate discussion about adding some sort of way to run code live for those and other languages directly on Stack Overflow. There have been suggestions about this before. It is a separate discussion.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .