38

Increasingly, I am seeing questions posted to Stack Overflow like this one:

Simultaneous overfitting and underfitting

I'm not a machine learning expert, but it seems to me that this question is under-specified and somewhat speculative.

What should we do with these kinds of questions?

Play the MCVE card? I suspect that coming up with enough code and data to demonstrate the question is a bit onerous.

Kick them over to Software Engineering or Computer Science? Software Engineering is a bit of a wildcard, and I'm not sure there's enough intellectual rigor in the question to survive at Computer Science.

In the absence of a migration, what can we say to new users in the way of guidance so that their question can be improved?

6
  • 13
    ML questions (conceptual/theoretical ones) are on topic on Cross Validated.
    – user2285236
    Commented Jan 1, 2019 at 21:02
  • 3
    There's Data Science; no idea if they'd want this sort of question though.
    – Ben
    Commented Jan 1, 2019 at 21:10
  • 5
    Definitely on topic for data science/cross-validated. It looks like OP is asking for a resource but actually s/he is describing a scenario and seeking its solution. As a general guideline, I think ML toolkit/software questions should come to SO and theory/conceptual questions should go to one of the two websites.
    – Autonomous
    Commented Jan 2, 2019 at 4:31
  • 1
    I think that Machine Learning questions can be easily on topic at Stack Overflow. That one was looking for a discussion though, which is not really on topic at any of the exchanges in any format.
    – Travis J
    Commented Jan 2, 2019 at 19:44
  • @ParagS.Chandakkar Is there a really strong reason to ban code questions on Data Science when they are based on data science technology? Geographic Information Systems allows code questions related to GIS technologies and doesn't seem to have any major problems because of it. I can see excluding general purpose language usage questions, but not ones that are actually about making effective use of data science software and libraries.
    – jpmc26
    Commented Jan 4, 2019 at 0:39
  • Not really. My comment was only applicable to questions posted on SO to start with. If a question was posted on DS initially, it is upto that community to decide. I have not used Data Science community enough to know what kind of questions people ask there. I have used Cross Validated and I see mostly conceptual questions there. Moreover, on SO, you will probably find more users who can help you regarding the code aspect.
    – Autonomous
    Commented Jan 4, 2019 at 0:46

3 Answers 3

25

In the question you linked, the main question posed is:

Is this a real issue and, if so, where can I find a good introductory discussion on how to manage it?

I would close this as too broad, since it's asking multiple questions as well as a recommendation for an off-site resource.

I would hold back on migration, as questions like these usually wouldn't receive great answers here. It should be the onus of the user to register on the other network sites, naturally take the tour, and figure out whether it will be on-topic there.

If one is very familiar with another site on the network, and feel that it will be better received there, then by all means suggest that the user cross-post there instead (in the comments) after the question has been closed. These questions should not be custom-flagged for migration unless there are great answers already posted to it (and of course asked less than 60 days ago) - which we will migrate on a case-by-case basis (discussing with the target site's mods if necessary).


Here are a few comment suggestions we could use to guide the users (feel free to edit or append your own):

  • General canned response for off-topic questions:
    Please take some time to read the documentation in the [help], especially the sections "[What topics can I ask about here?](/help/on-topic)" and "[What types of questions should I avoid asking?](/help/dont-ask)".

  • A variation on one seen here:
    If applicable, please provide a [reproducible example](/help/mcve) to help us understand your question better. As this may be more of a data modelling/implementation question, it will also be better suited for [Data Science](https://datascience.stackexchange.com). If you need statistical help than your question may be more relevant on [Cross Validated](https://stats.stackexchange.com).

  • A variation on one seen here:
    Your question is more concerned with theoretical aspects of machine learning. Please ask these kind of questions on [Cross Validated](https://stats.stackexchange.com) or [Data Science SE](https://datascience.stackexchange.com) after this question has been closed/deleted.

3
  • 2
    Actually the main question posed is in the title. In the absence of a migration, what's the best way to guide new users about such questions? I'm not talking about the askers who clearly lack the requisite expertise; those are easily dispatched.
    – Robert Harvey Mod
    Commented Jan 1, 2019 at 22:40
  • 2
    "good introductory discussion" is pretty much the main problem. We don't "discuss" here.
    – Braiam
    Commented Jan 1, 2019 at 23:33
  • 1
    By "here", if you mean on StackOverflow, I agree. But on cross-validated/data science, you will need to "discuss". I think the OP worded the question wrongly to give a sense that s/he is seeking a discussion instead of a solution to the question.
    – Autonomous
    Commented Jan 2, 2019 at 4:45
12

tl;dr: While there is no clear-cut rule of thumb to this, we might be able to, in most cases, educate our users on whether it belongs here or somewhere else, and ultimately what they should bear in mind when asking. MCVE's in machine learning (and even more so in deep learning), are very hard to make.

I've observed the machine learning environment around Stack Overflow for a while, and noticed that a significant share of questions indeed do not seem to belong here. This includes deep learning questions, which is another beast in itself: due to the unique concept of neural network design, deep learning may be perceived as something distinct from traditional machine learning methods, to the point of resembling an engineering task. This answer will have a special focus on deep learning, but the advice should apply to other machine learning questions nonetheless.

In the absence of a migration, what can we say to new users in the way of guidance so that their question can be improved?

This can be a bit.. uh, misguided? ^_^ It's true that we might need to communicate our position better, but a fair share of these questions are not a proper fit in the first place, and our users will still have to be told so and why they are. The question linked above is clearly not appropriate. So do many others that show up regularly on the site (example). When thinking whether or not to close a question related to machine learning, we admit the same standard reasons for closure as for any other question. In particular, a fair bunch of questions in this field tend to come from clueless users asking for "some pointers", making them too broad for Stack Overflow. With that aside, what makes the subject more confusing than others is their general topicality, and whether we can treat them as programming questions.

So, is it programming or not?

We're living in times where machine learning is interestingly ubiquitous, so much that people may just mix all concepts in computer science together to create this amalgamate of ideas related to computer programming. Some (often sensational) magazines and blog posts have employed the term Programming 2.0 when referring to the use of artificial intelligence algorithms to solve problems with a computer (a term which I personally reject and despise). However, we can, and should define boundaries based on the fields of expertise that are involved. As much as one would like to split the waters, I believe that there is no clear consensus in the Stack Overflow community on this, but my current stance is the following:

  1. First, it's safe to say that we prefer not to address purely theoretical matters. For such questions that do not seem unsalvageable other than that, I usually place this custom closure message:

I'm voting to close this question as off-topic because it's purely specific to machine learning rather than programming. You may visit Cross Validated or Data Science SE instead.

  1. It gets harder when a concrete machine learning model is presented, followed by a question on how to solve some concrete behaviour or how to improve its evaluation metrics. Questions seeking advice on "How do I improve this model?" are usually open ended, and will often attract generic recommendations that involve much more concepts than those in the scope of the site. They should be closed as Too Broad until they are severely narrowed down to a concrete concern. Providing advice here is tricky unless you are confident in proposing migration.

  2. If the question presents a very precise problem statement and is (at least close to) a proper MCVE that reproduces the problem, then it might just seem to be worth improving. However, your mileage may vary, because such a question may still require an expert in machine learning to be well answered.

It's also worth noting that there are machine learning frameworks which do not require writing any code (example: see and its questions that use the built-in neural network trainer). Models and processing pipelines may be described using other textual formats as data, which in turn are fed to a program. Is this programming? I feel that this is a bit subjective, and one can make arguments towards both ends.

What would be an MCVE here?

Let's suppose that a decision as made, and that we do want to keep it here. It's very important to understand that making an MCVE for a machine learning question is awfully hard. Minimalism is not very hard to achieve if familiar frameworks are used (such as Scikit-learn, Keras, TensorFlow, ...), but the variables which contribute to reproducibility are more than what most think of:

  • Some questions only provide the model and forget about the optimization process of that model. When training the model through stochastic gradient descent, this includes the kind of optimizer, the learning rates, and related hyper-parameters.
  • Even different frameworks or framework versions may yield different results, so that should be explicitly specified.
  • Since the training process often has a stochastic nature, it may rely on a random number generator. Without fixing all PRNG seeds, the specific issue presented by the asker might not always occur.
  • One part of an MCVE that may be hard to state here is what makes an expected outcome. Most other programming questions have a well known positive output, but the ideal solution of "making this classification 100% accurate" is unrealistic. In machine learning (or even data learning as some people call it), the quality of a model is only as good as the data we feed it. As already described above, this kind question may fall prey to the "no concrete problem statement" trap, leaving everyone disappointed. We should be able to solve that question which applied a soft-max non-linearity twice, but this is likely to be phrased in the question as an "exaggeratedly low accuracy", or "loss not decreasing", which could be anything until we look at the code.
  • Most critically, unless you are using public data sets, this might not be reproducible by someone else at all. Questions with the title "CNN predicts only '1's" is not uncommon, but if they need to give us their 1 GB data set of images that they should not be disclosing anyway due to signed data end user agreements, then we're close to a brick wall. The asker can only be advised to try changing the problem to another one (e.g. try the same architecture and processing pipeline with common public data sets such as MNIST) and see whether the issue persists.

I once provided a bit of guidance on how to curate these questions. A similar process can be laid out for asking a question: If the question effectively narrows down to a programming problem, you may then put this matter aside and use the remaining criteria on whether the question is suitable for this site. If it's purely about machine learning rather than programming, please visit another site. And finally, the points above should be kept in mind when creating an MCVE. Yes, it is hard, but how can we make the question useful to future visitors without it?

-4

Overfitting and underfitting are statistical concepts. They are only relevant to computer science or software engineering in that we sometimes implement statistical analysis in code. However, in this case, there is no code. Nothing's broken, so it's not Stack Overflow. And it's not a theoretical question about code that would fit on Software Engineering or Computer Science.

Statistics without code are on-topic on Data Science or Cross Validated. From the linked help topic, Data Science seems to embrace questions like "How do I avoid simultaneously overfitting and underfitting data?" It's unclear if they would also accept the recommendation part of the question, so it might need rewritten.

It's unclear to me if the question is specific enough to ask on Cross Validated. As is, it's rather vague. You'd have to ask their community.

You might also consider pointing the person to existing questions on other sites.

Compared to

In my personal opinion, the best response to this kind of question would be to migrate to a new, beginner's site, where people could receive more help and guidance to the correct site. Stack Overflow is probably the wrong place to try to figure out where to direct someone or correct site misconceptions. But under the current system, someone would have to take ownership of the concept, propose it on Area 51, and garner sufficient support to launch.

5
  • 6
    Stack Exchange sites "for newbies" rarely take off because they do not make good grounds for a long standing, high quality repository of knowledge.
    – E_net4
    Commented Jan 2, 2019 at 15:09
  • But that wouldn't be the goal of this site. Not a repository of knowledge (regular stacks have that). What the Stack Exchange Network is missing is a place where beginners can receive help understanding the system. That's fundamentally not a repository of knowledge where a single post can be canonical for all time. It's a constant repetition of the same general information.
    – mdfst13
    Commented Jan 2, 2019 at 18:10
  • 2
    We already have a site where people can get guidance to the correct site: Meta Stack Exchange, or the current site. Commented Jan 2, 2019 at 19:24
  • Yes, I understand, and it's a legitimate proposal (even Jeff Atwood mentioned it recently as a possible solution to the "welcoming problem"). But since it deviates from the greater goal of making a useful resource to visitors, our community (and the company) are currently not looking forward to having that site. Multiple concerns arise from tiered platforms, after all.
    – E_net4
    Commented Jan 2, 2019 at 19:55
  • But people on the current site don't know where the post should go, just that it should go somewhere else. That's why we get questions like this, by third parties asking how what to recommend to the actual poster. Also, would it be appropriate to migrate the post to either M.SO or M.SE? If not, we don't already have a site like I describe, where any beginner's off-topic question can be migrated. And this isn't a tier. Questions would only be migrated when they would otherwise be closed. Questions might be migrated again, or reposted in better form on another site.
    – mdfst13
    Commented Jan 2, 2019 at 20:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .