61

I have seen a lot of questions on tagged with machine learning that are along the lines of "here is my ML training script and a link to a dataset. I am getting x% accuracy, and how do I improve it?". I am not sure how to answer these kind of questions.

The description for the machine learning tag says

NOTE: If you want to use this tag for a question not directly concerning implementation, then consider posting on Computer Science, Cross Validated, Data Science, or Artificial Intelligence instead. Otherwise you're probably off-topic.

However, I am not sure what exactly includes implementation. If someone is asking questions that are clearly conceptual in nature, voting to close as 'off-topic' it and leaving a comment asking to post it on some other Stack Exchange site is probably the way to go. However, I am not sure if questions asking ways to get performance improvement would be welcome there as well.

In my opinion, these should be flagged as "too-broad" as it's not possible to give a clear single answer without a lot of experimentation which just amounts to doing the work of the OP for them. On the other hand, if we try to answer in terms of "possible things to try", that will be inviting somewhat opinion based answers since it's not possible to know what will actually work and what won't for the given data.

Or should these be tagged as "off-topic".

What is the way to go in these type of situations?

Should these type of questions be allowed on Stack Overflow?

11
  • 24
    Heavily related question: meta.stackoverflow.com/questions/378431 In one of the answers: "Questions seeking advice on "How do I improve this model?" are usually open ended, and will often attract generic recommendations that involve much more concepts than those in the scope of the site. They should be closed as Too Broad until they are severely narrowed down to a concrete concern. Providing advice here is tricky unless you are confident in proposing migration."
    – E_net4
    Commented Dec 28, 2020 at 9:28
  • 5
    It's true that those questions are mostly unanswerable here. What might have confused you is the possibly subpar curation efforts around tags related to machine learning, which was becoming problematic some years ago.
    – E_net4
    Commented Dec 28, 2020 at 9:31
  • 5
    This probably depends very much on the specific question. "how do I improve it?" is usually poor because the goal is not clearly defined (there are tons of ways to get 1%, 2%, 3%, ... more) but it is still possible to ask such a question with enough detail and focus. For the kind of questions I assume you ask about, any of "Needs Details" (missing constraints for the solution) "Lacks Focus" (no clear direction for answers) or even just "Needs debugging details" (definitely not minimal and self-contained) can apply. Commented Dec 28, 2020 at 10:04
  • 3
    All the usual stuff applies. Good questions can be answered unequivocally by a domain expert. Very likely they will need more focus or more clarity or both, but if they are so well explained and focused that a domain expert in machine learning can answer them, why not. Commented Dec 28, 2020 at 15:00
  • 6
    Note that nearly always, questions about model performance need a dataset, and nearly always, this dataset can't be provided on Stack Overflow, so these questions lack a minimal, reproducible example. An exemption might be questions about models based on a well-known public dataset (e.g. MNIST digits/imagenet).
    – Erik A
    Commented Dec 28, 2020 at 15:09
  • 4
    The tag description clearly says "Implementation questions about machine learning algorithms". In plain English, this means writing an algorithm, not using it-tweaking it, which is perfectly on-topic for SO. Most of the questions (if not all) tagged with machine-learning seem to ignore this. The tag could be renamed to machine-learning-implementation or something more clear, to state the difference. Machine learning, in general, is a mathematical concept that has nothing to do with programming and people asking here are unlikely to be helped at all. Commented Dec 28, 2020 at 16:07
  • 7
    @Countour-Integral Machine learning is far from just a mathematical concept. It has deep roots in math, but saying that it in general "has nothing to do with programming" is hardly true. ML is inextricably tied to programming... unless you've found a way to do it with pen and paper. Commented Dec 28, 2020 at 20:06
  • 6
    @rocksNwaves The first neural nets were implemented before digital computers, as electronic circuits. And most of the basic concepts of ML are from before computing as well (e.g. Bayes died in 1761). ML has become practical due to programming, but is not inextricably tied to it. Commented Dec 29, 2020 at 2:30
  • 3
    @CrisLuengo Electronic circuits are/were early computers. And if you want to get specific with throwing the word "digital" around... I challenge you to implement an LSTM or GAN on an electronic circuit. You really don't have a point, I'm afraid. Commented Dec 29, 2020 at 14:15
  • 3
    Questions asking how to improve a model, whether or not there is some clear problem with the code, have basically zero future value because others with the same problem won't be able to find the question (possibly less than zero, in fact, since it makes it harder to find posts here and elsewhere taking a more general approach and discussing many things that could go wrong). I'd say that's enough of a reason to close them, but others would probably disagree. I don't have a problem with such questions in principle, I just don't think they align with SO/SE's goal.
    – NotThatGuy
    Commented Dec 29, 2020 at 23:46
  • Similar, sample question (was also in the triage queue): Which metric should we examine while making the model? Commented Jan 27, 2021 at 15:18

7 Answers 7

29

I'm a regular contributor to the and tags, where many users come and ask how they can improve their model's performance.

These questions are definitely allowed on this site. Active answerers tend to concentrate on certain tags, and therefore determine to a large extent which questions are closed or remain open for the tags they closely follow. Questions about performance improvement are usually tolerated and answered by those who monitor machine learning related tags. Users frequently come to this site due to an abnormally bad performance, which suggests that there is some kind of error in their code. Correct coding of machine learning obviously goes beyond having a model that doesn't crash.

As a regular contributor to the Keras (and Tensorflow) tag, I participate in voting on what remains open or gets closed in this tag. This is how I generally handle questions about performance improvement:

  1. Leave open if:

    • Low performance is caused by an identifiable, wrong use of an object (e.g., wrong loss/activation function), and can be solved by a correct use of other objects (example)
    • There is a clear answer that would make the model a "working" one
    • I can edit the title to a searchable format so users can find it by describing their approach and problem (e.g., accuracy doesn't improve with binary neural network with softmax activation)
  2. Close if:

    • Everything seems in order, no "error" was made, the performance is satisfactory, and there is no obvious way in which the model should be improved (example)
    • It's asking about improvement beyond a basic, running model
6
  • 6
    I've benefited from your answers more than once. I appreciate them, and I also appreciate your willingness to help within reason. It's folks like you who make SO worth coming back to after bad experiences. Commented Dec 29, 2020 at 20:44
  • 5
    I mostly agree with you, but I think what makes the decision hard is that it is sometimes difficult to see whether there is a clear answer. The reason why people ask for help is that they don't see a clear answer. For me, the threshold of posting in a machine learning topic on SO would be a really bad result compared to a baseline that should be overperformed in nearly all circumstances (linear regression), especially if I feel that it's going to be a coding error. Anything more advanced should go to other stacks. In classifiers, worse than chance-level accuracy does signal an error.
    – boomkin
    Commented Dec 29, 2020 at 22:59
  • 2
    Agree, and this is how I approach these questions in practice myself, too.
    – desertnaut
    Commented Dec 29, 2020 at 23:29
  • 1
    Thank you for the response. I think this is the general consensus on how to approach these questions and it makes sense. I have accepted this as the 'right' answer.
    – Ananda
    Commented Dec 30, 2020 at 6:51
  • Judging whether to close questions in this way means no-one will be able to know ahead of time whether the question they want to ask would be appropriate here (generally: closed = not appropriate), no-one would be able to know whether a question is appropriate without digging through the code in detail, and maybe not even then. The reason why a question is closed should always be clear and, apart from the occasional duplicate, users should be able to avoid having their questions closed for the same reason in future, which isn't the case here at all. It just doesn't make sense as a broad policy
    – NotThatGuy
    Commented Dec 31, 2020 at 1:47
  • @NotThatGuy you can generally know by the tone of the question whether the person is trying to improve normal performance, or trying to determine why their model isn't performing up to standard. While some users may not know ahead of time if their question is appropriate, it can be edited in a way that will make it so, and future users will be able to find answers without having to dig through the code. For instance, you can edit the title, e.g. generic "My model isn't learning" to "Model with SoftMax activation and binary crossentropy isn't learning". The latter will be useful to future users Commented Jan 14, 2021 at 15:43
13

This question confuses me because you haven't made a case for how ML performance optimization questions are any different from any other question on Stack Overflow. As such, the standard guidance of "answer what is answerable, downvote and/or vote to close what is not" applies.

If someone provides an algorithm and a dataset, with an explanation of what they've tried WRT performance so far and why it isn't sufficient, that's perfectly answerable. If they're just throwing a dump of code and essentially expecting Stack Overflow to optimize it for them, no, sorry, that's not sufficient information and that question should be closed as "needs details or clarity".

1
  • 3
    My point was that even if someone is proving all the algo and data details, performance improvement in ML is something for which is very difficult to provide a specific answer for. Along the lines of "do x and your performance will improve". You would have to try out all the potential approaches and see if they make sense for the data. At that point, the person answering the question is essentially training the data for the other OP. However, the consensus seems to be to handle these in a case-by-case basis as explained in the accepted answer.
    – Ananda
    Commented Dec 30, 2020 at 7:09
11

Agreed this is a widespread phenomenon in keras and tensorflow tags; lost count of "Keras bad validation loss" questions. Some contain legitimate bugs, others are purely advice-based and are not at all Stack Overflow material, but more Data Science, Statistics or AI. What gets done about these questions, however, is matter of convincing their experienced watchers, for these two tags one of whom is myself.

Personally I either flag for migration to aforementioned networks, or leave the question be. Even if not a fit for SO, such questions often produce much value for the community that may otherwise have been lost in the less-popular networks. I, for one, barely ever browse those three networks, so answers from those like myself get automatically thrown out if not posting on SO.

Not all such questions are worthy of keeping, however; some are very basic and specific to the user, who should likely learn more theory than call it a problem. My problem is: I actually can't flag to migration to Data Science or AI networks, only to Stats. What's the deal there? Many things aren't fitting for Stats but are for DS or AI, so one way to get me cleaning more is by adding said options.


Edit: if true that flagging migration to X requires some rep in X, I suggest easing the process by establishing credence by tags. So if I have 100+ pts in machine-learning, I can recommend Data Science and Artificial Intelligence.

5
  • 5
    "My problem is: I actually can't flag to migration to Data Science or AI networks, only to Stats. What's the deal there?" SO sadly doesn't know that migrations are hardly recommended and only limited to users that are at least established on both sides.
    – Braiam
    Commented Dec 28, 2020 at 20:10
  • 7
    Most questions shouldn't be migrated. You're better off telling the OP to post their question elsewhere, rather than having it migrated and pissing off the people on the other site because it's of-topic there as well. Commented Dec 29, 2020 at 2:33
  • 3
    As someone frequenting related tags (Python mostly) I can't say it is clear cut that such questions "produce much value for the community". I have lost track of people justifying bad Q&A in non-ML topics because of the precedence set by ML questions. Commented Dec 29, 2020 at 9:59
  • @Braiam I do not believe that is true at all. I have always been getting these possibilities: meta.stackoverflow.com superuser.com tex.stackexchange.com dba.stackexchange.com stats.stackexchange.com I am not established on most of them and I am established at some I would like to see there, but I don't (math, scicomp). Commented Nov 11, 2023 at 8:48
  • 1
    @VladimirFГероямслава what do you not believe is true? Most suggested migrations are not recommended to be migrated due scope/quality reasons. The likely most successful migrations that can be suggested would be from users that actively participate on both sides. I'm not talking about the list, I'm talking about the like-hood that questions migrated would be succesful.
    – Braiam
    Commented Nov 21, 2023 at 9:27
6

There are three types of "off-topic" questions asked about ML which I would categorize as

    1. The model's predictions are really low - the algorithm is not performing well

This describes the kind of question. These questions, even though there is a lot of code involved, are not programming related and it is very unlikely that their authors will get helped.

Why is this? Because the bug is MATH RELATED, they are using an insufficient algorithm for their problem-dataset. The code works correctly and MINIMIZES the error, just not on the desired level. It's like using linear regression to do image recognition. The algorithm will work fine, but will not bring the desired results.

Solution

Vote to move them to the Stats or AI Networks, after a bit of filtering (e.g the autor could translate the tensorflow or keras code in plain statistics, so instead of just throwing the code, they could explain the structure of their network)

    1. Terminology - Design of neural networks

here or here or there or everywhere. These questions clearly violate the "General questions about machine learning should be posted to their specific communities" rule.

Solution

Same as the previous, they are not programming related

    1. Library specific errors-questions

Questions such as this or this or even this and this.The authors are facing a specific problem with their library unrelated to ML, but are still using the Machine Learning tag, such as Exceptions like

TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you are trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
ValueError: Unknown layer: Functional

or simply things such as

How to find the len() of a tf.Dataset

Solution

Simply remove the Machine Learning tag, questions like this, concern custom datatypes-classes of the libraries that they are using and are not directy on-topic. It's like asking what is an np.array and using the tag math. Yes the library may be related to math but the question is not.

But what is on-topic?

Those that clearly follow what the description clearly says, obviously

Implementation questions about machine learning algorithms.

A simple example is this.

The OP is asking a question about their implementation of a machine learning algorithm. There is no math bugs and even though the specific one, may be library specific, it is still a machine learning question that is DIRECTLY PROGRAMMING related.

5

I also regularly (try to) answer questions in the usual tags like , , and .

The problem with this kind of questions is that I believe in most cases they are opinion based. Improving the performance of a machine learning model, while in most cases requires coding something, is not a task that only requires programming skills, but actual scientific research. If a question puts a dataset, some code, and some expectation on performance improvement, it is not always obvious how this can be achieved. It might require to run experiments, and basically throw things at the wall and see what sticks.

And this is even if the question is well written and has all the seemingly important details. As far as I know, questions in Stack Overflow (and other sites) should be answered in an objective way based on facts. If the question cannot be answered in this way, then its opinion based in my mind.

My professional opinion is that there are many non-programming issues that affect the performance of a machine learning model, particularly data, if you do not have enough data for the task, it is low quality, or not correctly captured, then your model will perform bad and this is not a programming problem.

So I use the following decision making process:

  1. Questions might be on-topic if the problem is related to a programming error or easily solvable with a particular library or technique, but this should not be a guess, and should be well explained with facts. Note that many common coding errors in ML have been asked and answered before, so this could be a duplicate question.
  2. If the question require you to give your opinion on how performance might be improved, then it is opinion based and should be closed.
  3. Some people ask for recommendations on models, datasets, etc. These are obviously off-topic as we do not make recommendations.
  4. Some people actually want to have a discussion on how performance might be improved, in this case, maybe it is better to suggest migration to one of the other SE sites like data science, AI, or stats.

In some cases I might leave a comment telling that this is not easy to answer and that the answer is opinion based, or to redirect to another site, but users do not always like the truth, or I might not have the time for that.

In any case many of these questions will not be answered just because literally nobody knows the answer. Question askers should be aware of this. Even with a PhD in ML and many years of experience, there are many things we still do not know.

4

I read comments suggesting that questions are often "unanswerable" or that "machine learning, in general, has nothing to do with programming". While the first might be true under the current paradigm, the second is a little bit of SO elitism in my opinion.

The obvious solution is to migrate questions to the "right" network, but I take issue with that for two reasons.

Silent Failure

The reality of the situation is that ML is practiced by programmers. Programmers are sitting in front of an IDE or notebook and the code that they have written is not having the expected results. This might mean a runtime error that points them to an exact line of code, or it may mean some type of silent failure.

Silent failure is hard to address, but that doesn't mean it's any less of coding failure than a TypeError. The conceptual nature of a TypeError is simply easier to understand and therefore explain and address.

Good Questions Unanswered

  1. If it's possible, the Stats network is even more elitist and un-friendly than SO. Questions asked there may be answered, but it is equally likely they will be ignored or derided by "intellectually advanced" community members.

  2. The AI and other such communities are significantly underdeveloped. Questions asked on those networks don't get views because the people who could answer them are here on SO happily dolling out answers. (I know this because I've committed the mortal sin of cross posting and watched my answers blow like tumble-weeds through the desert on one network while they get answered on the other).

A proposed solution:

Don't worry about it. If an ML question is bad, it is probably bad for the usual reasons. It's not focused or it doesn't include a minimal working example or whatever. Just leave a comment on how it might be improved and vote to close it if you feel like it. However, if you take issue with the fact that the failure is silent you don't like questions that are hard to answer it might just be an ego problem.

3
  • 7
    More likely that Stats is small enough to get away with labeling trash as trash.
    – Ian Kemp
    Commented Dec 28, 2020 at 21:18
  • 2
    @IanKemp Calling someone's question trash, ill posed or not, is a far cry from helpful or in line with community guidelines that you seem to hold in such high esteem. You should consider rephrasing your comment. Commented Dec 29, 2020 at 20:41
  • 7
    There's nothing wrong with calling a spade a spade. Low-quality questions can and should be designated as such. Questions cannot have their feelings hurt, and getting the correct label is quite helpful in figuring out the appropriate response. Stack Exchange communities focus on creating high-quality content, optimizing for pearls, not sand. Smaller sites can be more effective and thorough in their curation, since they aren't suffering from massive scale problems like Stack Overflow.
    – Cody Gray Mod
    Commented Dec 30, 2020 at 5:08
-14

Questions related to improving machine learning performance are absolutely on-topic. We don't want to contribute to a culture where ML development is treated as a knob-turning exercise. We want a culture where understanding the reason why a ML model implementation is poor should be a top priority. This is no different from, say, database performance tuning. My database query is running very slowly. What can I do? The wrong answer is to kick the question to another StackExchange board.

EDIT: To the people down-voting my answer. It is clear that you are wrong on this matter. The sooner you accept that you are wrong, the sooner our community can heal.

5
  • 6
    Can you please explain a bit more how the "knob turning" issue makes questions suitable for SO? From my POV, many such questions are actually "turn the knobs for me" so they aren't even about turning the knobs themselves, least of all understanding which knobs to turn. Even if they were, explaining all the potential ML knobs and meta-knobs (e.g. models,...) seems way out of scope for a Q&A format. Commented Dec 29, 2020 at 10:10
  • 3
    Wait, why is another Stack Exchange site (they're not discussion boards) the wrong answer? What if there's a site specifically tailored to improving working code (there already is), or more specifically improving working ML code (there might already be)?
    – TylerH
    Commented Dec 29, 2020 at 17:01
  • 5
    "We don't want to contribute to a culture where ML development is treated as a knob-turning exercise" sure, but is irrelevant to topicality. We would also like to end world-hunger, doesn't make it a good question on SO. Tuning ML models (especially deep learning) is significantly different from database performance tuning. To oversimplify, we don't understand at the microlevel what's going on in ML, so we can't give completely subjective answers.
    – Passer By
    Commented Dec 29, 2020 at 17:07
  • 1
    It is not the first time you bring forward the unfounded and puzzling allegation that closing ML questions as off-topic for Stack Overflow is equivalent to encouraging numb knob-turning and an obstacle to a deeper understanding; it is even more puzzling given the fact that you are an experienced user in Stats SE, too.
    – desertnaut
    Commented Dec 30, 2020 at 1:06
  • Database tuning is a very different field, it is less data driven and more algorithm/technique driven, that is what makes it more on-topic for SO than machine learning modeling. A model that works in one dataset might completely fail in another.
    – Dr. Snoopy
    Commented Dec 31, 2020 at 4:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .