40

We see many questions each week, often multiple questions a day, asking to use a library / tool / language / feature in a non-recommended way. According to this Meta SE answer, "Don't Do It" with an explanation is a valid answer.

Specific case

I'm focusing on to demonstrate the problem, but the general issue may be applicable to many tags. There's a current trend "because it's possible" to put lists in columns / series. Here are some recent questions:

  1. Python Pandas: multiple aggregations -> list of values
  2. Assigning to pandas DataFrame column behaves differently depending on other columns
  3. Pandas- How to check if list of strings in DF row contains any of strings in series in another DF?
  4. Summing multiple lists stored in dataframe
  5. Adding lists stored in dataframe

Note: These questions are not duplicates, but have a bad premise / starting point / workaround. So canonical posts will not help here.

There are hundreds, possibly thousands, of these questions scattered across SO. In my opinion, each and every one of these can justifiably be answered with:

Don't do this. Pandas was never designed to hold lists in series / columns. You can concoct expensive workarounds, but these are not recommended.

The main reason holding lists in series is not recommended is you lose the vectorised functionality which goes with using NumPy arrays held in contiguous memory blocks. Your series will be of object dtype, which represents a sequence of pointers, much like list. You will lose benefits in terms of memory and performance, as well as access to optimized Pandas methods.

See also What are the advantages of NumPy over regular Python lists? The arguments in favour of Pandas are the same as for NumPy.

The problem

I fear plastering this on every such question won't go down well, even though it's exactly what I want to say, and nothing more, i.e. I purposely want to avoid getting into the details of the question because it adds noise to the answer.

Furthermore, such duplicated answers will come up as "duplicated answer" in moderator tools and I expect repercussions.

So what can I do?

A comment is ephemeral. Even with a link, it does not do justice to the extent of the problem / issue. The generic "Don't Do This" answer is likely to cause trouble, though I believe professional or enthusiast programmers will find it helpful.

Before the alarm bell sounds, I'm not looking for rep out of this, more than happy to community-wiki such answers.

18
  • 2
    per my reading canonical posts would address this issue just fine. If I missed something, consider editint your question to help readers see why it is not so
    – gnat
    Sep 6, 2018 at 11:52
  • @gnat, Sure, will edit my question. The actual questions themselves are not duplicates, so I'm not sure canonicals are appropriate. They just have a very bad starting point / premise / workaround.
    – jpp
    Sep 6, 2018 at 11:53
  • I see. Does that mean that we are talking about cases when it's unclear why asker wants to do things in a way that contradicts the proper practice?
    – gnat
    Sep 6, 2018 at 12:02
  • 2
    @gnat, Actually usually the question is very clear: Here's what I have, here's what I want. The bit they have though is bad practice and not recommended. Often, as here, it stems from an XY problem.
    – jpp
    Sep 6, 2018 at 12:09
  • 1
    @user2285236, To labour the point, there are documented and undocumented bugs / deficiencies relating to holding lists in series. It should never be part of a production process.
    – jpp
    Sep 6, 2018 at 12:33
  • 1
    I am talking about communication between that db and pandas. If I have a db designed that way and I am processing a small chunk in pandas the best way to handle this is to have a columns whose entries are lists.
    – ayhan
    Sep 6, 2018 at 12:36
  • 4
    You are suggesting that I should burn down my house and build it from scratch because I had a small problem in the kitchen.
    – ayhan
    Sep 6, 2018 at 12:44
  • 5
    From the accepted answer on the linked post: "I would consider also to answer the actual question, too. As in 'Don't do it because of A, B and C. But if you decide to do it anyway, I would follow this approach:...'" - this seems like the best approach. If there's a single question or offsite resource dedicated to explaining why it's a bad idea, you can also link to that either in your answer or in a comment. Sep 6, 2018 at 12:57
  • 2
    @Dukeling, The second answer has However, an answer that tells the poster how to do something stupid, just to strictly answer the poster's question, is worse than useless. I put this in the same category. But, in this case, we needn't worry. There's no shortage of [pandas] followers willing to answer these questions, I can simply add If you wish to go ahead, use @abc's answer to make it "unique".
    – jpp
    Sep 6, 2018 at 13:40
  • 1
    we already have such similar question in CSS ... where people want to do something not possible and there is only workarounds : stackoverflow.com/questions/42176419/… so you can probably do the same Sep 6, 2018 at 13:44
  • 1
    @jpp If someone's already made up their mind about doing something stupid, you can't do much about that. The most you can do is add a disclaimer and present an alternative for those who haven't quite made up their mind about doing it. I'd say "Don't do it" is mostly not really an answer by itself. Sep 6, 2018 at 13:56
  • 8
    "Don't do it because..." answers are extremely useful and valuable posts, as they highlight good practice (and bad practice). If the OP needs calibration, so do others that will read the question in the future. SO should provide it, instead of seemingly condoning hacky turnarounds. Sep 7, 2018 at 5:02
  • 1
    @ReblochonMasque, Completely agree, which is why I think this Meta is important. I envisage 90% of each such answer to consist of a shared disclaimer. Mods may see this as copying (we'll see..), which is why I'm looking for a consensus. Initially I'm planning to not community-wiki these answers, but if it's an issue (or recommended by community) I will wiki them.
    – jpp
    Sep 7, 2018 at 9:50
  • 1
    Yes, to me, these "don't do it" answers are invaluable, and some of the value is in consistently repeating it, and he reasons for it; however, as you wisely underline @jpp, the devil is in the details; I trust a good implementation will be found. Sep 7, 2018 at 9:59
  • 1
    I have to strongly disagree with the assertion that lists in pandas cells are never appropriate; it's exactly the same situation as storing blobs in a database (or blobs in pandas, for that matter). Sure, there's often a more machine-efficient solution, but potentially at the cost of some other consideration (memory, atomicity, etc.). For this Meta question, I fear that creating a hammer like this just encourages people to swing it whenever they see what they think is a matching nail, without trying to understand the actual need of the question. Mar 30, 2021 at 16:36

3 Answers 3

15

This answer is based on @Dukeling's helpful comment.

Only post such an answer if you also differentiate it by addressing the question. It can be differentiated in a couple of ways:

  1. Answering the question directly, despite its perceived flaws.
  2. Linking to another answer which addresses the question directly.

In either case, the disclaimer may be placed before or after the question-specific content. This way you are addressing the question directly, even if the bulk of the answer happens to be a disclaimer. Conversion to community-wiki is then not necessary.

13

It might not be advisable to close all those as duplicate to a canonical "don't do it" post, but it is a very good thing to have such a post that you can link to in comments.

I would absolutely recommend to establish such a canonical post among the community, either by picking a good one already existing on the site, or by writing a new one (possibly as community wiki).

The bottom line is that SO should care more about technical quality and correctness than "does it answer the question" meta policies. Linking to a canonical dupe instead of dupe-hammer closing a question is less intrusive, especially if the questions at a glance seem to have little in common.

That being said, if the main issue with a question comes from a bad practice and the best answer is "don't do it", then don't hesitate to close the question as a dupe to a canonical post.

4

Comments are for when OP should know something that is not actually answering the question. If someone wants to ask an XY question, that's fine; the fact that OP asks Y when really needing to know about X, does not negate that Y could be a valid question that could help others.

If you want to say that the real question should be X and it's a common duplicate, use a comment. You can have a form comment ready to go for it; and you should ideally have already picked out a canonical (if there isn't one, that's a separate challenge).

For example, I have this in my user profile ready for copy and paste:

For questions and answers that make inappropriate use of eval and/or exec in Python:

**[Do not ever use `eval` (or `exec`) on data that could possibly come from outside the program in any form. It is a critical security risk. You allow the author of the data to run arbitrary code on your computer](/q/1832940). It [cannot easily be sandboxed, and proper sandboxing is harder than using a proper tool for the job.](/q/3068139)**

It's on a single code-formatted line to preserve Markdown (and because line-wrapping is not needed; I just select the entire single line); the entire comment is boldfaced because it's a critical security warning. It links to two relevant canonicals about the topic. It does not try to represent that either question answers what was asked, because that generally isn't true. There are many reasons eval, exec etc. exist in Python and many good use cases for them, but most ordinary users are misguided when trying to use them. (This does not only apply to cases where OP is explicitly asking about how to make exec or eval work when it shouldn't be used at all; sometimes it's just a response to "I tried exec/eval but it didn't solve the problem / I want a more elegant way"; and sometimes it's a comment on bad answers that I downvoted for needlessly proposing those tools where they aren't appropriate.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .