73

I think there is a problem with the question about unit tests and the conclusions drawn:

enter image description here

...they are overwhelmingly in favor of unit testing...

The problem is that you don't catch people in the 62% that do use unit tests but don't like it. You only catch the opinion of the 38% that don't use unit tests. Theoretically, 100% of the people using unit tests could hate it - even the 20% that could feel forced to do it even though it's not a part of their official process.

I have nothing against unit testing, but I don't think you can state what you do about the data.

18
  • 3
    What about the "How are Job Satisfaction and Unit Tests Related" part? Commented Apr 9, 2019 at 15:03
  • 3
    @NicolBolas Marginal at most I'd say.
    – Alex
    Commented Apr 9, 2019 at 15:07
  • 2
    Not liking to test what code you wrote is like not taste-testing what food you cooked to be sure you didn't mistake the salt and sugar again. I cannot fathom a demographic of people who don't like to test their code. If they do identify themselves, I know whose pot luck food to avoid for the future, though.
    – Makoto
    Commented Apr 9, 2019 at 15:24
  • 26
    @Makoto That's not relevant, the data isn't there to support what they claim. And there are more ways to test than by unit tests. Automated system tests for example.
    – Alex
    Commented Apr 9, 2019 at 15:32
  • 7
    I would conjecture that "unit" test is more of a colloquialism to encapsulate all forms of testing - be that unit, integration, end-to-end, manual, automated, etc - but I see your point.
    – Makoto
    Commented Apr 9, 2019 at 15:33
  • 38
    @Makoto Then we have even more issues with the question :)
    – Alex
    Commented Apr 9, 2019 at 15:34
  • 10
    @Alex: "Problem is that you don't catch people in the 62% that do use unit tests but don't like it." Incorrect. Well, not entirely correct. 20.5% of the people are clearly choosing to use unit tests. They are doing it because they prefer it, not because they are being asked to. So your statement should just be about the 41.8% who are forced to do it. Commented Apr 9, 2019 at 17:36
  • 39
    The entirety of the survey results is full of unsupported assumptions. It's clear they care more about pushing their bias than actually talking about the data. Commented Apr 9, 2019 at 17:41
  • 3
    @NicolBolas Then it's a subjective question of wording and interpretation of that alternative. I'm sure it is like you say, that many do it because they want to , but it could as well be that the company does not enforce it, while a majority of the developers do it, practically forcing some to do it just to fit in event though it's not stated in some "process".
    – Alex
    Commented Apr 9, 2019 at 17:45
  • 15
    "It's part of our process" bears no indication of whether a developer agrees with/prefers/favors that process. It just means that the "process" said to do something and they did it. My process says I have to put javadoc comments on all my getters and setters; do I favor this practice? Heck no. Commented Apr 9, 2019 at 19:36
  • 34
    @rlee827 "About 65% of professional developers on Stack Overflow contribute to open source projects once a year or more." This number includes the values for "Less than once a year" most likely to make it look like more people contribute to open source regularly than the actual number "once a year or more", around 35.5%. Why say something blatantly false when the numbers are, literally, right there? If not pushing an agenda, either they are including numbers they aren't representing, or they flipped around their own findings and presented them completely wrong. Commented Apr 9, 2019 at 19:55
  • 2
    "Company" can also be misleading - my team makes extensive use of them (and I believe that's how I answered it), while most of the rest of the company does not.
    – Izkata
    Commented Apr 10, 2019 at 14:32
  • 3
    @EthanTheBrave if you scroll down a couple of questions to Years Since Learning to Code you can see the results are separated by "All Respondents" and "Professional Developers". Seeing as they say "65% of professional developers" contribute to open source, it appears they're referencing data they're not explicitly showing. Granted, the data should be shown, but I don't think they're pushing an agenda. Commented Apr 10, 2019 at 14:40
  • 3
    @crazyloonybin Fair enough - there's an error in there somewhere, and maybe they aren't misrepresenting things intentionally in that instance. This is one of many assumptions however. I'll abstain from getting into all of them as they will, as history has shown me here, paint a target on my head for not agreeing to the hivemind. Commented Apr 10, 2019 at 14:53
  • 2
    @NicolBolas 20.5% of the people are clearly choosing to use unit tests it's also incorrect. 20.5% of the people are saying that some devs do unit tests on their own, it doesn't say that they do it
    – Lamak
    Commented Apr 11, 2019 at 15:06

1 Answer 1

9

I don't think you can state what you do about the data.

As the saying goes: there are little lies, there are big lies, and there are statistics.

You have identified one issue, but there are quite a few if you look closely. Let's be real, though, the SO Survey is not really an attempt to be scientifically rigorous. It's more of a fun little thing that's meant to be mildly interesting, and they're not submitting this for peer review or trying to make grandiose claims based on their minor conclusions.

Nonetheless, there are some things I didn't find tenable.

There are claims in the summary which are obviously unsupported or flat-out contradicted by the data (not counting the one you mentioned):

  • About 65% of professional developers on Stack Overflow contribute to open source projects once a year or more. (In fact, 64.4% contribute to open source less than once a year or never.)
  • When asked what they primarily believe about blockchain technology, respondents on our survey are largely optimistic about its broad usefulness. (In fact, only 29.2% of respondents said anything about broad usefulness, although 67.6% thought it was useful somehow.)
  • When it comes to their own perceptions of engineering management as a career, half of respondents don't think they necessarily need to move to people management to continue to grow their salary. (actually 70.9%, although there is some ground for an argument about how to interpret "Not sure")

There are claims which may be untrue depending on additional information:

  • Many developers work on code outside of work. About 80% of our respondents say that they code as a hobby. (Could be misleading - how many of those 80% don't code for work? It's 4%, so here it's fine.)
  • Over half of the respondents on our survey have taken a new job within the past two years; frequent job changes are the norm for software developers. (No mention is made of attempting to filter out internships or people who entered the field, despite the fact that 20% of respondents have been in the field fewer than 5 years.)

There are claims which are blown out of proportion or could mislead:

  • "As we've found in previous years, developers tend to be more satisfied with their career than with their current job" (The charts are on different scales, which makes direct visual comparison prone to exaggerate the difference that is highlighted. Just look at the middle line and switch back-and-forth a bit. Then look at the numbers.)
  • "Developers who belong to gender minorities in tech rank the office environment and company culture as their highest concern when assessing a new job" (52% vs 48% is less than a 10% difference. In contrast, there are huge (200%) disparities in the importance placed on diversity)
  • Greatest Challenges to Productivity section. (Once again, the differences highlighted are minor. I get that they have to say something, but there's not really much to say about the gender comparisons here, but they do anyway.)

There could be more, but I don't have time to keep reading it so closely.

2
  • 3
    "Let's be real, though, the SO Survey is not really an attempt to be scientifically rigorous. It's more of a fun little thing that's meant to be mildly interesting, and they're not submitting this for peer review or trying to make grandiose claims based on their minor conclusions." Though correct about what it represents, that is not how SO presents it or treats it. They're using it as an argument that SO and the broader programming community have diversity problems and as justification for their diversity policies. They also brand it with concepts like "fastest growing programming language."
    – jpmc26
    Commented Apr 12, 2019 at 5:22
  • 4
    I agree with @jpmc26 about your opening lines - We can't just slap a "Oh it's all for fun" label on it when it's both done poorly and being used as a decision-making tool. Commented Apr 12, 2019 at 15:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .