76

From the methodology section itself, the survey included "101,592 software developers from 183 countries". Of these, only 67,441 people completed the entire survey. Some estimates put the number of software developers in the world at over 18 million. The number of people surveyed is just a small, small fraction of the number of global software developers.

Also consider that the users who took the survey were all self-selected and tended to be users of Stack Overflow, I'm not convinced that you can take any useful data out, if you want to learn about the broader software development community. It's probably useful to people who want to learn about the Stack Overflow community, however. It's very unlikely that the results are truly representative of the broader software development community because of the lack of rigorous sampling applied.

My biggest concern is that various outlets are picking up this annual survey and presenting it as a representation of the state of the software development community. This is absolutely not true, and I don't see any clear warnings on the survey results page itself about how to properly interpret the results. I don't want to say that this survey is bad, because it does offer interesting insights and can launch some deeper thinking and investigation. But from what I can see, the results are often being used or presented improperly by third parties.

I don't think you can stop everyone from cherry-picking results or twisting data around, and I don't think it's even fair to ask that. However, I do think that there should be more information at the beginning of the survey results page that talks about the limitations of the survey.


To be very clear and have a concrete proposal, I'm asking for some minor changes to be made to the presentation of the results. One way to do this would be to make minor changes to the Overview section:

Each year, we ask the Stack Overflow community about everything from their favorite technologies to their job preferences. This year marks the eighth year we’ve published our Annual Developer Survey results—with the largest number of respondents yet. Over 100,000 developers opted to take the 30-minute survey this past January.

{{ With some data analysis on visitor demographics vs survey results, perhaps say how representative this is of the whole SO community. If there are any recent enough other surveys, talk about how representative this survey is of the global population. Be clear: Does this survey accurately represent the SO user base? How well does it represent the global population of software developers? Be sure to cite the estimates 18-23 million global software developers (or more recent surveys, if any are available) as well as the number of active Stack Overflow users during the survey timeframe. }}

This year, we covered a few new topics ranging from artificial intelligence to ethics in coding. Here are a few of the top takeaways from this year’s results:

  • DevOps and machine learning are important trends in the software industry today. Languages and frameworks associated with these kinds of works are on the rise, and developers working in these areas command the highest salaries.

  • Only tiny fractions of developers say that they would write unethical code or that they have no obligation to consider the ethical implications of code, but beyond that, respondents see a lot of ethical gray. Developers are not sure how they would report ethical problems, and have differing ideas about who ultimately is responsible for unethical code.

  • Developers are overall optimistic about the possibilities that artificial intelligence offers, but are not in agreement about what the dangers of AI are.

  • Python has risen in the ranks of programming languages on our survey, surpassing C# in popularity this year, much like it surpassed PHP last year.

  • When assessing a prospective job, different kinds of developers apply different sets of priorities. Women say their highest priorities are company culture and opportunities for professional development, while men say their highest priorities are compensation and working with specific technologies.

Want to dive into the results yourself? In a few weeks, we’ll make the anonymized results of the survey available for download under the Open Database License (ODbL). We look forward to seeing what you find!

28
  • 33
    I'm not convinced that you can take any useful data out. - It's probably useful if you're thinking about advertising and/or putting job posts on Stack Overflow.
    – BSMP
    Mar 16, 2018 at 17:30
  • 4
    @BSMP True. I suppose "useful data about the broader software development community" is better. It's probably a decent picture of the Stack Overflow community. Mar 16, 2018 at 17:37
  • 32
    Whether or not the survey is representative of the development community depends on whether or not SO itself is representative of the development community. Maybe that's a question we should be asking.
    – ArtOfCode
    Mar 16, 2018 at 18:09
  • 11
    @ArtOfCode Which it isn't, since certain demographic groups are more likely to show off online than others.
    – user6655984
    Mar 16, 2018 at 18:50
  • 2
    biggest concern is that various outlets are picking up this annual survey and presenting it as a representation of the state of the software development community it's an opinion, do you have data to support it?
    – pogibas
    Mar 18, 2018 at 9:03
  • 13
    @PoGibas Not sure if you're expecting Thomas Owens to have collected massive survey data about how the SO survey is being used... But as an anecdote, OP's description is pretty much how the survey is being used in water-cooler conversations at my workplace. So I'd say OP's concern is a valid one. Mar 18, 2018 at 9:22
  • 4
    That reminds me that there are 100 devs in my open-space, 1 % of them is on Stack Overflow. I let you guess who is running all the time to help colleagues. Note: it's some kind of cultural omerta: none of them is willing to participate to SO.
    – Cœur
    Mar 18, 2018 at 9:37
  • 5
    @Chris_Rands This isn't about transparency. Of course SO can't control how other people use the results, but they can make it more clear that the results are not representative of the entire development community and cannot be used to draw conclusions on a global scale. I'm not even sure that the results are representative of the Stack Overflow community, but maybe there's enough visitor demographic data to compare results and determine this. There's nothing that makes it clear up-front to visitors, and that's unacceptable, especially with the people who are using the results improperly. Mar 18, 2018 at 10:29
  • 2
    "The number of people surveyed is just a small, small fraction of the number of global software developers" I wouldn't worry about that. Unless you start drilling down to seriously under-represented demographics that's a plenty big enough sample! If you have a truly random/representative sample then it rarely makes statistical sense to go above a sample size of ~1000, however big the population. "...the users who took the survey were all self-selected..." this is a much more valid concern. You could have got 100% of SO users to complete the full survey and you'd still have this concern.
    – Adam
    Mar 18, 2018 at 11:43
  • 2
    I'm not quite sure what you mean.Isn't it universal and reasonable that only people who want to be surveyed respond to an online survey? Or that a survey in a site consists mostly of the site's users?
    – Kobi
    Mar 18, 2018 at 12:08
  • 6
    @Kobi One would think so. But results are being presented as representative of the global development community by third-parties. Visiting the results page doesn't make it clear who the sampled population is. Consider the Overview: "we ask the developer community" (no - the Stack Overflow community was asked) and "we've published our Annual Developer Survey" (no - it's a survey of primarily Stack Overflow users, with some outside participation) and "over 100,000 developers" (that's maybe 0.6% of all devs worldwide and 1.1% of registered SO users and not necessarily a representative sample) Mar 18, 2018 at 12:34
  • 5
    @ThomasOwens Without specific examples of misuse of the survey results, I don't really understand what you're unhappy about Mar 18, 2018 at 14:28
  • 9
    @Chris_Rands Are you familiar with survey methodology? You can't invite a small subset of the developer community as a whole and expect to get meaningful results. You need to selectively target the population and ensure that the right people are responding. Even if you just says that the survey results were supposed to be representative of the Stack Overflow community only, you need to make sure that the respondents are actually representative of the community. There's no evidence that the survey respondents truly represent the SO community. Mar 18, 2018 at 21:07
  • 10
    @FélixGagnon-Grenier No, a disclaimer on every survey isn't necessary. Some surveys define the population that they are attempting to sample. However, this survey is promoted as a "developer survey" and many are taking it to be representative of the global population of software developers. That is incorrect - it needs to be clear to readers that, regardless of the name and the text, this survey is not representative of the global software development community, nor is it likely to be representative of the SO user community. This is more about ethical (technical) communication than statistics. Mar 18, 2018 at 21:55
  • 8
    @AlexQuilliam So you're saying that, in order to talk about communication ethics, I need a specific example of someone misusing the data? That's silly. But fortunately, Jeremy found one good example. There's no way that Stack Overflow can prevent that story from being written the way it is, but Stack Overflow can ensure that people who click through to the full results can read, in plain language, how to interpret the results. Mar 19, 2018 at 15:30

3 Answers 3

12

Thanks for this thoughtful feedback! My belief, as someone who worked on this survey, is that there is a lot that is useful to be learned from it, both for the Stack Overflow community and the tech industry as a whole, from underrepresentation of certain groups to how technologies are used together.

However, it is so important to understand a survey sample's biases to interpret it correctly; from this feedback and others, we saw that we had improvements to make. We have updated the main Insights site based on this; the "Overview" section is still the same but discussion is updated throughout to be more clear about what the sample is.

13
  • 3
    Do you have some of the changes to highlight what changed? Just visiting the Insights subdomain and a few pages, I don't see anything that addresses the concerns here. Is there also a particular reason why Overview wasn't updated? I suspect that most people hitting the survey results page would hit that. A 3ish sentence summary of the methodology and limitations really should go there so people can see it. Mar 19, 2018 at 18:54
  • Unrelated question: I took the "dev" version of the survey before the official version was put up. Was my response counted?
    – gparyani
    Mar 19, 2018 at 19:26
  • 1
    Even with a population of 18,000,000, 66k responses is a very statistically significant sample, and is representative of the population as a whole. Mar 20, 2018 at 1:41
  • 6
    @BinaryPatrick Why would you assume it's a representative sample?
    – Jeremy
    Mar 20, 2018 at 13:00
  • It's not an assumption. With a confidence interval of 99% and a statistical margin of less than 1%, 60k is still well above what would be required for any normal statistically significant sample. At least 6 times larger. As someone said earlier, devs aren't a fringe group, so a statistical sample doesn't need to be a great percentage of the statistical population. Mar 20, 2018 at 13:18
  • 9
    @BinaryPatrick Your numbers don't matter if the sample was taken from only one subset of the population--say, a single Internet community. You could have a huge percentage of the population and still have a non-representative sample if you haven't sampled in a truly random way from the entire population. Sampling bias is a real thing. Mar 20, 2018 at 18:09
  • 9
    @BinaryPatrick You're discussing confidence intervals relying on math that assumes a representative/random/uniform sample of the subject group. I assume that the Stack Overflow user population is a highly biased and non-representative sample of the overall developer population: biased towards less experienced developers, towards certain personality types, towards different cultures, and different technologies. Do you think that's mistaken?
    – Jeremy
    Mar 20, 2018 at 19:34
  • @JeremyBanks I don't think the user base is highly biased, and you can deduce that it is not. The number of people who answered the survey are a sample of the population of SO, and the population of SO is a sample of the entire user base. Both samples are significant of their populations. You might argue that there is a slight bias because the sample is only SO developers, but the sample size is so large I would believe it is negligible. 66k results is a lot of results, more than most studies. If anything SO might have a bias of being too eager to report, but would that effect accuracy? Mar 20, 2018 at 23:41
  • 3
    @ThomasOwens I just looked at the diff to give you some example changes and there are almost twenty changes in the discussion to change from "developers" to "respondents" or "developers on Stack Overflow", to emphasize who we are talking about. This is in addition to discussion we already had about who made up our sample. For example, the very first data visualization (the maps) demonstrates the similarities/differences between the survey sample, our traffic, and our estimate of professional developers. Mar 20, 2018 at 23:50
  • 5
    @BinaryPatrick Estimating sample bias is complex and important to think about, and even surveys from big, established polling operations deal with these issues. With the way the survey was fielded and responses on, for example, the "Community" section questions, it is reasonable to conclude that we have a sample of SO users, not all registered or active ones, but biased toward those who are. (I'm using "bias" here in a technical, non-pejorative sense.) Mar 21, 2018 at 0:02
  • 5
    @JeremyBanks I am still willing to argue for the value of a survey like this, both for the SO community and for the larger tech community, but we do have great evidence that survey respondents and/or SO engaged users are not representative of all developers. As one example, ~18% of US CS undergrad degrees currently go to women but ~9% of US survey respondents were women. These are issues I think about a lot, for multiple, probably obvious reasons, but it does not mean this exercise is pointless or there is nothing to learn. Mar 21, 2018 at 0:05
  • 1
    @JuliaSilge Oh definitely; it's still one of the best resources for the kind of information, and a valuable service that Stack performs for the community. I am just nitpicking against Patrick about how certain results might be interpreted. I didn't mean to question the value overall.
    – Jeremy
    Mar 21, 2018 at 0:09
  • 1
    @JeremyBanks And to be clear it is our responsibility to do the right thing and make clear who we sampled and how, and to fix that when we need to make it better. Mar 21, 2018 at 0:24
5

It's unlikely that any amount of such additional writing would make a significant difference, unfortunately. In the modern day, most article writers understand survey methodology not at all, and make no effort to adjust their writing based on any sort of qualifiers. As someone who works in the field, it's quite frustrating.

That said, there are both good and bad things about what Stack Overflow does here. They clearly explain the methodology, including reasons for excluding cases, and they explain specifically who is included. Any competent survey statistician could take the data here and generate appropriate weights to make this approximate a representative sample. They also clearly explain how they recruited respondents, so someone could understand how well this is likely to represent the population due to methodological reasons. The methodology section of this report is as thorough as most survey methodology reports I've seen, even from much more rigorous surveys.

They don't use weighted data in their analysis, which is unfortunate, but it's likely because of a fundamental problem: how do you determine the targets for weighting? Likely, this survey is the largest of its kind - certainly the largest I've heard of - and thus there really isn't anything you could weight to. It would be nice if there were comparative weights, perhaps, that held some changes in demographics static from year to year so that other year to year comparisons could be made - for example, use last year's nationalities as a weight target, so you could see how salaries have changed irrespective of nationality - but that's probably beyond the scope of the basic analysis here. It's not like there is a census that one could use for targets, or even something along the lines of the American Community Survey (which is often used for things like this) at the international level.

Realistically, I don't think there's a lot more they could do to affect how this data will be misused. It's probably as representative as it could possibly be, and the extent to which it is not representative is relatively easy to understand from the methodology section.

3
  • 13
    It's unlikely that any amount of such additional writing would make a significant difference - It's not about making a difference in people (mis)using the data. It's about the ethical responsibilities of the publisher of content to communicate the intent and limitations of data up-front. The Methodology section is at the end. By naming the survey how it was named and not having a summary of the methodology in the Overview, Stack Overflow is not doing all they can do to ensure the survey data is understood. Mar 19, 2018 at 16:36
  • 2
    "in the modern day" Misuse/lack of understanding is hardly a new phenomenon.
    – jscs
    Mar 19, 2018 at 17:35
  • 6
    Your answer tacitly assumes that the results are geared toward statistically literate people in the first place, since its discussion focuses strongly on how they would interpret it. It's not. While SO probably can't stop other media outlets from being incautious, people who are not statistically literate that read the original reports may benefit from a more explicitly laid out discussion of the limitations.
    – jpmc26
    Mar 19, 2018 at 18:27
1

People should be able to discern that the likely respondents are Stack Overflow users, that this is a small representation of the global programming community. The survey is clear how many people take the survey, it doesn't purport to be a peer reviewed article.

You're never going to stop people from cherry picking results from surveys or twisting results to suit their perspective. This is something we have to deal with in all walks of life, not just here. If you find yourself up against an employer or peer misquoting the survey results, you can explain your reasoning for this, maybe, not being an accurate representation for the programming community at large.

None of this means that the survey is not useful. The results are interesting and probably indicative to some extent of global programming trends (but don't hold me to that, I haven't checked out other surveys or research).

enter image description here

Mind you after seeing this pic of a dog with a piece of ham on its face... nothing would surprise me.

8
  • 24
    Given the current branding of the survey, there is absolutely nothing to indicate that respondents are Stack Overflow users. Coupled with the fact that this is being posted on multiple third-party sites as a survey of the software development community, there's a lot of bad information out there. Stack Overflow needs to be proactive - when someone hits the survey page, it should be very clear up front who the respondents are and that the survey isn't representative of the broader development community. Mar 18, 2018 at 11:51
  • 2
    @ThomasOwens the survey gives a break down of who has answered the survey with respect to SO
    – user3956566
    Mar 18, 2018 at 12:10
  • That is buried in the results. It should be in the Overview. Especially when the Overview says things like "we ask the developer community" (no - the Stack Overflow community was asked) and "we've published our Annual Developer Survey results" (no - it's not a Developer Survey, it's a survey of primarily Stack Overflow users who self select, with some outside participation) and "over 100,000 developers took the 30-minute survey" (that's maybe 0.6% of all developers worldwide and at best 1.1% of registered SO users, and with self-selection, not necessarily a representative sample). Mar 18, 2018 at 12:28
  • 13
    @ThomasOwens I honestly don't see why you're so upset about this. It's called the Stack Overflow Developer Survey. Why are you so upset? What is it about the results you don't like? There's something getting to you and I'm not trying to be mean here, I'm honestly asking. It's not a developer survey it's the stack overflow developer survey our developer survey.
    – user3956566
    Mar 18, 2018 at 12:31
  • 23
    There's nothing about the results that I don't like. I don't like how Stack Overflow isn't taking appropriate steps to make sure the data and limitations in the data are communicated. Steps should be taken to ensure that all readers have a good understanding of what the data truly represents. Mar 18, 2018 at 12:33
  • 3
    +1; if anyone or any organisation is expected/required to prevent individuals/organisations wilfully/incompetently use information then you end up in a world where there is only the approved orthodoxy and no contrary information is allowed. This is admittedly rather more general than this answer but unless it is normal for other people/organisations to take responsibility for their own actions rather than blaming others (johnny hurt my hand when I punched him) then you've got a society sized FUBAR
    – Richard
    Mar 19, 2018 at 9:57
  • 2
    @Richard You cannot prevent third-parties from misusing information. However, you can, in the content that you own and control, take appropriate steps to ensure that everyone does understand how to use the information properly and what limitations exist. Mar 19, 2018 at 16:37
  • If you want to move this discussion to chat, I could engage in an exchange of views later this evening if you want to discuss it.
    – Richard
    Mar 20, 2018 at 11:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .