41
  1. Which of the following best describes the highest level of formal education that you’ve completed?

Some of the answers here don't make sense. Specifically "Some college/university study without earning a bachelor’s degree" - if you select this answer, doesn't it mean that you've successfully completed at least secondary school? I suppose this could mean that you went to a post-secondary trade school or a college that awards Associates Degrees, but these should probably be two separate options next year.

  1. Which of the following best describes your main field of study (aka “major”) in college or university?

Most of these groupings make sense. "Management Information Systems" and "Psychology" as stand-alone entities don't, to me. I don't know if this is generally true, but at my university, MIS was a degree program in the business college and Wikipedia indicates this is common. As for Psychology, it's usually considered a social science and is called as such in its Wikipedia article. Looking at the article for social science, some humanities (like history and linguistics) are also considered social sciences.

I think that if you read the list from top down, some people who actually studies what is being identified as a social science may check humanities simply because it's first in the list. Someone who studied Psychology may also miss that and check either the humanities or social science option.

  1. How often do you work from home or remotely?

I found this misleading. I have the capability to work from home (officially, up to 2 days a week) as a benefit from my employer. However, I choose to not regularly work from home unless there's something I need to be at home for. "It's complicated" doesn't seem right since it's not complicated. I chose this option, simply because "never" isn't right and "a few days each month" is too much.

  1. Which of the following best describes the industry you work in? This information will be kept private. If your employer is involved in several industries, please choose the one most relevant to your area of the organization.

Another poorly worded question.

My company provides software as a service for pharmaceutical and medical device companies for clinical trials. There's really no one else who can use our software or that we market to. If asked, I work in the software industry and my company is a software company. However, my employer is involved in the pharmaceutical industry.

Really, there's a disconnect between the "what industry do you work in" part and the "if your employer is involved in several industries" part.

  1. Which following best describe you? Please select all that apply.

All of the things listed are technical in nature. Nothing about process improvement (agile coach, ScrumMaster), project management, or leadership/management. It's not clear if these should go into the "something else" box or not.

  1. On which of the following sites do you maintain a profile, CV, or resumé? Please check all that apply.

Stack Overflow Jobs isn't listed? And there's no indication as to if this should go into the "Some other site(s)" box or not for any reason. I listed it just because it seems to make sense.

  1. Which of the following methodologies do you have experience working in? Please select all that apply.

This question is all kinds of broken. Agile is listed, but so is Scrum (a specific Agile framework), but others like Disciplined Agile Delivery, Nexus (scaled Scrum), and other agile methods aren't. PSP/TSP isn't listed. CMMI isn't listed. Domain-Driven Design isn't really a methodology in the same sense that Scrum is. You call ISO 9001 and IEEE 12207 "waterfall methodologies", but I can create an agile implementation that is compliant to ISO 9001 and get it through a CMMI appraisal.

  1. What version control system do you use? If you use several, please choose the one you use most often.

Although I don't use it anymore, I fully appreciate the inclusion of ClearCase.

  1. Which of the following libraries, frameworks, and tools have you done extensive development work in over the past year, and which do you want to work in over the next year?

I thought that this was pretty lacking. I took a look at the top tags on Stack Overflow. No PHP, Python, or Ruby frameworks. Ruby on Rails is on the first page of SO tags, as is Django.

Randomized Questions

I didn't get any really interesting random questions. :( Personally, I'd rather have a crap ton of questions and then have a lot more data rather than randomly selected questions at the end.

28
  • 7
    Agreed with nearly everything you said. Only reason I care is that Stack Overflow uses data from stuff like this to guide their future activities and it's concerning when it comes across as fairly haphazard.
    – enderland
    Commented Jan 13, 2017 at 1:00
  • 11
    I went to college for a period of time but did not finish, as such I did not obtain a degree, hence I fit within the "Some college/university study without earning a bachelor’s degree" category
    – user4639281
    Commented Jan 13, 2017 at 1:32
  • 4
    @TinyGiant The question asks what the highest level of formal education you've completed is. Attending college/university without earning a degree is not completing. It doesn't fit the wording of the question. Commented Jan 13, 2017 at 1:33
  • 17
    I completed spending a period of time attending college without obtaining a degree.
    – user4639281
    Commented Jan 13, 2017 at 1:35
  • 2
    Attending college/university without earning a degree is not completing. - They'll let you audit college courses though. It's possible to attend college with no intention of earning a degree.
    – BSMP
    Commented Jan 13, 2017 at 2:01
  • 2
    @BSMP Still not complete. The only way to complete a college or university program is to get a degree at some level. That's it. Either change the wording the question or remove the answer that doesn't reflect completion. Commented Jan 13, 2017 at 2:03
  • 1
    Huh.. I'd like to review my survey answers. I don't remember seeing "What version control system do you use?"
    – muru
    Commented Jan 13, 2017 at 2:39
  • 1
    Regarding the "where is your CV", I noticed the lack of SO Jobs too. I thought that was weird and did the same thing you did - fill it in in the "other" box. Of the ones you've listed, that was the most confusing - especially since there are SO Jobs specific questions.
    – Andy Mod
    Commented Jan 13, 2017 at 6:53
  • 23
    There should be an option at the very beginning like “I have the time, give me all the questions instead of filtering out random ones”.
    – poke
    Commented Jan 13, 2017 at 10:29
  • 2
    All of the "best describe you" should be technical nature given it's a developer survey. What best describes the technical work you do as a developer for the survey? Instead of asking about other parts of work you may do on a standup or retrospective.
    – random
    Commented Jan 13, 2017 at 16:20
  • 1
    The blog post asks, "So if you code for a living, we need to hear from you!" which implies heavily that developers they're targeting with this survey are people who code for a living
    – random
    Commented Jan 13, 2017 at 16:29
  • 1
    @random Then the survey should make that clear. If you hand me a developer survey and I'm in a software development organization, I shouldn't be confused. Poor UX. The blog post doesn't count - it needs to be part of the text of the survey. Commented Jan 13, 2017 at 16:34
  • 1
    There is a massive lean on programmers and engineers with the wording of the intro on the survey, "We’d like to ask you questions about your favorite technologies, coding habits, and work preferences, as well as how you learn, share, and level up as a developer."
    – random
    Commented Jan 13, 2017 at 16:38
  • 4
    @CalebKleveter Some of the questions were randomly chosen. Different people got different questions. I'm disappointed in that. Commented Jan 13, 2017 at 17:44
  • 2
    OP, re: your first point, I didn't catch the disconnect between the "some college..." answer and the use of the word "completed" in the question wording until I read your comment responding to Tiny Giant. I think the objection should be clarified in your question. Commented Jan 13, 2017 at 17:47

4 Answers 4

6

Thanks for this feedback. I'd like to share with you the "design thoughts" behind the choices you're criticizing, and hopefully convey the extent to which we've been deliberate about those choices and are willing to improve the survey when we get feedback -- especially when it's this thorough and well-thought-out.

So, without further ado...

Specifically "Some college/university study without earning a bachelor’s degree" - if you select this answer, doesn't it mean that you've successfully completed at least secondary school? I suppose this could mean that you went to a post-secondary trade school or a college that awards Associates Degrees, but these should probably be two separate options next year.

A lot of survey design involves tradeoffs. In this particular case, we wanted to keep the educational attainment tiering as simple as possible in order to avoid having lots of response options that would be confusing to respondents in non-US/Canada educational systems. (There are Associate degrees in the UK and Ireland as well, but they haven't fully "caught on" on the Continent, partially due to the excellent high schools and apprenticeship programs that are present in many European countries.) For similar reasons, we didn't break HS diplomas up into sub-groups such as the many varieties of technical/vocational vs. university-prep HS credentials you'll find in the UK, Germany, etc.

There's no one canonical right way to ask about this, but most surveys that ask about attainment do include a "some college, no degree" option. (The American Community Survey includes two -- less than 1 year of credit, and more than 1 year of credit, but still no degree.)

Do we lose fidelity by not separating Associate's degree holders from the "some college, no degree" students? Certainly, and I suspect that we'll see wider variance within this group than the tiers where the credentialing is more clear. (It is, after all, a group that includes Bill Gates.) We don't know how many respondents to expect in this interval, because we didn't ask about formal degrees that weren't directly related to CS in last year's survey. We'll be taking a close look at it during our analysis and determining whether we should break it out more finely next year.

"Management Information Systems" and "Psychology" as stand-alone entities don't, to me. I don't know if this is generally true, but at my university, MIS was a degree program in the business college and Wikipedia indicates this is common. As for Psychology, it's usually considered a social science and is called as such in its Wikipedia article. Looking at the article for social science, some humanities (like history and linguistics) are also considered social sciences.

I think that if you read the list from top down, some people who actually studies what is being identified as a social science may check humanities simply because it's first in the list. Someone who studied Psychology may also miss that and check either the humanities or social science option.

We thought MIS, being more IT-focused than other business majors, deserved special analysis, if we can get a sufficient sub-sample. Psychology is a top-level discipline in the US/Canada Classification of Instructional Programs (CIP), and given the computational focus of some sub-disciplines (e.g. neuroscience), we felt it also could use some special analysis.

I have the capability to work from home (officially, up to 2 days a week) as a benefit from my employer. However, I choose to not regularly work from home unless there's something I need to be at home for. "It's complicated" doesn't seem right since it's not complicated. I chose this option, simply because "never" isn't right and "a few days each month" is too much.

OK, thanks for telling us. Last year we only had the options full-time, part-time, rarely, never -- so this year we tried to break that out a little more finely. (What do "part-time" and "rarely" really mean, right?)

We'll be taking a look at the "it's complicated" write-ins to see what we might do to adjust the scale in the future. We might simply ask how many hours you work each week, and of those how many remote (on average). I'd like to hear from people here whether that sounds invasive, or too much like filling out a form in a doctor's office.

Really, there's a disconnect between the "what industry do you work in" part and the "if your employer is involved in several industries" part.

Yes, it seems so. The intention here was to encourage someone who works at, for example, Verizon to choose "telecoms" or "internet/web services" or maybe even "retail" or "media," depending on what sort of projects they usually find themselves on, the division they're assigned to, or that sort of thing. We considered making this a multiple response, but didn't want to turn it into a "name as many divisions of {GE/United Technologies/Bombardier/etc.} as you can off the top of your head" exercise for respondents.

All of the things listed are technical in nature. Nothing about process improvement (agile coach, ScrumMaster), project management, or leadership/management. It's not clear if these should go into the "something else" box or not.

It appears you answered the first question "I am a professional software developer." Those who selected the "non-dev, some code" or "ex-dev" options got a list that was more management-oriented. We intend to look at the write-ins for both lists and see whether additional options might be needed next year.

Other than branching the list the way I just described, this year we moved from having this be single response to multiple response, which will hopefully allow developers and other SO users express a more complete picture of themselves.

Stack Overflow Jobs isn't listed? And there's no indication as to if this should go into the "Some other site(s)" box or not for any reason. I listed it just because it seems to make sense.

We were concerned that listing SO Jobs here would lead to confusion about what types of profiles we were talking about. ("Well, I log in to Stack Overflow, and I see ads for jobs there, so I guess....") We asked a question towards the end breaking apart having "only" a SO/SE profile vs. having a Developer Story. And a side benefit is we get to see how many respondents volunteered SO Jobs unaided here.

Agile is listed, but so is Scrum...

These suggestions will be helpful in the future, thanks. One input that went into the list was the number of questions under a tag on the Software Engineering SE site, but we're certainly sensitive to the notion that some methodologies/disciplines might be under-represented there.

No PHP, Python, or Ruby frameworks. Ruby on Rails is on the first page of SO tags, as is Django.

A little background here: This question grew out of about half of the options having been grouped with the languages from the previous Q last year. We made languages a separate question in order to accommodate additions both young (Julia, Hack, Elixir, et al) and old (Smalltalk, Assembly, Common Lisp), as well as break out Visual Basic into various flavors.

That said, these are not meant to be representative of every framework in the world. They're really ones that we think might be growing over time, and that we don't think we could reasonably deduce somebody used/wants from the combination of what role they chose (e.g. web developer) and the language(s) they chose on the previous Q. From that point of view, the "ones that got away" for me on this Q are probably Laravel and Flask.

I didn't get any really interesting random questions. :( Personally, I'd rather have a crap ton of questions and then have a lot more data rather than randomly selected questions at the end.

Sorry you felt that way. As I said before, designing surveys is about tradeoffs. One of the things we tried to weigh was making it engaging for both highly involved community members like you and people in their first semester of CS study who don't even know there is a meta site. It may be that we swung the pendulum a bit too far. Thanks again for providing such detailed feedback.

1
  • I found some of the questions difficult to answer, some were not clear, some were not applicable to me and I wanted a "Skip" option. How was the survey validated? Was it trialled on 100 or 1000 of the fussiest and pickiest members of this community before being opened to all?
    – AdrianHHH
    Commented Jan 15, 2017 at 9:55
14

I found a mistake in question 13. It said that Alice was my cousin, and at the end of the question it asked something about Caroline, which was never mentioned.

The question about libraries and frameworks was severely lacking. It was too much web-centric and even then many popular web frameworks were missing (Django?!).

Also, I opted-in to allow further surveys and they couldn't accept my email!

dev-survey

Now, I realize that this is a SurveyMonkey issue... Still, it's pretty infuriating that I have to give up my email tag.

3
5

I found a couple of the questions to be unclear about the difference between requirements and nice to have. On the question about directing a recruiter, where we're asked how important a bunch of things are, in some cases I wanted to express "not important if it's absent, but its presence is a bonus". I wasn't sure how to do that, and ended up saying "not very important" as a compromise.

This was also true of the question about what factors are important in a company. I'll take positive note of a company that supports sabbaticals or provides lunch, but (demonstrably) I don't require either from an employer. Instead of checkboxes (choose N), maybe a scale of "don't care / meh / important" would have worked better.

0

I found the following one confusing:

What version control system do you use? If you use several, please choose the one you use most often.

"Team Foundation Server" has its own option, but this is not a version control system, as TFS can use its own custom version control (Team Foundation Version Control) or Git (see https://www.visualstudio.com/en-us/docs/tfvc/comparison-git-tfvc). So what should I choose, when we use TFS with Git as its version control system...

2
  • 6
    Git, because you are using it as your VCS :D
    – Philipp
    Commented Jan 14, 2017 at 19:56
  • Yea, that one confused me too. But I choose Git because in the end it's Git that does all the "magic". Commented Jan 28, 2017 at 17:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .