2

As stated in the Stack Overflow 2017 survey test meta post, there are conditional questions/question groups in this year's Developer Survey.

This year some questions are only asked of a subset of survey takers based on previous responses or randomly assigned groups.

Assuming they were actually used in the final questionnaire:

  • Which of the questions were conditional, and what are the trigger conditions for them?
  • What were the question groups?
  • How are the question groups used? E.g. Are you randomly assigned to a group of questions at the start of the questionnaire?
8
  • 2
    Might want to hold off answering this one until the survey closes, don't you think?
    – jscs
    Jan 13, 2017 at 23:48
  • 1
    @JoshCaswell I thought about that, but then I reasoned that it's not like it's a test or quiz - assuming you trust the users to answer truthfully, it shouldn't change anyone's answer. I suppose someone could use this to answer specific questions in a certain way, but to what end? I think the point of the conditional questions is that they are probably not relevant to you anyways, so why would you?
    – tehDorf
    Jan 13, 2017 at 23:54
  • ... I suppose this might encourage users to retake the questionnaire if the question groups are assigned randomly per session, just to answer all the questions. Although, if they are that interested, they would probably retake the quiz anyways.
    – tehDorf
    Jan 14, 2017 at 0:00
  • 1
    Some people just enjoy tweaking noses and messing around with systems that others have set up, such as surveys. And a fair number of those people become programmers. If there aren't a handful of people who have already tried to figure this out by just re-taking the survey, I'll eat my hat.
    – jscs
    Jan 14, 2017 at 0:00
  • So would this be more or less likely to encourage that? Instead of writing some kind of script to run through the quiz a hundred times, they could just look here and be satisfied. If by 'tweaking noses' you mean deliberately provide false information, I don't see how this question/answer would change their mind/actions.
    – tehDorf
    Jan 14, 2017 at 0:04
  • ...programmers are also lazy.
    – tehDorf
    Jan 14, 2017 at 0:06
  • 1
    Yeah, maybe it wouldn't. /shrug
    – jscs
    Jan 14, 2017 at 0:07
  • Yeah, I'll second you on the /shrug. This is all hypothetical. :)
    – tehDorf
    Jan 14, 2017 at 0:13

1 Answer 1

2

There are two techniques we're using: conditional triggers and random assignment of blocks of questions.

The conditional triggers are used, for the most part, to ask appropriate follow-up questions and eliminate, as much as possible, irrelevant questions. We heard a lot of complaints last year about too many questions where "this does not apply to me" was the best response option.

As for the random assignment of blocks of questions to different respondents, we did that to cut down the total amount of time respondents have to spend on the survey. Currently each respondent is being shown six out of eight blocks; depending on analysis of the first few days' results, we may dial this down if too many people are dropping out before the end. In addition to the randomized blocks, there are a lot of questions that are being presented to everyone; the blocks represent most of the "heavy" questions that take up a lot of time (i.e. grids).

We'll be publishing the survey instrument after the close of the field period, so everyone will be able to see content that wasn't presented to them. Please don't take the survey twice -- it will bias the sample.

2
  • 2
    I'm not sure if this was brought up during user testing, but what do you think about providing two survey links - one for the full survey, one for a 10 minute survey (or whatever optimal length you decide)? If not this year, then maybe next year, if you use question blocks again.
    – tehDorf
    Jan 15, 2017 at 3:04
  • @tehDorf that's an interesting idea, and certainly worth considering.
    – Kevin Troy
    Jan 15, 2017 at 20:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .