Impostor Syndrome
Inspired by Sayse's answer, I think it would be interesting to measure confidence in self vs confidence in others:
- On a scale of 1 to 10, how would you rate your programming ability?
- On a scale of 1 to 10, how would you rate the programming ability of your peers?
- On a scale of 1 to 10, how would your peers rate your programming ability?
This would not just measure the Dunning-Kruger Effect; it would also measure impostor syndrome, which is an issue that affects women, minorities, and other under-represented groups in computer science. (Note: I know that impostor syndrome also affects the rest of us. But it's a big issue when talking about diversity, so I'd be curious to see the results.)
With all of the other questions about diversity on the survey, I'm surprised this one's not already on there.
With this data we could do interesting things like compare impostor-syndrome-ness across groups- maybe we're all more alike than we thought? Or maybe certain groups are more self-conscious or critical than others? Then if somebody smarter than me comes up with a good question to accurately determine the survey-taker's actual skill level, there are a ton of interesting comparisons we could make.
Reach Out
I'd also like to see the survey taken by a more diverse group of people. Right now it is mostly focused on pretty active users of Stack Overflow. That makes for a good amount of self-selection- which is great if we're trying to survey users of Stack Overflow, but not so great if we're trying to survey the real world.
Anecdotally, I work at a place that does software prototyping for the government. We do very little web development. Most of my coworkers are over the age of 40, and they would never think to take a Stack Overflow survey. So most of the people and technologies I work with are not reflected in this data, and I would bet that most of the rest of the real world is not reflected either. And that's okay, but then I think it's a bit dishonest (and maybe even dangerous) to declare that JavaScript is the most popular language. The real world contains a lot more diversity, and not everybody is a web developer. It's not surprising that web technologies come out on top of an internet survey on a site that a lot of web developers visit. I'm not sure how much we can really extrapolate from that though.
I don't know what the solution to this problem is- maybe "brand" the survey a bit differently and "advertise" it on other sites a bit more? Maybe reach out to groups that might not normally take the survey? Saying something like "we're measuring tech trends across the entire computer science world, so get your voice heard" might be a bit more encouraging to non-SO-users than "take a Stack Overflow survey"?
Computer Science isn't just Ones and Zeroes
This might just be my daydream of the day, but I'd also like to see questions that fight (or at least measure) the stigma of computer science as being "boring ones and zeroes". Maybe a couple questions like this:
- On a scale of 1 to 10, how much of your job is math?
- On a scale of 1 to 10, how much of your job is science?
- On a scale of 1 to 10, how much of your job is art?
- On a scale of 1 to 10, how much of your job is a craft?
- On a scale of 1 to 10, how much of your job is fixing computers?
The results to this might be interesting, and might help dispel some of the misconceptions about what programming is? We could go a level deeper and ask "on a scale of 1 to 10, how much do other people think your job is ___?", but that might be a bridge too far.
Edit: (Gall's comment refers to the above. Below was added afterwards.)
Computer Science Education
With organizations like Code.org pushing to get computer science taught in K-12 schools, I'd also be curious to hear what people think about this hot-button topic. Maybe questions like:
- Should computer science be offered in schools?
- Should computer science be mandatory in schools?
- Should computer science courses count towards graduation?
- At what age should computer science start being taught?
- At what age did you start learning computer science?
This is complicated, because as soon as you say "teaching computer science", people picture 3rd graders being forced to learn C++. In reality, "teaching computer science" usually means having them play games like Light Bot or working with
Scratch. So the wording would have to be pretty carefully chosen. But this is a highly-debated topic right now, so I'd be curious to see the results.