First, thank you for giving us the opportunity to provide insight and feedback in the form of surveys and via follow-up responses here; I am sure the product research team has worked hard on getting these built. With that said, here are my thoughts (a critique, really) on the "technical skills" survey, question-by-question.
General thoughts not specific to a question:
Instead of a picking the least difficult and most difficult out of a series of mutually exclusive options, it would be better (if you really want responses in the form of "least difficult vs most difficult") to provide a slider and let is answer each question on a sliding scale.
Page 1:
Please compare the two options below from the first question:
Think about when you are about to start learning something technical that is brand new to you. Please select which of the following is least and most difficult:
- Find a dedicated time for learning
- Get my manager's support to spend time learning
- [...]
These two seem like the same thing to me. How I spend time at work is based on what my manager wants... because I work for them. So if I need to find a dedicated time for learning, that has to be something my manager supports/approves of, otherwise I'm going to be looking for a new job soon.
The same issue occurs on the second question:
Think about when you're starting to learn something technical that is brand new to you. Please select which of the following is least and most difficult:
- Figuring out how to start learning the new topic
- Find a trusted resource for learning
- Figure out which resource is right for my learning goal
Not only are options 2 and 3 here almost exactly the same thing, but they are both sub-parts of the first option... so how can I possibly choose, here?!
Page 2:
This question just asks the wrong thing, I think. "Difficult" makes a whole bunch of assumptions that might not be correct or might invalidate a bunch of experiences... I don't know if we can just outright skip these answers... and I can't test that either because there is no 'back' option if you change your mind or figure out how to answer a question later.
Think about when you're learning something technical that's new to you. Please select which of the following is least and most difficult:
- Apply or fit my new skills to a language/framework I already know
What if I already know R but I am learning CSS... those are two totally unrelated things and I don't even plan to try and apply or fit them together? Is that difficult? Yes, because they are totally unrelated/not planned to work together. Is that really an accurate answer to the question or the right question to ask? No, not really.
- Apply my new skills to a project at work
Same issue... if I am a Java dev at work, what am I going to do with this new C# knowledge I am learning? Is it difficult when C# is not even allowed at my job? Yes. Is that really what you are asking/wanting to know? I'm sure not.
Page 3:
Think about when you're learning something technical that's new to you. Please select which of the following is least and most difficult:
- Communicate new knowledge/skills with team
If I work on a team, this is a fine question, but what if I work alone? I believe someone else has already raised this issue. It's extremely difficult for me to communicate with a team that does not exist. So... should I put most difficult here because it is impossible? Or should I put least difficult because it doesn't actually apply?
- Writing a blog post to share what I've learned
I have never written a blog post in my life about programming and I don't ever intend to. However, I believe I'm quite competent at writing in general and at communicating information when I need to in written form. How should I answer this, even regardless of that last fact?
- Being able to present at a conference or meet-up
Same as above; never have and probably never will. At this point in the survey I'm seriously considering attempting skipping this question (again, don't know if I even can, and if I find that I can, boy I wish I would've skipped some of the previous questions!) And what does "being able" mean? Having the courage to speak in front of a live audience? Physically able to attend/stand up/speak out loud? Financially able to travel/take time off from work to attend a conference? Technically skilled enough to give a talk about something I know enough about for other people to want to listen to me?
- Finding time to write up internal documentation
OK, this one is weird. It is almost the first good/answerable option of the question, but misses the mark by changing the subject of the question completely; whereas the first three questions are ostensibly about my communication skills, this one is about time management instead. Maybe this was intended to be about writing documentation, not finding time, and just got missed during review?
Think about using courses specifically to learn something brand new to you, please select which of the following is least and most difficult
I tried skipping this question because I don't really learn from "courses" so I don't have any way to answer this question. Finally found out I can't skip. And I can't partially answer either (I tried to mark something as "most difficult" in the question above on page 3 but not mark anything as "least difficult", but it complained about that, too). OK, guess I will pick one at random; this does not bode well for the research team's ability to make the best decisions/recommendations based on survey responses.
Page 4
Suddenly the word "important" is colored orange in each question now, but wasn't before.
Think about when you are about to start learning something technical that is brand new to you. Please select which of the following is least and most important:
OK, I'm finally noticing a trend here. You are asking the same questions over and over again (without even bothering to reword it at all), and offering a different set of answers to choose from each time. This should be your biggest clue that the format of the survey is bad. Aside from the confusion over everything I mentioned above, now I'm getting frustrated about repeating myself and imagining fatigue from it all, which is probably going to harm the accuracy/quality of my responses even further.
Instead of the current format, you should post every possible answer at once and let us answer each one, again on a sliding scale (or a radio button/checkbox at the end that says "N/A or I don't do this").
I mean, it's even the same set of answers as previous (just scrambled):
- Figure out how in-depth I need to learn the new thing
- Figuring out what I don't know
- Get my manager/supervisor/s support to spend time learning
- Find a dedicated time for learning
If this is some kind of weird 'elimination round' technique, you could just get all these ranked automatically by comparing how we applied the sliders of each question if you let us answer them all independently, at once. If it's not that, then I don't know why you are asking me the same question multiple times in a row... is it a bug?
I am giving up on the survey now since it seems to just be asking me the same question or two over and over again and I can't even skip those to see if there are any new questions later on.
I imagine this is how many people will respond to the survey. This is probably what Oleg meant in the comments above regarding "unless these are meant to be discarded" --- it is what is referred to as Garbage In, Garbage Out. The term "garbage" here is not meant to disparage the effort the team put into the survey, but in how accurately/truthfully the questions can be answered by the intended audience, and thus how well the team can take the responses and apply meaning/discern a 'signal' from them.
In short, I don't see anything useful coming from the responses to this survey because the responses themselves aren't useful due to how the questions are posed to us.