37

UPDATE (July 13): We have made significant revisions to the Site Satisfaction Survey and have incorporated a lot of the feedback from the community in these revisions. We are planning to launch the revised version of the survey on July 15.


From mid-May to mid-July, we will temporarily be pausing the Site Satisfaction Survey

Based on feedback both here on MSO and internally, we've decided to revise the survey so that it will better capture useful information so that we can use it to provide a better user experience for the people visiting and using Stack Overflow. We understand that many of the community members feel that the wording in this survey can use improvement and thus we are currently working on revising it. We will provide an update when we are getting ready to launch the revised version of the survey.

If you really love surveys and want some to take in the interim, try out these:

  • The Annual Developer Survey - Just in case you haven't taken it yet.
  • How you gain technical skills - a 5-minute survey from the Product Research team that is exploring how developers and technologists learn new technical skills. The Product Research team is interested in learning more about what's most important and most difficult when learning something new.

That's all I have for now. If you have questions about any of this, let me know but I may have to defer until we're getting ready to launch the new version of the survey.

15
  • 26
    Who is "How you gain technical skills" targeted at? questions are unskippable... and most of them don't seem to even have applicable answers to me, a developer who routinely learns new skills.
    – Kevin B
    Commented May 18, 2022 at 21:25
  • 2
    Which chunk of questions are you asking about? The first sets where it's asking which is the most/least difficult or the most/least important?
    – Catija
    Commented May 18, 2022 at 21:43
  • 16
    the questions related to courses, blogs/presenting, in both sections were the ones i struggled with the most. I don't... use courses, or present, or have a blog, or meetups or need to present to a team
    – Kevin B
    Commented May 18, 2022 at 21:44
  • 1
    Would it help at all if the question was framed more as "Which of the following are..."? I understand it may be difficult for us to take open-ended answers there, just based on the sort of survey it is. For me (I just went through it) I considered "I don't do this thing, so it's not difficult at all" as my justification for those questions.
    – Catija
    Commented May 18, 2022 at 21:46
  • 21
    @Catija just an option for "not applicable" (the wording can be anything) would be nice. I, too, don't have any manager - I am my own. I don't have anything to select in the "most difficult" unless you want me to rank how easy everything is and select "most difficult" from that list. I don't have a team to communicate to. I, too, don't give talks at conferences. I, too, don't write blogs. I, too, don't use courses. Commented May 18, 2022 at 21:49
  • 8
    It just generally felt like I wasn't able to give the answers that I wanted, which means the answers I gave would most certainly be used to say something that I didn't intend from what I chose, etc.
    – Kevin B
    Commented May 18, 2022 at 21:51
  • 6
    Yep, that's it, I also missed indeed a "NA" Option for many of the Answers..., ending up selecting/ranking the "least Bad" Answer(s)...
    – chivracq
    Commented May 18, 2022 at 21:54
  • 14
    And then you will see statements like "most developers found it difficult to give talks at conferences [replace with any randomly chosen statement from the set of questions]" when what we meant was "we do not give talks at conferences"... Commented May 18, 2022 at 21:55
  • 2
    I can understand how y'all are feeling. I went through it myself and had to kinda put myself in a hypothetical situation for some of them. I do think that the confluence of your answers can direct that a bit - for example, if you say "talking at conferences is least important" that kinda indicates that, even if it's the most or least difficult, it doesn't really matter since it's also not important?
    – Catija
    Commented May 18, 2022 at 21:58
  • 26
    No, it does not mean that, @Catija. I find giving talks at conferences important, but I do not give them. I find communication of the knowledge to the team extremely important, yet I do not have anyone to communicate it to. Commented May 18, 2022 at 22:06
  • 18
    Unless this survey results are just meant to be sent to a garbage bin (at which point... why have it in the first place?), there are going to be conclusions drawn from responses. However, they will be wholly inaccurate and misleading given the options provided. Commented May 18, 2022 at 22:08
  • 23
    I think what these questions need is a "not applicable" option. That is, a way to say "I don't do this."
    – Ryan M Mod
    Commented May 18, 2022 at 22:15
  • I would like to know whether "How you gain technical skills" survey gives a badge at the end of it just like "The Annual Developer Survey" does. Anyone?
    – holydragon
    Commented May 19, 2022 at 2:54
  • 3
    Census badge is the only badge issued for completed a survey (the annual one), IRRC, and I do not see any new created, so no, no badges. Commented May 19, 2022 at 3:05
  • 5
    @holydragon no badge... and to be honest I'm not even exactly sure if what I just filled really had much to do with explaining how I gain technical skills. It is instead more of a survey to ferret out whether it is a good investment to start doing courses and who should give those courses.
    – Gimby
    Commented May 19, 2022 at 12:33

2 Answers 2

53

The "How you gain technical skills" is sorely lacking a "not applicable" option. At the moment, it only allows a binary (no pun intended) choice which excludes anyone who does not fall under standard categories (for example, freelancers and non-professional programmers).

It also excludes anyone who does not experience issues with any of the categories (or vice versa). For example, the "think about when you're learning" question. I do not have any issues with either finding time to write internal documentation (I do not have anything internal!), communicate with the team (I do not have one!), write a blog post (I do not have one!), or present something at a conference (I do not give talks at conferences!). Same goes for courses.

Given the above, what should I chose? Should I define a sorting algorithm in my head, assign weight to each option, sort them, and choose the first element as "least difficult" and the last one as "most difficult"? The need for that could easily be avoided with a single "I do not know" / "It is not applicable to me option".

11
  • 3
    Making them optional would similarly serve the same purpose, but I could see why having a choice to say not applicable could provide a bit more of a data point
    – Kevin B
    Commented May 18, 2022 at 22:17
  • 25
    Regarding the blog post, I cannot understand why I would even consider writing one about something that I just learnt. Meaning that I know for sure that I'm a novice on the topic, that I may have misconceptions still and that I am for sure not yet ready to become a teacher of the subject. So what is the actually purpose of this supposed beginner blog post? Is it for narcissistic posing or narcissistic social media "likes"?
    – Lundin
    Commented May 19, 2022 at 6:39
  • 12
    I have no idea either, @Lundin. Apparently, it is a common occurence, though (just visit dev.to) - which is quite puzzling as I too, would not find it appropriate to start spewing posts about something I just learned until I have a solid understanding on what it is. Commented May 19, 2022 at 6:55
  • 23
    Where else are they going to find contributors for the Stack Overflow Blog?
    – Cody Gray Mod
    Commented May 19, 2022 at 8:32
  • 5
    @Lundin because that completes the cycle of learning where you externalize your understanding, the thing is that that only works when you have someone to guide you and correct whatever misunderstanding or misconception you have, not a bunch of randoms that are as clueless as you, since no one knowledgeable enough to correct you will search for your blog post.
    – Braiam
    Commented May 19, 2022 at 11:38
  • 14
    @Braiam And yet the survey insists that the first thing you should do after learning new knowledge is to write a blog, tell your co-workers and hold a conference speech(!) about it.
    – Lundin
    Commented May 19, 2022 at 12:38
  • 5
    @CodyGray with the sickest of burns!
    – Ian Kemp
    Commented May 19, 2022 at 13:46
  • 6
    @Lundin it's entirely to allow that person to create an extremely low-quality blog post, riddled with poor spelling and grammar, lacking basic understanding and filled with as many buzzwords as possible, that can then be spammed to as many programming news sites like HN and /r/programming as possible, sometimes for vanity and sometimes for literal ad clicks. All of the people who do this are irredeemably terrible human beings.
    – Ian Kemp
    Commented May 19, 2022 at 13:52
  • 2
    "How you gain technical skills" could use an "autodidact" option. Commented May 19, 2022 at 14:11
  • 1
    @Lundin: people primarily write low-grade blog posts to boost their own or their site's visibility and SEO metrics, not to share knowledge. IanKemp: true, but SO in recent years is seriously losing Google SEO ranking wars against the paid tutorial/bootcamp sites. It's getting drowned out for new users out there. So if SO want to do a content-farm-type blog to counteract that, then it serves a good purpose, even if no human ever reads it. Anyway, the Site Satisfaction Survey sounds skippable. (Do the respondents who didn't complete the SSS show up in the results as dissatisfied?)
    – smci
    Commented May 20, 2022 at 5:46
  • 1
    @smci another reason that SO is dropping out of Google search results is because SO allows literal garbage to be posted, and stay up, as questions. Which then gets indexed by Google, causing people to click through, but when they see it's literal garbage they stop visiting SO search results, causing Google to derank SO. In other words, "be nice" is doing exactly what the community said it would and the opposite of what SE Inc. intended... karma is indeed a female dog.
    – Ian Kemp
    Commented May 23, 2022 at 10:01
16

First, thank you for giving us the opportunity to provide insight and feedback in the form of surveys and via follow-up responses here; I am sure the product research team has worked hard on getting these built. With that said, here are my thoughts (a critique, really) on the "technical skills" survey, question-by-question.

General thoughts not specific to a question:

Instead of a picking the least difficult and most difficult out of a series of mutually exclusive options, it would be better (if you really want responses in the form of "least difficult vs most difficult") to provide a slider and let is answer each question on a sliding scale.

Page 1:

Please compare the two options below from the first question:

Think about when you are about to start learning something technical that is brand new to you. Please select which of the following is least and most difficult:

  • Find a dedicated time for learning
  • Get my manager's support to spend time learning
  • [...]

These two seem like the same thing to me. How I spend time at work is based on what my manager wants... because I work for them. So if I need to find a dedicated time for learning, that has to be something my manager supports/approves of, otherwise I'm going to be looking for a new job soon.


The same issue occurs on the second question:

Think about when you're starting to learn something technical that is brand new to you. Please select which of the following is least and most difficult:

  • Figuring out how to start learning the new topic
  • Find a trusted resource for learning
  • Figure out which resource is right for my learning goal

Not only are options 2 and 3 here almost exactly the same thing, but they are both sub-parts of the first option... so how can I possibly choose, here?!

Page 2:

This question just asks the wrong thing, I think. "Difficult" makes a whole bunch of assumptions that might not be correct or might invalidate a bunch of experiences... I don't know if we can just outright skip these answers... and I can't test that either because there is no 'back' option if you change your mind or figure out how to answer a question later.

Think about when you're learning something technical that's new to you. Please select which of the following is least and most difficult:

  • Apply or fit my new skills to a language/framework I already know

What if I already know R but I am learning CSS... those are two totally unrelated things and I don't even plan to try and apply or fit them together? Is that difficult? Yes, because they are totally unrelated/not planned to work together. Is that really an accurate answer to the question or the right question to ask? No, not really.

  • Apply my new skills to a project at work

Same issue... if I am a Java dev at work, what am I going to do with this new C# knowledge I am learning? Is it difficult when C# is not even allowed at my job? Yes. Is that really what you are asking/wanting to know? I'm sure not.

Page 3:

Think about when you're learning something technical that's new to you. Please select which of the following is least and most difficult:

  • Communicate new knowledge/skills with team

If I work on a team, this is a fine question, but what if I work alone? I believe someone else has already raised this issue. It's extremely difficult for me to communicate with a team that does not exist. So... should I put most difficult here because it is impossible? Or should I put least difficult because it doesn't actually apply?

  • Writing a blog post to share what I've learned

I have never written a blog post in my life about programming and I don't ever intend to. However, I believe I'm quite competent at writing in general and at communicating information when I need to in written form. How should I answer this, even regardless of that last fact?

  • Being able to present at a conference or meet-up

Same as above; never have and probably never will. At this point in the survey I'm seriously considering attempting skipping this question (again, don't know if I even can, and if I find that I can, boy I wish I would've skipped some of the previous questions!) And what does "being able" mean? Having the courage to speak in front of a live audience? Physically able to attend/stand up/speak out loud? Financially able to travel/take time off from work to attend a conference? Technically skilled enough to give a talk about something I know enough about for other people to want to listen to me?

  • Finding time to write up internal documentation

OK, this one is weird. It is almost the first good/answerable option of the question, but misses the mark by changing the subject of the question completely; whereas the first three questions are ostensibly about my communication skills, this one is about time management instead. Maybe this was intended to be about writing documentation, not finding time, and just got missed during review?


Think about using courses specifically to learn something brand new to you, please select which of the following is least and most difficult

I tried skipping this question because I don't really learn from "courses" so I don't have any way to answer this question. Finally found out I can't skip. And I can't partially answer either (I tried to mark something as "most difficult" in the question above on page 3 but not mark anything as "least difficult", but it complained about that, too). OK, guess I will pick one at random; this does not bode well for the research team's ability to make the best decisions/recommendations based on survey responses.

Page 4

Suddenly the word "important" is colored orange in each question now, but wasn't before.

Think about when you are about to start learning something technical that is brand new to you. Please select which of the following is least and most important:

OK, I'm finally noticing a trend here. You are asking the same questions over and over again (without even bothering to reword it at all), and offering a different set of answers to choose from each time. This should be your biggest clue that the format of the survey is bad. Aside from the confusion over everything I mentioned above, now I'm getting frustrated about repeating myself and imagining fatigue from it all, which is probably going to harm the accuracy/quality of my responses even further.

Instead of the current format, you should post every possible answer at once and let us answer each one, again on a sliding scale (or a radio button/checkbox at the end that says "N/A or I don't do this").

I mean, it's even the same set of answers as previous (just scrambled):

  • Figure out how in-depth I need to learn the new thing
  • Figuring out what I don't know
  • Get my manager/supervisor/s support to spend time learning
  • Find a dedicated time for learning

If this is some kind of weird 'elimination round' technique, you could just get all these ranked automatically by comparing how we applied the sliders of each question if you let us answer them all independently, at once. If it's not that, then I don't know why you are asking me the same question multiple times in a row... is it a bug?


I am giving up on the survey now since it seems to just be asking me the same question or two over and over again and I can't even skip those to see if there are any new questions later on.

I imagine this is how many people will respond to the survey. This is probably what Oleg meant in the comments above regarding "unless these are meant to be discarded" --- it is what is referred to as Garbage In, Garbage Out. The term "garbage" here is not meant to disparage the effort the team put into the survey, but in how accurately/truthfully the questions can be answered by the intended audience, and thus how well the team can take the responses and apply meaning/discern a 'signal' from them.

In short, I don't see anything useful coming from the responses to this survey because the responses themselves aren't useful due to how the questions are posed to us.

5
  • 8
    I think asking the same/similar question twice is a trick so the respondent is self-validating the accuracy of their answers, giving the data analist a much easier task in reducing the error margin and/or giving more weight to those surveys that self-validate better. I doubt if surveys like these are a good tool for an on average highly skilled population. We're programmed to outsmart the goal of the survey.
    – rene
    Commented May 20, 2022 at 6:14
  • 2
    I had the same experience with this survey, except I gave up on page 3. Congrats on making it to page 4! Commented May 21, 2022 at 2:46
  • 1
    Generally speaking, management wants one to spend their own time learning, hence the difference in the options. Commented May 23, 2022 at 13:19
  • 2
    @JosephDoggie I'm sorry to hear if you have had that experience, but it's not at all a reasonable expectation for a company to want an employee to learn some job skill and not provide the time or at least the funding to do so. education in general (e.g. a new degree) is one thing, but an applied skill (like a language or tool) is something that the company absolutely should provide time and resources for if they want their employee to learn it.
    – TylerH
    Commented May 23, 2022 at 13:53
  • 2
    "Important" isn't highlighted in the second set of questions vs the first - the first set of questions doesn't ask about "importance" at all - it's asking about "difficulty". I don't think important and difficult are the same, so maybe it's just that you didn't realize that the difference between the two sets is the difficult/important change. Lots of important things are actually pretty easy. :) As to the questions about taking courses, they're now skippable (though the others are still required).
    – Catija
    Commented May 25, 2022 at 22:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .