Three general suggestions:
Follow up on "part of the community" question
Test your assumptions from last year's "part of the community" question
This was mentioned in an earlier answer, but I'd be particularly careful here to use a good set of questions that follow survey best practices, and hit it from a few different angles. E.g., one conclusion was:
Stack Overflow Isn’t Very Welcoming
The following statements with a likert agree-disagree response could help you learn more about that. Keep the "part of the community" question as a comparison (obviously).
- I feel welcome to ask and answer questions on Stack Overflow (or maybe to participate in, rather than to ask and answer questions)
- The Stack Overflow community welcomes people like me
This would, on its face, measure your concern (people don't feel welcome), but I'm not a huge fan of the use of the word welcome here. For one, by using the "welcome" word you may be measuring a mix of the actual thing you're trying to measure (whether people feel welcome) and the response of users to a set of new features (the welcoming effort), which you should also be measuring, but more explicitly. In this case, it might be useful to use different phrasing, or slightly adjust what you're trying to measure. This has the added benefit of interrogating why experienced women and minorities don't feel like part of the community (instead of just directing programs and features at inexperienced users). For example, you could use:
- The Stack Overflow community values people like me
- People value my help when I answer questions on Stack Overflow (with an opt out for haven't answered a question)
- People help me when I ask questions on Stack Overflow (with an opt out for haven't asked a question)
These questions would, of course, need to be analyzed by both experience and demographics.
Free response questions could help you generate more specific hypotheses about what makes people feel like they are or aren't part of the community. You could use these to develop better questions for the 2020 developer survey
Follow up on free response items from 2018 survey
Don't let your free answer response items like this one from last year go to waste. Their utility is largely hypothesis generating. Follow this up by developing new specific questions. For example, one hypothesis drawn from a common word analysis was
Developers were largely positive about Stack Overflow, focusing on the helpful nature of the community
Test this with the following three questions, phrased as statements the respondent can agree or disagree with on, e.g., a 5 point Likert scale. These questions can be phrased differently, but the point is to measure the valence of the respondents feelings regarding (1) the helpfulness of community and (2) the site, and (3) more objectively identify whether the respondent was helped. Looking at the concordance/discordance of these responses gives you a strong sense about your strengths and weaknesses here. You might consider tweaking (3) so that, rather than a statement with a likert on agree/disagree, it asks how many times Stack Overflow helped the respondent solve a problem in (time period). Time period should be shorter for this version of the question, and analysis should also involve normalizing by frequency of use (presumably you're asking that question elsewhere).
- The Stack Overflow community is helpful
- Stack Overflow is a helpful website
- Stack Overflow helped me solve a problem at some point in the last year
That's just one example, but generally, it's a good idea to look at hypotheses drawn from whatever analyses were done on the free response questions from last year, pick some important ones, and interrogate them with questions that test those hypotheses.
Interrogate the relationship between the users and the brand, or institution that is SE
There are questions about how respondents relate to the main product (Q&A) and how they relate to each other (community). I would encourage you to also ask questions about how respondents relate to the brand or SE as an entity that produces the product they use. The language here will be in large part determined by your own branding decisions (e.g., you may not want to ask how respondents feel about the SE brand if part of your brand is that you are not a slick, inhuman, corporate entity), so I'm not going to suggest specific wording. Regardless, it will be useful to ask 100,000s of developers about whether they feel SO is responsive, on their side, just wants to make money off of them, is cold and unfeeling, etc. Right now you have a sense of that from recent meta discussions, but these are from orders of magnitude fewer individuals than respond to the developer survey.