331

Today I've noticed a blue bar suggesting I take a "how are we doing? site satisfaction" survey.

I ask that the bar be taken down and the survey scrapped. Why? Well, several reasons.

The survey is manipulative

(numbering arbitrarily to make commenting easier:)

  1. By the third question (not including your age) - already two questions have suggested that the "Stack Overflow community" may be "not welcoming".
  2. No mention of site rules ("code of conduct" etc.) as a potential source of dis-satisfaction.
  3. No mention of the penal procedures as a source of dis-satisfaction.
  4. Reduction of users' experience on StackOverflow to a single axis: degree of feeling-welcome.
  5. I speculate there will be a "damned if you do, damned if you don't" : If the "community is welcoming", then SE management's policies will be interpreted as being appropriate or even successful; if the "community is unwelcoming", then further repression will be necessary to combat those unwelcoming anti-socials.
  6. Underlying assumption that this site is fundamentally a social community for new immigrants to join, rather than a curated Questions & Answers platform.
  7. Another baiting question: "How frequently have you have stopped yourself from participating (asking, answering, voting, or commenting) on Stack Overflow Q&A because you’re worried about having a negative experience? " This encourages people to report they have had such a negative experience and have avoided participation. If you don't agree with my interpretation, then - how about adding a question such as "Do you believe the mass exodus of moderators over Stack Exchange's unfair conduct is a cause for concern of the community, and needs to be better addressed?"
  8. There are only a few questions overall, making the weight of the "unwelcoming community" rhetoric even more significant.

Weird US racial politics

You (= SE Inc.) continue trying to peg us into your US-political racist boxes. "Hispanic" is not a race, it's a cultural/lingual background. Most Spanish people are Hispanic (but not the Basques). "Latin" (or "Latino" in Castillian) is another somewhat vague characterization; for examples, the Italians are the most Latin of peoples - but they fit the "White/European" box. Also, let's talk about the "Middle-East". The term is itself problematic - it's to the East only if you're European (and was used by the European Colonial powers)... but even ignoring that: There is no such thing as a "Middle-Eastern racial background". This region has people with dark skin and light skin, speakers of Arabic, Kurdish, Turkish and Hebrew; peoples having migrated from different areas in Africa, Asia and Europe; etc. And it's not like you're asking "is part of your family from the Middle East" which would be a different question.

The title is false / misleading

Speaking of Stack Exchange Inc. and its management - you are not mentioned at all in the survey. Yet - this survey is about "how you are doing", right?

...

I mean, surely it could not be a survey about how we are doing, where "we" is the Stack Overflow body of members; as we did not initiate this survey nor author its questions.


PS:
  • Obligatory "Yes, Prime Minister" clip about getting the opinion poll results you want.
  • I am not opposed to surveying in general, like the developer survey etc.
38
  • 46
    If scrapping the survey is "not an option", at least please consider a complete rewrite. I almost felt offended by taking that survey, even when I chose that I would not give me ethical background (I don't understand how my background would matter if the survey is about SE, not me). Commented Jan 27, 2022 at 22:23
  • 168
    I can never understand why people are asking about my race. I care so little about that in my part of the world that I wouldn't know how to categorize people properly. I don't have time to become a racist just to fill out a survey. It's ridiculous.
    – Dharman Mod
    Commented Jan 27, 2022 at 22:37
  • 6
    it's so that they can make sure they get a response from certain smaller groups that otherwise would rarely end up being randomly picked. That however also leads to the survey not being a representation of the actual userbase overall, more, data representing each... "group."
    – Kevin B
    Commented Jan 27, 2022 at 22:40
  • 16
    People: Most things on meta are opinion based. We don't close them, and we shouldn't close this either.
    – Undo Mod
    Commented Jan 30, 2022 at 21:25
  • 8
    @Undo I seems like many comments have been deleted (and not moved to chat). Did you delete them? I thought the semi-official policy on Meta is to generally leave comments alone unless they're off topic or borderline offensive (or there's like 50-100+ of them)? If nothing else, a comment noting the deletion would've gone a long way. As far as I could tell, the post was only locked earlier because of an interaction between 2 people (a lock doesn't seem like the best solution to that, but I digress), while many other comments were generally on topic, but I can't double-check, so maybe I'm wrong. Commented Jan 31, 2022 at 1:02
  • 8
    Mods: Leave this question open! Also mods: deletes every comment and answer..... Come on, let's face it, this type of heated rhetoric has no place here. While the title is polite, the content is written like a tabloid.
    – Travis J
    Commented Jan 31, 2022 at 5:10
  • 7
    @TravisJ: I "dramatized" the rhetoric to better communicate the impact of the survey.
    – einpoklum
    Commented Jan 31, 2022 at 8:49
  • 11
    @MattK: 1. See my last comment. 2. I believe you're conflating resentment and hate. 3. It appears the sentiments expressed in this post are shared - at least to the extent of upvoting - by a large fraction of people on MSO.
    – einpoklum
    Commented Jan 31, 2022 at 15:27
  • 12
    @MattK the "US-political racist boxes" demonstrably take their racial categories from American politics, and asking in this way obviously and clearly has racist consequences which are detailed in the OP. If you're unclear on it, though, I'd be happy to explain in more detail in the chatroom. Starting with a proper definition of racism, not the one that self-styled "anti-racist" (hint: they are racist) activists in the US (and unfortunately also leaking into Canada and possibly other places) seek to substitute. Commented Jan 31, 2022 at 17:02
  • 9
    @MattK thing is, most people (of those who are knowledgeable about the topic) outside of the US think the US definition of race is either stupid, misleading, useless, harmful, or a combination thereof. SO always talks about being "welcoming" so including a question that's borderline insulting for some regions is interesting to say the least, unless they only care about welcoming US visitors.
    – l4mpi
    Commented Feb 2, 2022 at 11:08
  • 7
    @MattK: It's inappropriate to ask l4mpi for that. Anyway, a "how are we doing" survey should not ask about your race. This reminds me of my time in the Netherlands, where the obsession is with your birthdate. Every dumbest form asked for your exact birthdate. I connect to the Internet - they want my exact birthdate. I loan a bike at work - birthdate. What's it to them? ... and that's not nearly as touchy as race.
    – einpoklum
    Commented Feb 2, 2022 at 20:06
  • 9
    @einpoklum yeah, it's a whole different can of worms and far from optimal, but would at least be more rooted in facts than vague stuff like "white or european" (apparently from a US perspective, europe is not just a country but also a race /facepalm). "In which country did you grow up" would probably be a less-bad question, but that has its own issues as well. But again, without knowing the intent behind the question it's hard to make any recommendations except for "drop it entirely or at least pick one from these not-quite-as-bad questions".
    – l4mpi
    Commented Feb 3, 2022 at 12:18
  • 7
    Hmm, so it's been a week, the score is +312, and other than some periodic cleanups of comments done by the mods, the official response seems to be nil. I guess that says it all. :(
    – Dan Mašek
    Commented Feb 3, 2022 at 23:38
  • 7
    @MattK if the intent is "fairly well known", then please spell it out for me for this specific survey and question, because I honestly have no idea. Especially when it's so broad as to not even be usable for basic demographics (e.g. a black person born and living in the US, whose family lived there for 150 years, is supposed to check the same box as someone who shares their skin colour but was born and lives in an african country like Tanzania or Sudan).
    – l4mpi
    Commented Feb 4, 2022 at 9:47
  • 8
    @MattK: StackOverflow is not a census bureau. National census bureaus ask about many aspects of the surveyed people's lives and background, and the context of such surveys is different. They are not "how are we doing" surveys.
    – einpoklum
    Commented Feb 4, 2022 at 19:41

2 Answers 2

7

There seems to be an official response here: Temporarily pausing the site satisfaction survey

From Thursday, May 19th - Friday, June 17th we will temporarily be pausing the Site Satisfaction Survey.

Based on feedback both here on MSO and internally, we've decided to revise the survey so that it will better capture useful information so that we can use it to provide a better user experience for the people visiting and using Stack Overflow. We understand that many of the community members feel that the wording in this survey can use improvement and thus we are currently working on revising it. We will provide an update when we are getting ready to launch the revised version of the survey.

[...]

So while not exactly what this question is asking for, seems like a step in the right direction.

3
  • 1
    The survey I was presented on tuesday, was far less manipulative than past versions IMO.
    – Kevin B
    Commented May 19, 2022 at 15:01
  • 1
    @KevinB that's true, I also got it a couple weeks back IIRC and I agree. So it seems like we are going towards the right direction. Commented May 19, 2022 at 15:03
  • 1
    Note I asked the question 3 months ago.
    – einpoklum
    Commented May 19, 2022 at 16:46
2

It's been a while and there is no official answer yet, so let me chime in with a few observations.

As an alternative to scrapping the survey it could also be improved. It's not beyond repair, one would just need to respond to your points: for example wordings can be made neutral in order to not suggest things or the amount of questions and options could be extended to cover more areas and make it a more full fledged survey. The confusing "how often have you stopped" question could be clarified. The race question can definitely profit from a more scientific and professional approach (just reading Race (human categorization) - Modern scholarship and that already takes some time, it's complicated) or be scrapped altogether and replaced by country of origin or other relevant categories like level of education.

I wonder if the survey results have a great impact at all currently. Either the community comes out as quite welcoming or if not, it doesn't seem to have any obvious influence on actions taken currently. Layout changes, collectives, unpinning accepted answers - I'm not aware of any recent change that was concerned with a potentially unwelcoming community.

Indeed, the survey is only one of many ways to give feedback. 1:1 interviews with a few selected users, surveys, meta discussions, the general behavior of the users - all that is the pool of feedback the company can draw from. I remember the unfriendly comments robot (a machine learning project to identify potentially unfriendly comments) showed a lower and lower unfriendly comment rate year over year in the past. And the number of newly registered users or new questions remains quite stable on a high level over the last years, if I'm not mistaken. It could be that a potentially unwelcoming community is much less of an issue now than it may have been in the past.

As a summary: I agree with your points, but they can be corrected. The survey should either be improved or if that is impossible scrapped. As long as either of this doesn't happen, users should better not take part in it, because the result likely isn't useful and the time to take part in it might rather be wasted.


P.S.: A little more than three months later, the company seems to have come to the same conclusion and stopped the survey and works currently on improving it. Hopefully they will succeed.

Temporarily pausing the site satisfaction survey

We understand that many of the community members feel that the wording in this survey can use improvement and thus we are currently working on revising it

15
  • 4
    If the company doesn't answer, it's not interested in discussing repairs either. Also, I disagree that the issues I brought up are localized or anecdotal. I question SE inc.'s interests in feedback to begin with, and their intentions regarding the feedback they obtain.
    – einpoklum
    Commented Feb 4, 2022 at 10:08
  • 1
    @einpoklum "it's not interested in discussing repairs either" That's why I wrote that it's better not to take part until the issues are resolved one way or another. They are grave enough. "I question SE inc.'s interests in feedback to begin with" The lack of responses on meta may indicate that they are not so much interested in getting feedback through meta. "their intentions regarding the feedback" This is assuming bad faith. You simply don't know. In 2018/19 I would have actually agreed, but not now. I would not want to describe to malice what could simply be laziness or ignorance. Commented Feb 4, 2022 at 12:14
  • @einpoklum What we will learn here and elsewhere is (in my opinion) that we are not entitled to a response. The company is not accountable to us and we have no direct influence on it. They can simply ignore us if they want. That's the difference to open source software, non-for-profit organizations that may handle such issues differently. I mean, what will you do in the end, if no answer from the company arrives here or on the question on meta.SE asking about rebuilding relationships? Commented Feb 4, 2022 at 12:18
  • 3
    Well, the situation has been very obvious to me for a long while. But - some people buy the sweet-talk of higher-ups like Ms. Dietrich or, earlier, David Fullerton, even to the extent of ignoring some of the circumstantial facts. And that means that we have to use the means of expression that are available to us to underline the disingenuousness of such rhetoric. And - I did not believe, when asking this question, that the company would actually answer it.
    – einpoklum
    Commented Feb 4, 2022 at 12:31
  • 2
    @einpoklum "sweet-talk" Everyone should know by now: If a company speaks of wanting to build a relationship with their community, they never mean it. I'm so used to not listen to that, that I hardly realize if it's been said anywhere. SO as a company is not that much better or worse than others, in my experience. People who buy "sweet talk" are just inexperienced, but then people who linger on might have too much time. Underlining the disingenuousness of such rhetoric is okay, but might become a bit repetitive. Commented Feb 4, 2022 at 15:41
  • 1
    RE: sweet talk / "build a relationship with their community": There's thousands of feature requests from real users on meta if SE wanted to help tackle its biggest problems. I don't work for SE and I probably never will, but if I were hired to help the community, I wouldn't "announce" I was "building a relationship" or "acknowledge that they hear concerns", I'd ask the hard questions first, like does SE want new users or quality content?, what kind of ad dollars are more valuable?, and then just be transparent about it. Anything else is stalling, which is maybe what SE is doing.
    – jrh
    Commented Feb 4, 2022 at 19:30
  • 1
    There's a gold mine of material on what's wrong and how to fix Q/A, for any one of several futures that SE wants to work towards. I'm not really sure if they have any vision at all for Q/A, or maybe they have ideas but none of them are exponentially growing. It's not a great feeling because reality will catch up to them eventually. SO was a pretty great idea, jobs was an okay idea, docs was a weak idea, collectives seems like an even worse idea.
    – jrh
    Commented Feb 4, 2022 at 19:34
  • 1
    @jrh "if they have any vision at all for Q/A" I remember a slogan from the CEO from like 2 years ago, something difficult to understand "code your life" or similarly short. It wasn't a really detailed vision though. I guess the overall topic is kind of online education by crowd sourced knowledge transfer but much of the success probably depends on the details of how you do it. Commented Feb 4, 2022 at 22:14
  • 2
    @Trilarion I think it was "write the script for the future" or something, which was a mildly entertaining observation of how SO can work, but I think somebody said in the comments "I'm pretty sure Nike doesn't have 'Just do it' as a vision". Personally, I think SE always was operating on the thought that "we will be rich before we get to {the point we are at now} where we have to worry about this hard stuff". Now we're here, it doesn't scale, the designers ignored these problems. Who will be bold enough to fix the contradictions in the original design? Will anyone?
    – jrh
    Commented Feb 4, 2022 at 23:05
  • 2
    @Trilarion: If you try to code your life (in which you code your life etc), isn't that the kind of recursion that gets you straight to Stack Overflow? He might have been on to something :-P
    – einpoklum
    Commented Feb 5, 2022 at 17:17
  • 2
    @jrh: "write the script for the future"... so, I guess he was making a lot of promises then?
    – einpoklum
    Commented Feb 5, 2022 at 17:19
  • 1
    @einpoklum and not unlike a poorly designed JS program, nobody thought about what happened if a Promise ended up failing. They removed the error handling from SO, like how most people strip that from their answers. Better just to increase that timeout, surely it will come true eventually right?
    – jrh
    Commented Feb 5, 2022 at 20:08
  • 1
    This is just an "I agree" answer that should not have been posted and just make it feel like we are patting each other on the back. It gives the false impression that our concerns have been answered to an extent, while they haven't even been acknowledged yet. This post could only get a real answer from a CM. Commented Feb 7, 2022 at 6:48
  • @MarcoBonelli No, our concerns haven't been answered and the existence of an answer should not be taken that the problem is solved. On the other hand I tried to put a bit more than just agreement into the answer to make it a bit valuable in the discussion. What I see as additions are: the survey doesn't need to be scrapped, there is not much impact of it on what the company does currently and we should not take the survey until the issues are fixed. I think this is valuable advice, especially if an official answer never happens. Commented Feb 7, 2022 at 8:03
  • 1
    @Trilarion "he existence of an answer should not be taken that the problem is solved" - yeah I know, and I agree with most you have said, still staff will just see "answers: 1" and go "oh ok no need to say more". Commented Feb 7, 2022 at 12:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .