I don't feel welcome in places that do not want me, so I don't think that managing expectations would help.
For example, let's assume, for the sake of argument, that ServerFault Server Fault is still a place for system administrators in a professional capacity. I know that I am not welcome here because of this description.
You say that Breitbart hasn't made you feel unwelcome? Okay. I say that ServerFaultServer Fault has made me feel unwelcome, by putting up a big sign that says "You aren't welcome here".
but isn't the case of you feeling unwelcomed on you?
Yes, yes it is. But I still feel unwelcomed.
let's say I'm having a party and I've invited a bunch of friends and you (a complete stranger) happen to walk past and want to join in, if I say 'Sorry, only people who are invited or at the very least know me are invited.' am I now in danger of being unwelcoming?
Whether or not you're "being" unwelcoming is beside the point. The point is that I "feel" unwelcomed.
To quote the infamous blog post: "When someone tells you how they feel, you can pack up your magnifying glass and clue kit, cuz that’s the answer. You’re done."
If this is the logic that SO is using, then this is an unwinnable battle. There will always be people that "feel unwelcome", despite everything.