Regarding:
"should the ... question closed if they pick a valid but suboptimal community? E.g., an algorithms question that is valid for S/O but that the person casting the vote feels would be better suited for math.stackexchange.com?"
The model for this site is and has been that site users can vote as they see fit. Unless their voting is part of a voting ring, or unless they are voting targeting an individual user, I don't see how changing this model by accusing site members of bad faith in voting would be helpful or achievable.
And,
"should the OP be downvoted..."
downvotes are on the question as a sign-post of perceived quality of the question, and they are not on the poster, again unless targeted voting is occurring. If targeted voting is suspected, and if the site does not automatically reverse this in the expected period of time, a moderator should be involved.
"A frequent occurrence with the algorithms tag is that questions get downvoted or closed by people who think such questions are better suited to math.stackexchange.com or cs.stackexchange.com."
There are of course two edges to this, and in fact, some may be better suited being posted in other communities. To best avoid this would be for the original poster to show in a concrete fashion why their question is best suited for this site, probably by their showing the relevant code in their language of choice, and discussing the code-specific problem(s) that they are having.
"Where there are multiple natural communities to post a question, should there be support for built-in crossposting? E.g., this could work by having the OP or an editor with sufficient privilege tag the question for crosspost to the relevant communities, after which the questions, all answers, and all comments would show up on all communities."
My fear is that this would unnecessarily muddy the water. How would this be achieved? Multiple discussions on multiple sites? This would risk unnecessary duplication of effort, allowing a member to "spin their wheels" posting something that has already been well solved elsewhere. Would we allow other communities to vote and contributing to data on a shared site? One of the keys to this site is to maintain a high degree of focus, and this would tilt things in the wrong way, and so I don't think that this is either feasible or desirable.