It's only a downer if we refuse to engage in relationship with such people, and confirm their long held suspicions of what "the other" thinks of them.
I'm not sure I follow this. Freedom of association means that, if I find someone too vile and disgusting to stand, even if it's "only" because of what they advocate politically, then I am free not to socialize with them. Without socialization, there is no "relationship."
I placed the word
only in scare quotes above, because in a democratic republic such as the United States, advocacy can translate into voting in popular elections, which can translate into representation, which can translate into advocated legislation, which can, depending on the legislation, negatively impact peoples' lives. Speech is a form of political power. Hate speech is banned in certain countries precisely because it can be influential in furthering movements to deny groups of people their rights. If I choose not to, I don't have to have a relationship with someone who wishes to deny people their rights, full stop.
I believe there is evidence to support the premise that people with close relationships to marginalized groups tend not to support oppression of those groups. (Apologies, I don't have a link to any article regarding this at this time, but perhaps others can post links to support or refute this.) But, as far as I know, it does not follow from this that attempting to form close relationships with people who support marginalization will tend to be successful in influencing them to change their position, nor does the cost/benefit ratio favor taking this up, at least on an individual basis. (Ditto, here, regarding links.) If you oppose extending rights to a marginalized group, there's a fundamental difference between on the one hand, say, discovering that your own child is in fact a member of that group and on the other hand being invited to dinner by people you don't know very well but who support the rights of the group. (And here.)
Anyway, I have better things to do than to invite the neighborhood Nazi over for tea and biscuits. If he thinks less of me because I turn my nose up at him, to hell with him. Or her.