Using that metric, you should deport American Jews because almost all of them support the same anti-liberty agenda.
Plus, I'd be willing to bet others you could define in some way would be the same.
And, maybe you see nothing wrong with that.
But how can you expect anyone to respect your rights when you won't do the same? Yes, the same goes for them, too. We have everyone wanting to violate everyone else's rights in a big statist free-for-all.
That isn't something I want to be part of.
Even if everyone wants to gang up and violate my rights, I won't violate theirs. Self defense isn't a violation, by the way.
I also remember the words of H. L. Mencken: "The trouble with fighting for human freedom is that one spends most of one's time defending scoundrels. For it is against scoundrels that oppressive laws are first aimed, and oppression must be stopped at the beginning if it is to be stopped at all."
But it's still the only right way to approach it. Liberty for all.