I suppose I am probably guilty of that as well- although you probably won't find anyone who wants more liberty- for myself and for YOU- than me. So, to me, almost everyone seems to reside on the "fascist" end of the spectrum.
The thing is, I don't care how much or how little liberty you want for yourself as long as you don't try to impose your wishes on any other person. That is where the "fascists" expose themselves as the bad guys- most of them can't be content controlling themselves, but feel the urge to control you as well.
*This isn't to say there is anything at all wrong with being a libertine, as long as you don't aggress or steal- it's just what most people think of anyone who is more liberty-oriented than themselves. I'd probably be considered a libertine by the majority of the people, even though in reality I'm pretty boring- unless opportunities arise.