Either own what you said, accept both the reaction and the consequences of what you say, or stop saying things that only an asshole would say. To paraphrase Randy Millholland, for someone who doesn’t thrive on controversy, you seem to say some shitty things about people a lot.
Either way, for the love of christ, stop crying that people are thinking unkind things about you when you say things about them that are kind of shitty. If being thought of as an asshole bothers you, stop being one.
[If nothing else, it’s to the point.]