Freedom of speech is not freedom from consequences. If people are free to speak, I am also free to speak out against them. If people support particular speakers, I am also free to speak out against those supporters, and to shun them.

If you don't believe me, maybe you'll believe a lawyer who does a lot of free-speech work: https://www.popehat.com/2013/09/10/speech-and-consequences/

Yes, it is legal to do many immoral things.

Edit: According to you, shouldn't Twitter be at fault for letting Pax say such hurtful things? They're providing him a platform.