Is he in jail? Was he fined or deported? no. When it comes down to it, he has freedom of speech, like everyone else, and really hasn't suffered for it.
On the other hand, other people also have freedom of speech. Saying someone's speech is offensive isn't an attack. No one has deprived him of what he has to say.
However, he has a contract with A&E, and there's a good chance that contract says something about saying offensive things, moral turpitude, etc. Maybe he violated his contract, maybe he didn't. Without seeing the contract it's impossible to know. Either way, it appears A&E has the right to suspend his contract, and chose to do so, again, they are exorcising free speech.
Freedom of speech doesn't come with freedom from responsibility. I'm just not sure why anyone is surprised. He's a redneck from the south. Such opinions are not uncommon. So a redneck has an unenlightened opinion of gays and a patriarchal opinion of blacks. No surprise there.