Well then, you need to stop getting upset.
Don't let him yank your chain.
I have a Tibetan Buddhist teacher who never gets upset. He responds with "You might be right" or "I'll have to think about that".
Without giving in, this is about as non-inflammatory as you can get.
Understand this, though. He is used to getting you upset with his behaviors. When you stop reacting this way, he will INCREASE his efforts to upset you for 2-3 weeks.
This is called "the extinction response".
Like when a pigeon is used to getting a food pellet when it pecks the button .. and then the pellets stop coming, and the pigeon goes crazy pecking that button .. until it finally gives up and stops that behavior (the behavior "extinguished").
So if at first he tries harder, once you stop responding with anger, then understand that this IS the start of the behavior stopping. And you just have to hang in there.