Well, I think it is common courtesy to know when your husband is coming and going.
What if you needed something at the store that he could have easily picked up? That would have been a missed opportunity.
Asking him where he is going when he gets up to go into another room is ridiculous.
You two seem to be having a communication problem. Rather than ask him where he is going or has been, why not change things up and communicate differently? How about "I am going to the gym, please keep an eye on (son's name here). If he would have said he had to go out and pick up his daughter, then that would have been a good time for you to say you would skip the gym to be with your son.
See, all you have to do is communicate. Knowing that when you ask him where he is going or where he has been makes him feel like you are mothering him, then you should stop that.
Perhaps if you two cannot talk, you should have a note pad in a handy spot so you both can leave notes about when you will be home or will be going out.