No, you shouldn't feel bad. My mother-in-law's boyfriend has a son who has several disorders as well, and he acts out like that a lot. Just because a child has a disorder doesn't necessarily mean that he or she won't listen; the little boy I'm talking about listens quite well to his father, but his father very rarely says or does anything about his behavior so it goes unchecked. We never go over to visit them anymore, because I got sick of him putting his fingers in my daughter's face (this kid plays with his own poop and doesn't wash his hands), and constantly trying to pick her up. Not to mention while I was pregnant he thought my belly was a big target for head-butting and punching. He's eight, so he's not a small child. He hurts people, because he doesn't realize how big he is.
I think you should feel perfectly rightous in what you said and did. I have a hunch that she doesn't let him destroy everything in her house; why on earth would she think it's okay for him to destroy everything in yours?
If you really miss her friendship enough to try to patch it up (which I can't see happening, honestly), you need to explain to her why you can't have her son around. List the things he's destroyed, and point out that many times she sat there and allowed him to do it. Ask her bluntly why she feels it's appropriate for him to act that way in your home and around your baby. It'll probably turn into a confrontation, but that's the only way anything will get resolved. Maybe she's under the incorrect impression that you don't mind the way he acts. But as I said, if you don't clear the air and get everything out in the open, there's no way to salvage this friendship.