Okay, a lot of grounds to cover, but let's start with bigger doesn't always win. Matter in fact once you get weapons involved they are they are as vulnerable as tiny old lady if not more so because it's easier to hit bigger mass than small ones.
Size doesn't matter athleticism does. Size will give you reach and will give you mass to better absorb the impact, but those two are things you can learn to work around in your regular training. What people usually say is bigger and stronger, with the stronger refering to more athletic. More athletic is the part that will give you trouble, but if you train hard in martial arts you will also be athletic as well. So no major advantage there.
But what if they are still more athletic than you? Well that's why you have martial arts. Martial art is a force multiplier. It takes your strengths and uses it to uttermost effeciency to give you far greater results than what you should be able to acheive with strength alone. Train hard and you gain more skill, which will let you fight much more athletic people and win.
Sparring isn't real. This is very true, but it actually work to the advantage of the skilled not the purely athletic. Your strikes will connect harder without the gloves, all the illegal strikes just become okay to use which only helps magnify your strengths even more, and if you're weapons user like kali guys then you just went from leaving bruises with sticks to leaving cuts and stabs with knives. Realistic situation makes martial artists more dangerous, not less.
The hierarchy of self defense works in this way. Strategy, skills, then athletism. In self defense to people will not enter into fight fulling knowing and prepared as you would in sparring. Usually one party would ambush or surprise another and attempt to end the fight before the ambushed as a chance to even mentally recognize that they need to defend themselves. So suddenness and ability to quickly finish everything is key. This is the real issue with self defense. If there are people that are only concerned with strength vs skill in terms of self defense, they need to get their head out of sparring mentality. Because the real key will be how fast can you switch from social to violence and apply everything that you have to saving your life and well being.