I think this is a clear "heart vs head" situation. Your head says to cut him out, but your heart says no, am I right?
But in all seriousness, he sounds awful. You don't need to be a doctor to recognize that someone who lets himself get to the point of having lice and scabies and weird rashes also has mental issues. There is absolutely no shame in being poor, but disregarding your wellbeing to that level is another thing entirely (there are poor people that are healthy and tidy, and rich people that abuse drugs, etc).
If a situation is giving you a bad vibe you must trust it. Better safe than sorry. If he is a true friend he should understand. Firstly I suggest installing a security system in your house, then I suggest telling him, when he is in a very calm mood, that you think his lifestyle is toxic to your family, and only if he cleans himself up can you hang out with him again. Just be very open and non-condescending, and if you can try sneak a compliment in there as that will lessen the blow. I do not just recommend just ghosting him because he is a paranoid schizophrenic after all and who knows what his mind will assume.