I lost my Family in similar fashion 43 years ago, never entirely recovered , don't expect to & that's a reality that I deal with as perpetually as the reality of the loss.
But not for lack of trying, & the efforts would include but not be limited to 2 failed Marriages & a myriad of relationships that left most involved emotionally spent-out, some even chewed-up & spat-out.
So here's some qualified incite into your currant place in his life,,, It's Someplace Else!,,,And always Will be,, Because the consummate reality is You wouldn't even exist were it not for something that wasn't supposed to happen 20 years ago happening anyway whether it was supposed to or not.
And, the reason he's such a space cadet & seemingly out in left-field is because no matter where he is or goes He's always gonna be someplace else too & it's NOT always gonna be with you because Your not supposed to Be there anyway.
After 20 years he's as through it & Over it as he's ever gonna get & the only person who needs to get Over anything is You if your not comfortable with that Place you occupy in his life.
He's no-doubt been to counseling & may even take meds for clinical depression & that's also working as well as it's ever going to.
Reality SUCKS! & he probably isn't in the least dissociated from it as he perpetually lets go of the reality of the Past, but odds are he's always gonna "Come & Go" and as long as Your gonna be a part of his reality in the Present your only choices are to 'Let him',, or just 'Let him Go'.
It perfectly doesn't matter what's "Fair" to either of you, because if "Life" was fair, as I already mentioned, in virtually the same reality you both live in, You wouldn't even exist in his.
In your relationship with him, Reality is as reality does, and you can either Deal with it as it is or Move-on & at least You have a choice,,
But He Doesn't, Because he can no more change what life has done to his presence of mind than regrow a limb if he was an amputee.
Exemplified by the fact that it took me nearly 3 hours to type this answer, but if you think that's odd,, try it yourself after losing your mind & finding only Half of it to make any use of on your Best day...