Men should not be able to enter the women's locker rooms or restrooms in my opinion, because they are not women. If we want to allow the opposite sex in, then we should have coed locker rooms and restrooms.
Everyone has a legal sex and that's what should matter when it comes to legal access. A man can't get out of selective service simply because he identifies as a woman. I'm not eligible for women owned business advantages simply because I say I identify as female, some for domestic violence shelter access, and other women only venues, so why should we treat restrooms and locker rooms differently? If a man wants to be treated legally like a woman, then he should go through he process of having his sex legally changed, and when he does that should apply across all sex specific practices.
Some answers have claimed, have mixed sex restrooms or locker rooms is not big deal, to which I say fine, then we should move to coed restrooms and locker rooms, but to call a locker room a women's locker room and allow men to enter (or vice versa) is to misrepresent what that restroom or locker room is. If someone of the opposite sex is allowed to enter, don't call it a woman's or men's restroom, call it what it is: a gender neutral locker room, coed locker room, etc.