The divergence of pop music and musical theater I think started around the rock era, when musical theater writers were more hesitant to embrace that style (there's a great episode of the Hit Parade podcast that sort of goes through the history if you're interested), and I think the song writers have just been in different camps ever since. I can't say I know what it takes to make it onto the radio but I assume that's not a medium that theatermakers are necessarily targeting specifically. Though with the slight surge in popularity of musical theater songs as TikTok audios, I don't think it's impossible that we see it happen again in the future. (I took a friend who had never seen a Broadway show to see Six and she jolted up during All You Wanna Do because she recognized it from TikTok, and I had another friend specifically see Beetlejuice because she had heard That Beautiful Sound on TikTok as well.)
I think the shape of the music industry has changed too. Streaming makes it easier for people to diverge in tastes and there's less of a mainstream sound than there is before, so I think it's harder to come up with a surefire hit (unless you're Taylor Swift, apparently). There's also been some noticeable trends in music that have been influenced by the streaming era, like the hook coming in sooner to try to get people to keep listening, songs getting shorter, etc., and those may not be conducive to musical theater songs, which in most cases have some sort of story/journey to tell.
I do think musical theater as a genre is having a bit of a resurgence, what with more musical movies being made recently and just a better general awareness of new musicals thanks to things like TikTok. And I think there are also other pop stars that have done covers--Greatest Showman also had covers from Panic at the Disco and Kelly Clarkson (who's also featured musical theater on her talk show), Katy Perry covered Dear Evan Hansen, the entire Hamilton Mixtape had a bevy of pop stars/rappers, Ariana Grande sampled My Favorite Things (a bit of a different situation, but still), but I think cover songs also just aren't as popular as they used to be either. I was wondering if someone like Ross Golan would've tried to push some of his songs from The Wrong Man as singles, but it doesn't seem like it's happened.