I think the inherent struggle of Mark's character is that he, in his heart of hearts, wants to believe in God, and isn't sure if he does or not. He has rejected his parent's version of Christianity, but he speaks passionately when speaking about faith and is in an unresolved place with his faith and his marriage at the end of the play. He's contradictory because he's the sibling most caught in the two worlds of the play - the world Ginny and Bill prefer to live in (where Diana still very much lives) and the outside world where Evie is thriving and Johnny is finding himself after a period of struggle. Mark is stuck and adrift, and the only time in the entire play he seems confident is when he returns to his identity as a preacher. I think it works, though I'm not sure how much of the inconsistency is due to Quinto's performance, which I liked but didn't love in an early preview. I preferred the actor who did it in Berkeley.
Rachel is reluctantly participating in carols and other dinner stuff because she and Mark have agreed to keep their separation private, so they're keeping up a veneer of normalcy. At the start of the play, everyone, with various degrees of resentment and reluctance, is just trying to get through the holiday without incident, which obviously, does not work. It's also implied Rachel has spent a lot of time trying to fit in with the family (she converted and has spent every Christmas Holiday with her husband's family), and now she's barely going through the motions.
I think Rachel walks out in the end because this is her after reaching her breaking point. It isn't necessarily the events of this Christmas, but a previous incident where Diana told her that her miscarriages were due to Mark's loss of faith, combined with Mark's repeated inability to put her needs in front of his family's, that's lead to a breech in the marriage that the events of the play expose as being too deep to come back from. The moment where she witnesses his speech to Lorren is really ambiguous and can be taken a couple of different ways.
The part with Mark and Rachel singing together in darkness (Is it Ginny now? it was Rachel in early previews. Did they change it?) seems to me to be a non-literal moment, much like the singing of Shenandoah at the end. It played to me as a moment depicting their love but disconnect from each other; it was telling that Rachel dropped out of the song at the mention of children. It also covers the transition scene between the play's two scenes; I think in early versions of the play it was a two-acter, and this was where the intermission was. So it's probably more logistical than anything else.
The moment where Diana comes downstairs after feeding her baby, with her dress undone and visibly out of it, very clearly indicates early on that something is up. But you're right, it is a deliberate reveal that her mental issues are so severe.