I've been on both sides of this - I've waited by the stage door, and I've also been the actor coming out to sign autographs. For me personally - I love doing it - I remember how much it meant to me to meet a performer I admired, so, even if I'm tired I just think about that, and it becomes a joy.
The problem is when you get the crazy fans. It's one thing to meet nice, normal people who hand you their playbill, and chat with you for a minute - it's another to be all but attacked! LOL! I've full on been followed into the subway by fans who didn't know when enough was enough. I think the problem is that you never know who you're going to get on any given night. I know a couple actresses who have leads on Broadway who full on have stalkers. Sometimes you just can't deal with the crazy fans, and as I think was stated before: yes, that ruins it for everyone. You never know who's going to be out there - and there's no screening process to get to stand by the stage door. It's a difficult line to walk because on the one hand you want to meet the fans, and on the other, you want to feel safe and comfortable.
Most of the stage doors I've been at have been perfectly lovely! But then you get the crazy ones - and the problem is there's just no way to tell what a stage door will be like on any given night.
Madame Morrible: "So you take the chicken, now it must be a white chicken. The corpse can be any color. And that is the spell for lost luggage!" - The Yellow Brick Road Not Taken