Religious affiliation - unlike race, sex, or sexual orientation - is a choice, and it is therefore valid to consider it when judging other people. Though there surely are reflexive anti-Muslim bigots, reasoned criticism of Islam is not bigotry, any more than criticizing the Republican Party is bigotry.
Muslim belief has often led to oppression of, and violent acts towards, women and LGBT people. This is also true of other belief systems, of course, but today the worst of these atrocities are indeed committed at the hands of those who profess Islam in places where Muslims have significant political power.
There are more than a billion Muslims and I know that most of them do not support such violence. However, the choice to align oneself with a philosophy that is connected with such violence is not a neutral one. Just as I would use membership in the Republican Party as one factor in sizing someone up, so I would use the choice to affiliate with an anti-LGBT religion. (I am quite certain that some religious people judge me negatively for my atheist beliefs, and that too is fair game.)