To be honest, I'm having a very hard time wrapping my head around your point. Physical attraction has many different factors, but for MANY people, sex and gender play a very large roll. For some people, gender and gender expression has everything to do with their attraction. They are only aroused/attracted by people with very specific gender expressions and/or sexes. Not always, and not for everyone, but it absolutely has the ability to be a large factor for certain people.