Although I agree with colleen in that the people doing the casting won't care much about what you're wearing, if you really are concerned I would suggest wearing something that makes you look like you "fit" a part.
Personal example: I know that I appear much younger than I actually am (I'm 26 and most people guess 17-18!), so when I audition, I try to wear something that makes me look like the grown-up and mature woman I am--unless I'm auditioning for a teen character or a children's show.
I figure it might be hard for casting panels to picture me playing an adult if I look like a kid (and vice versa), so I try to help by fitting myself into the show even for the audtion.
But don't go overboard! I wouldn't dress in costume or anything like that!
On the road of life, do not pause for suicidal chipmunks who freeze in your headlights, seeking death by your tires...