is there a conversation for "men's" and "women's" clothing?

hey I'm non-binary (A Man At Birth) and started loving the feeling of leggings ever since my mom bought me a pair of "thermals" and was just wondering if it even matters the size difference. I have strong legs but I don't think that matters