Midrash for Nazir 117:5

(Devarim 22:5) "A man's vestment shall not be upon a woman": What does Scripture come to teach us? If that she should not wear colored clothing, is it not written (Ibid.) "for the abomination, etc."? And this is not an abomination. It means, rather, that a woman should not wear what a man wears and go among the men (for licentious purposes), and a man should not wear colored clothing and go among the women.