You could say a man is fragile without referring to his masculinity. It sounds like the same old 'Real men...' trope. You're challenging his manhood because you disagree with his perspective. It's hateful and unnecessary.