Elle,

I'm not sure this can completely be lain at the feet of the Patriarchy. The average man is simply taller than the average woman. That makes us men believe that as a man we should be taller than the woman we pair with, otherwise we're too small. Conversely women tend to think they should be shorter than their man, or they feel too big. I think it has more to do with norms in general.