You can imagine worst case scenarios about anyone and stereotyping them accordingly is harmful. That stereotype and the fear surrounding is something you are cultivating and that's what men are often objecting to when they say, "not all men". If a woman looks at me and determines I'm a threat based on my gender alone, that's bigotry...definitively speaking.

Is what you're saying is that the bias, harmful as it may be, is justified? If that's how you think, then there's really nothing more discuss. Feeling justified, you will continue to make harmful generalizations about men and there is nothing I can say to deter you.