Based on knowing very little. I think there is an assumption that the fanatics in any culture are extensions of the everyday folk. Or, in reverse, if we see something evil in the fanatics, we conclude that the everyday folks are watered-down versions of the same evil. I think this is equal-opportunity for fanatics of all stripes.
Thus, we find people who devote their lives to a style of video game and we discover that many are misogynistic. Then we have fun trying to explain why they are this way, and assuming that people who devote a little free time to those games are also misogynistic, just in a milder way. And we want to clean up the lot of them.
We read extreme feminists, who seem to devote most of their waking hours to grievance, and find all sorts of man-hating evil in many of them. Then we have fun trying to explain why they are this way, and assuming that women who are just generally feminist as one of their collection of personal descriptors are also man-hating, just in a milder way.
Zip in "gamblers." Or "emergency preppers." Or "exercisers."
Not everyone who cares about germs is borderline phobic. Not every dieter is a potential anorexic.
But you can make a name for yourself politically if you keep trying to force normal people to defend extremists you can squint and associate them with. It doesn't matter if it's true. It only matters if you can make people believe its true.