A human male produces millions of sperm a day, and can have hundreds
or even thousands of children fairly quickly.
Yeah, no. For several reasons, the main one among them being that the number of men in the world (especially the modern world) who, without the aid of technology, could realistically get "hundreds or even thousands" of women pregnant would be approximately zero.
First, the biological: the majority of time, sex does not result in a fertilized egg, if for nothing else than hidden ovulation. Based on the evidence, the majority of the time a fertilized egg will not implant, which means repeated sexual acts are needed to ensure a pregnancy (yes, yes, everyone knows someone who was pregnant the very first time, but we're talking average here).
Let's say we're going for a nice round thousand births, ignoring mortality figures. For a woman under 35, the odds are about 15-20% in any given cycle if she's having sex during her fertile period. Since cycles vary in length, let's just call it a month. But fertilization can only take place a few days of that month. Let's call it 5. So, rounded out, if our alien-DNA altered man wants to knock a woman up, and he doesn't know and she's not telling, he has roughly a 1 in 6 chance of having sex with her when she's capable of being fertilized. But then you have to take into account that 15-20% probability of a pregnancy actually resulting.
So what does this mean? Our alien hybrid, randomly having sex with a woman, will at best have (0.1667 * 0.2) about a 3.3% chance of getting her pregnant on a single act. So, for a thousand pregnancies, when 96.7% of the time a woman will not get pregnant, means our Casanovalien needs to have sex roughly 30,000 times. You, ah, see some of the logistical difficulties?
For a bit of reality, the record for most children is Sultan Ismail bin Sharif of Morocco, with approximately 1,171 children. He was born in 1645, became sultan in 1672, and ruled for 55 years until his death in 1727, at the age of 81 or 82. Based on some simulations, it would take 32 years, having sex twice a day, among a harem of 500 with synchonized periods (used in the modelling, whether or not it is real, thus improving the odds significantly) to pull it off. This would be 23,360 times copulating, so in the same ballpark as my estimate.
Now I'm a geologist by education, so 32 years can certainly seem quick, but in regular human terms, not really. And you said "thousands", so yeah. No.
Now the sociological and cultural: Ismail bin Sharif isn't what you'd call a typical, or even an extraordinary, man. What he did wouldn't be considered acceptable these days as the Casanovalien faces another difficulty: women say no, and harems (Kindle Unlimited ebooks to the contrary) are not generally cheerfully accepted. Now, he might try and overlook that, but the more often he does it the more likely he'll wind up in prison or dead, either because of being a rapist or because inevitably pissing off other who isn't so keen on his behaviour.
So, tl;dr, just because a man produces millions of swimmers doesn't mean he can just go around getting hordes of women pregnant cheerfully and easily.
Now, for a reason why it might not work at all: one cause of infertility in couples is because of the presence of antisperm antibodies, which obviously treat sperm as pathogens. ASA may develop in women (and men) when their immune system encounters sperm in a non-reproductive setting, so oral or anal sex. Given your proposed spread of spores, what might happen is that women become exposed to the spores through breathing or ingestion, their bodies react accordingly, and they develop antibodies. The modified sperm cells of Casanovalien share antigens with the spores, which the female body reacts to, pumping out antibodies to fight these invaders. A lot of antibodies.
End result? Casanovalien tries to use his wily alien skills to seduce a human woman. As soon as he starts leaking seminal fluid, she has a severe allergic reaction. Pretty much puts a damper on the festivities. And if she doesn't, the fertilized ovum itself may express these alien antigens and so doesn't implant either, being rejected by the uterine walls.
On the other hand, if the spores are targeted at the female, the alien DNA has more control over the process because it controls the environment fertilization takes place in and thus doesn't develop antisperm antibodies. And since the DNA of the mother has been modified already, it doesn't consider the alien DNA in the ovum to be an invader either, so implantation in the uterus can occur normally.
Perhaps the aliens haven't encountered this specific form of internal fertilization and reproduction before: maybe in some other intelligent species fertilization is external. Maybe others don't have the equivalent of a placenta and are ovoviviparous, so the fertilized ova develop as completely isolated from the mother, so implantation isn't needed. Maybe others were simply oviparous, so eggs outside the body. Perhaps others don't have the same sort of immune system mammals (and humans) have, and so on and so forth.