Botulism comes to mind. It's a particularly terrible disease that almost no humans ever get (though there is a risk from spoiled food; this is why you shouldn't eat canned food from a bulging can), but dormant spores from it are common, especially in honey, where the extremely low-water conditions keep them dormant. (this is why you're told to never give honey to babies, by the way.) These spores are quickly killed by the human immune system in children and adults (but as mentioned before, not infants), but your fantasy people could very well have no immune response to it at all. As far as I know person-to-person transmission is uncommon if it exists at all; it's only really carried in foods, especially preserved foods like honey and canned goods (it's resistant to most sterilisation processes, and it's not enough of a problem in humans or any Earth animals to go to more extreme measures), but it's not too hard to imagine an explorer to another world brings a jar of honey or some tins of beans, one of which may contain botulinum spores.
An alternative is having some perfectly benign bacterium to humans turn out to be deadly to elves or whatever species your fantasy world is inhabited by. There are a ton of bacteria that live inside the human gastrointestinal tract, including in the mouth, where they could easily be spread by a human explorer kissing a particularly attractive orc that they fall for. (or by sharing a drink in a pub, but that's got less of a story behind it!).
Harmless microbes causing havoc when they jump to a different species is a well-known thing. Swine flu is a well-known recent case of this; it's harmless to pigs, but when a mutated strain of it made the jump to humans, it caused all sorts of problems. I believe it's generally thought that most diseases that run the risk of killing the person with the disease originated this way, as it doesn't make much sense evolutionarily for a parasite to actually kill its host if it can't survive outside of a host. But if a virus that evolved as a mild disease in pigs suddenly finds itself in a human, well, it doesn't know it's in a human now, so it still does the stuff that is mildly irritating but not harmful to pigs, which it turns out is very harmful to humans, and that's how you get swine flu.
I think COVID-19 is a case of this too, but I don't know what particular animal it made the jump from or even if that's actually true anyway.