I don't exactly want the science on how portals work (though I wouldn't complain if someone did), but I suppose they would work lore-wise. Portals can often be found behind long forgotten entrances, such as behind the doors of an ancient mausoleum.
The portals themselves are invisible and often unusable, meaning most people will pass through one without knowing that they did. The portals usually are only able to be used by things from the fantasy world, because what the portals try to do is to separate things from the fantasy world and Earth. But because the majority of the things from the fantasy world's ancestors come from Earth, the portals get confused on whether they belong on Earth or the other place, so they end up just being able to transverse them without any problem. The only exception are semi-intelligent gremlins, which can teleport between the worlds without the need for portals.
The biggest issue I'm facing is exactly how things from Earth would even get to the fantasy world. For instance, an elf can travel between the two, because its ancestors came from Earth and and it was born in the fantasy world. But how would said ancestors even get to the fantasy world? I was thinking maybe things that touch the gremlins while they're teleporting back to the fantasy world teleport along with them, but this is kind of underwhelming.
Note: Gremlins are the only things that are proven to not come from Earth. And I'm okay with any answer, as you can guess I'm pretty early in the process so currently nothing would really break the rules in my universe because I haven't really established any.