Something to think about here. It took 70 years from scientists officially recognizing the giant panda as a species before we managed to capture a live one.
Large parts of the world are still very hard for humans to thoroughly explore and catalog, even outside of how little we know about the ocean.
Yeah, most of Europe is explored and we're pretty sure barring actual magic IRL that things like wolpertingers, tatzelwurms, and wyverns don't exist. Similar for most of the US and large parts of Japan and China.
The thing is though, Earth is huge, and human population is actually rather concentrated. A vast majority of the surface of the earth is not being observed by humans at all at any given point in time. Every continent and a vast majority of countries have some region that is mostly uninhabited on a regular basis, often a rather large one, and quite often we don't even have good satellite coverage in modern times either. Stories of yetis in the Himalayas may not be very likely to be true, but if they are it's not all that astonishing that we haven't actually found any if they're even remotely intelligent since most of the mountain range is unobserved most of the time. There are similarly huge swaths of the Amazon that haven't been touched by 'modern' man, and even some parts of Africa that are insanely difficult to get to even in modern times.
The other thing to think about is that humans are remarkably good at not acknowledging things that don't fit their world view. This was a huge issue during times of European colonial expansion, when it was quite often assumed that the natives, despite having lived there for their entire lives, could not be relied upon to list what animals lived in an area. That mentality is why it took so long for the okapi, pygmy hippopotamus, saola, and many many other animals to be 'discovered' by western science.
Together, I'd argue that these facts mean it's pretty likely that humans wouldn't even know some of these creatures exist even if we had expanded to cover the whole world.
However, let's ignore all that for a moment, because I think you're losing out on a very easy solution if you remove one of your restrictions. If we actually factor in those other sapient races and assume that they have much better access to magic than humans together with more ecologically minded outlooks, everything kind of solves itself.
Because humans have limited magical ability, any race that has significantly better magical ability than us is likely to wipe the floor with us in any war up until about WWII era technology is developed (and that assumes the other race doesn't develop technologically as fast or faster than us). The thing is, we would eventually try to expand enough that they would push back, hard, no matter how hands-off and isolationist they are, because they want to keep their own territory, and at that point we'll likely just stop expanding because we quite simply can't beat them. That means, at least until humans get tech that can out-match their magic (and even modern tech is just barely to the point that it could reliably deal with a platoon of level 10 wizards from D&D), large parts of the planet are effectively off limits to humans.