One very simple way is, nonlethal weaponry. Actually it is easier to develop a reputation as something to be avoided if the ones who have to avoid you survive. Think of skunks. Does anybody bother them? No. Humans could develop powerful repellents that would drive the predatory, belligerent megafauna away without killing them.
The simplest way to prevent humans from driving megafauna to extinction is geography. Put some kind of big, fat wall* between the humans and the giant beasts, and have humans just not really that crazy to go there. Suppose that on the American continent, there was a tree that killed off humans by poison. All the Megafauna became immune. Humans have not as yet felt the need to conquer America. Since the explorers never got back, maybe they don't even know it exists.
*Obviously, this would not be a literal wall, but a natural barrier, unless there was a super civilization that wanted to preserve the megafauna. In that case, it's still all about the barrier, but not really. Then we need to know why the megafauna is desired.