How to keep magic separate from science?
"Science" includes many fields of study. How to make it such that magic is not one of them? How to make magic resistant to study by experimentation?
Keep the "hidden" effects of magic measurable only by magic.
For example: Where does the energy for the magic come from? How is it conveyed from the source to the target? A magician could answer these questions, based on direct observation/perception. Yet, to any non-practitioner there is no evidence besides the end effect.
It will be a necessity of the practitioners to make formal studies of their craft. It's how they better themselves and share their knowledge. The very nature of a spell implies that a magical force has been understood enough to be captured by a repeatable ritual.
However, explaining this to anyone that isn't a practitioner doesn't amount to much. They can see something happening, but that have only the practitioner's word on how it happens.
Science must be "logical" and "rational" and magic not. A person with a rational mindset should not look to magic as the first answer. Why?
Human magic should be limited to a small number of uses, but presumed to be capable of more. Perhaps the magic can only be used to produce scientifically impossible physical effects: levitation, teleportation, spontaneous energy/heat/light, etc. Superstition would then have people believe magic can: bend your will, make you sick, cure you disease, etc.
Why? Because that's how people work in the real world. Superstitions abound in every part of Earth. Demons, bad spirits, witches, angels, gods, ancestral spirits, have all been credited in ways that a rational, educated mind would not.
The field of human magic should be full of disagreement. Perhaps they disagree on the origin of their magical powers, or methods. Maybe multiple methods exist to execute the same effect. The lack of consistency will make it even more difficult for scientific scrutiny. Imagine there are 10 known spells that do the same thing (let's say lift a boulder), and that it's truly random which spells and how many will work for a given mage. Then add to it that even mages casting the "same" spell will have slight variations based on personal style, and it's a quantification nightmare.
And how and why would the scientific method ever have developed if a major part of reality was resistant to it?
Science can still classify phenomena and experiment upon magic, but can't reproduce it. For instance, there's no reason science wouldn't try to figure out how a massive, wingless dragon can fly. But eventually, they're only going to be able to record observations. Science would have no way of explaining the hows and the whys, and wouldn't be able to develop applications for their knowledge.
I left the nature of magic open because that's an inseparable part of the question. What type of magic could fit this world? Is there any magic system that can make sense here?
I think a magic system that is based primarily on violating the physical sciences would work very well. Lift things up, pull things down, fly, transmute, teleport, areas of silence, casting light or darkness, invisibility, and pretty much anything a Bender can do in Avatar/Korra in regards to elemental manipulation. The magic should generally not be able to affect biological or psychological processes (at least not any more often than necessary to seemingly validate superstitions). This precludes increased longevity, healing, cursing, mind control, etc.
The "strength" of magic should be, to the outside observer, entirely random. Some purely fictional quality or qualities should be responsible for magical ability. "Spirit", if you will. There should be no correlation between race, body type, size, physical strength/speed, general appearance, intelligence, personality, family/genetics, etc. and the strength of Spirit.
You can take this further, and have the number of mages also be random, not dependent on population size or geographic location. There could be no correlation between the number of mages from one generation to the next.
Even creatures that exhibit magical abilities can have randomness applied. Let's use the wingless dragons as an example again. Their max speed, maneuverability, acceleration, max flight height, etc. should have no correlation with their size, weight, number of spikes, color, age, etc.
There are any other number of stipulations you could put into place, that would make magic less appealing than science for explanations. A remote tribe could have a very unusual power, but they can only demonstrate that power in their tiny geographical area. There could be some areas/creatures that actively interfere with some technologies (or even electricity), that make it harder or impossible to take measurements. Certain elements or objects could be immune to magic for no apparent or explainable reason.
In the end, science would need only be aware that magic exists, is evidently controlled in some way by certain animals and humans, and that it can do things science can't explain. But, by its nature, magic would be unpredictable when trying to achieve consistent results. Relying on magic may work in the short term if the resources are available, but it'd be up to science to make the real achievements.
Personally, I think such magic would also spur science, as it would try to "compete" with magic.
So, your village has a guy that can start fires by snapping his fingers? Well, our village invented matches and now all of us can start fires whenever we want.
So, this 100-pound man can lift a 5,000 lb boulder with a wave of his hand? Well, this machine can lift 20,000 lbs with the push of a button, and we can build as many machines as we want.
So, you've figured out a way to train a certain dragon breed for riding? Well, we invented the airplane and we don't need animals to fly.
And so on.