What makes an animal suitable for domestication? I know it has something to do with the danger to productivity ratio. But this is not helpful for fantasy animals in the worlds we build. What factors or ratios can I use to help determine what fantasy animals are suitable for domestication?
So, putting together all the excellent points made in other answers, I would say that any animal can be tamed and used in a symbiotic relationship with people. It is more likely to happen if it is harmless enough that the occasional bad behavior (panic or aggression related) can be tolerated. It might be that the animal is docile by nature, or it might be that it is small enough that it can't cause much damage.
Which animal will be domesticated, shaped over the generations, is decided by practical considerations: of how much value is a relationship with this species? Thus you might have things that are cute and cuddly "straight out of the box" being domesticated as companions. Animals with a strong sense of hierarchy (like canines) would be bred to be tools and helpmates. Food animals would be bred for their flesh and probably temperament. Each species would be bred to enhance and improve those characteristics which make it useful.
Why have we never domesticated tigers? Because they are dangerous and even the most gently raised of them has the potential for turned on its handlers. But what if the tiger was able to teleport not only itself but also anyone holding on to it. Would we have domesticated them? My guess would be yes. Even if their temperaments never improved much the value of instantaneous travel would be worth the occasional mauling.
Now, as an aside, that doesn't explain why the Egyptians put so much work into cats, but maybe, since all "house cats" are descended from a very small population in a very specific area (http://archaeology.about.com/od/domestications/qt/cat.htm) it might be that someone found a particular strain of feline that lent itself to domestication and they just took advantage of it.
If I Google for "what animals can be domesticated," the 3rd link is quite helpful, "Why Can't All Animals Be Domesticated? -LiveScience" (all of the first links are very helpful to you, but that one seems most well addressed to the exact wording of your question)
From that article, they identify 6 major characteristics of a domesticatable animal:
- Cannot be picky eaters
- Reach maturity quickly
- Willing to breed in captivity
- Docile by nature
- Cannot have a strong tendency to panic and flee
- Conform to a social hierarchy
There are counterarguments (the domestication of dogs from wolves lead some to question if "docile by nature" is a valid criteria), but it seems to be a reasonable start.
There is another theory going that the act of domestication is a particular neural crest deficiency that eventually occurs through the mutation process. Apparently it was noted since Darwin's time that domesticated mammals all show phenotypic characteristics of floppy ears, smaller teeth, and shorter snouts. It is now known that these traits occur as a result of changes in how the neural crest develops as a fetus. These changes also have brain effects, such as decreasing fight or flight responsiveness. Any fantasy creature which cannot, for some reason, undergo such a change would indicate that that species is harder to domesticate.
Any animal can be domesticated. All it takes is time... 10s of thousands of years.
What really drives domestication is a symbiotic relationship between the animal and human. House pets get shelter, food, water, and affection. However, they provide services for humans. For instance dogs bark at strangers to provide security. Cats kill rodents that might steal our food. Horses, do manual labor.
A fantasy animal would be no different. Obviously, humans would provide food, water, and shelter. All you have to do is think of what that fantasy animal would provide for humans. Pet gorgon? Sure, if you need lots of statues.
It depends on your threshold for suitable in regards to domestication.
For possible, the only things needed are,
Some way to gain substantial leverage over the other creature in regards to survival (food, shelter, etc) and reproduction.
The ability to conform to some kind of social hierarchy where you the "domesticator" have the ability to be the gatekeeper of their needs to live and reproduce, be that by force or by training or bonding.
That's it as far as domestication requirements goes. Now, if you want a good pet, that's an entirely different question :) And quite a bit more subjective...