You could certainly have a world in which there were no materials suitable for building a crossbow, although this might make regular bows and arrows also impossible to build, and might require significant deviations from a typical medieval fantasy setting.
A Lack Of Adequate Crossbow Making Materials
For example, you could have a world in which there were no good supplies of the kind of wood suitable for making a crossbow (e.g. consider the kinds of wood available in Hawaii, or the Sahara desert, or Death Valley, or Greenland, or "Waterworld", or an everglades scale swamp with no big trees that make suitable wood). In a world with only bamboo, or only shrubby small and thin woods, if any, or no wood at all, it would be very hard to build a crossbow.
(This is probably the best "hard science" solution.)
In a close variant of this possibility, it would theoretically be possible to make a crossbow, but the materials would be so expensive, that it would be far cheaper to train people to use other kinds of projectiles.
Or, maybe there is an absence of the kind of plant or animal fibers that are suitable to stretch for enough pounds of draw to make a crossbow or bow viable as a weapon.
In variants of that possibility, maybe it is so hard to stretch the bowstring fiber relative to the draw strength involved that no one is strong enough to pull it and no one has figured out how to use simple machines to draw the string, or maybe the only fabric that is suitable for use as a bow string can only be used once after which it loses all elasticity.
An Environment Hostile To Inventing A Crossbow
You could also have a world in which the utility of a crossbow was so low that nobody would bother to invent them.
For example, imagine a place that is almost always fogged over (e.g. in a mountain top cloud bank or an exaggerated version of London), or a place so dark that only firefly-like glowing plants provide illumination for very short ranges. In that environment, long range weapons would be useless.
Or, maybe the local atmosphere is so thick to the point of being almost gelatinous, or has such a strong gravitational field, that projectiles swiftly slow down even when propelled powerfully. Of course, this would profoundly influence a whole host of elements of your world.
A Lack Of Demand In A Militarized World
As implied in the question and noted in another answer, for many purposes, a crossbow is inferior to a bow in the hands of a skilled practitioner.
A crossbow can still have utility if you have lots of untrained peasants and you want them to fight with projectile weapons. But, if your world is full of places like legendary Sparta, where every able bodied person was trained for war from early childhood, the crossbow would fill a need that didn't exist, and no one would invent one.
Alternately, there might be an alternative that is just better in all respects than a crossbow that is easily to use. For example, maybe someone invents a "tandem bow" that requires one unskilled strong person to draw and notch, and another skilled person who need not be physically strong to aim, and each sub-task is much easier to teach than having one person do everything, that outperforms a crossbow in all respects.
There could also be countermeasures easily available that makes a crossbow an invention that isn't worth the trouble.
Maybe your world has a special kind of cork that is easily and cheaply produced and easily applied to armor, that functions like Kelvar and easily absorbs the blow of arrows and crossbow bolts distributing the energy harmlessly across the entire body of the person wearing the armor.
Or, maybe your world has extremely fast, easily trained and loyal companion birds to soldiers who are capable of rushing out and grabbing arrows and crossbow bolts with their beaks in the blink of an eye.
Or, maybe it has giant companion frogs who sit on a soldier's shoulders who can shoot out their tongue and grab arrows or bolts out of the air.