A strict (samurai-like?) code of honour which requires one-to-one combat and forbids shooting at humans, and the only non-human adversaries to defend against being giant-sized monsters would, in my opinion, be the best explanation.
Heavy, really heavy, wind might be a reason. As in, a planet where what we call a storm is the normal condition on a sunny day. Larger calibers would due to their mass be less affected by side wind.
On the other hand, of course, on such a world, the air would be naturally filled with deadly "projectiles" all day, every day, and nobody would need weapons to kill someone. Just wait and watch.
The inability to build sufficiently high-pressure barrels/chambers would also provide a reason, but this seems rather unlikely.
One could think about stealth being a major factor as well, firearms for some reason being used exclusively with suppressors. However, using a suppressor only really makes sense if the projectile is subsonic. Which means that in order to transmit the required energy, you need a projectile with higher mass. The length of a projectile is limited by practical means, and a mostly cylinder-shaped object's mass only grows linearly with its length (as opposed to quadratically to its diameter). Thus, calibers would necessarily be larger.
This would, however, not explain why small calibers don't exist at all.
Within reasonable bounds, and disregarding stealth, small calibers are always better. The projectile's momentum and thus the gun's recoil grows linearly with either speed or mass. However, the projectile's kinetic energy grows linearly with its mass but quadratically with its velocity.
Naively thinking, a smaller caliber would magically make more efficient use of the charge's energy by accelerating the projectile to a higher velocity. That is indeed the case with something like a bow, crossbow, or a railgun. Not so, however, with firearms. The reason is simple: Although the projectile's mass goes down with its caliber, also does the surface of its cross-section at the same rate, and thus the force acting on it (assuming equal pressure in the chamber). With e.g. a bow, the force indeed stays the same whether a stronger, heavier arrow is shot or not. Thus, physics require that the lighter arrow gains more velocity (and thus kinetic energy), and less energy is lost to the environment.
Notwithstanding the fallacy in the previous paragraph, a smaller caliber can easily be accelerated to a much higher velocity given a charge that creates, and a chamber that withstands, a higher pressure. The charge is no issue, the limiting factor is the chamber.
This provides higher kinetic energy with the same or lower recoil, and better ballistic performace (more velocity at distance, straighter trajectory).
For a real-world example, .440 Cor-Bon performs better than .50AE with much lower recoil. Similarly, .416 Barrett performs superior to .50 BMG (not in terms of muzzle energy, but at a realistic shooting distance), although the projectile only has slightly above half as much mass, and the charge is noticeably smaller, too.