Well, Homo floresiensis were tiny, measuring only about 3 feet in height. They were not midgets or dwarfs, simply a much smaller version of Homo sapiens. In an alternate universe they very well have been the norm for our planet, but I think much of our size has to do with competing against predatory animals, which tend to be larger than their prey. If you had a non-meat eating race with only limited exposure to predators it could be smaller since it doesn't need to out-chase or fight for food, or defend itself.