There is an excellent answer to that question, written by J.B.S.Haldane as long ago as 1926 in his essay "On Being the Right Size". The Wikipedia entry has link to the full text of the essay. It is an interesting read, I highly recommend it.
He does not deal with limitations found in different sze brain (except in a short remark), but he does show that body plans of living organisms are tied quite closely to the size of the body. You can't alter the body size significantly, without having to adopt a radically different body plan.
For example, if a man were the size of a mouse:
But it is time that we pass to some of the advantages of size. One of
the most obvious is that it enables one to keep warm. All warmblooded
animals at rest lose the same amount of heat from a unit area of skin,
for which purpose they need a food-supply proportional to their
surface and not to their weight. Five thousand mice weigh as much as a
man. Their combined surface and food or oxygen consumption are about
seventeen times a man’s. In fact a mouse eats about one quarter its
own weight of food every day, which is mainly used in keeping it
warm.
So, a mouse-sized human would also need to spend inordinate amount of his day, well, eating. Does human digestion even support that? Not sure.
The human vision would be much worse than its current range as well, if the human was the size of mouse:
But if [the size of eye's rods and cones] were diminished and their
number increased we should see no better. For it is impossible to form
a definite image smaller than a wave-length of light. Hence a mouse’s
eye is not a small-scale model of a human eye. Its rods and cones are
not much smaller than ours, and therefore there are far fewer of them.
A mouse could not distinguish one human face from another six feet
away.
So, even without considering brain size implications, we can see that simply scaling humans (or indeed any living organisms) down (or up for that matter) is not a trivial thing at all. Even at the range humans occur naturally, very tall or very small people already usually have different proportions of the parts of their bodies, compared to the mean body size. Additionally, as far as I remember (don't have the quote on this) being very tall or very small carries it's own health problems as it is, and such people tend to have lower life expectancy.
Other portions of essay also discuss limitations of size in why a human couldn't be a giant (without changing the body plan a lot) or why insects can't be the size of a eg. a car.