This is simple and complex question, at the same time.
- but overall, what you describe as small small aliens, keep together is ordinary multicellular live
Environment and lack of food will kill them efficiently, without limitations, but if you wish just limit colonies by size.
In case queen, which is only one who reproduces and because of that she is bottleneck for growth(eventually at some point, speed of how much units it may physically make, will be equal rate of death), is out of equation by 'cells' simply are able to endlessly reproduce , but it still may be a moderator of that collective consciousness, with limited range by distance as example, size of living place, or by size of colony literary. Bees not only enjoys queen smell in general - but they transmitting smell by touching each other and queen. Queen have escort they touch queen and redistribute she's smell among others, escort also rotates one bee come other go. And such way they are able to determine size of colony, which triggers bees to make places for new queens, divide family and so on.
- let say group have one source of tokens of some sort. Size of that token may be known, or may be not known. Aliens have believe, law etc - which says you may keep that token for year and gods(who else) are happy if you reproduce in that year, but after year you have give that token for any random member (as they are dumb, they can't distinguish members and falsify results - it will be some random member). If it needs 100 of aliens to build that source and it is limited in production rate - this will set some population limit.
Swarm logic algorithms working in determination of sizes, no one have to be smart there for doing it.
So any cell outside the reach will let's say loose will to reproduce, and probably this will itself and ability to, have to be triggered by belonging to that collective consciousness, and presence or queen or such.
In real live, with bee's as example, situation is reversed, they all have ability to reproduce(except drones, which usually is small portion of them, for short time), but they do not do, until queen not dies or flies out from hive.
Queen is produced not as ordinary unit, and it needs collective work of may bees, it needs more resources from bees, comparing to ordinary units. That means just single bee is not capable to make new hive.
It not only needs more resources, but they just not capable to make it to happen without cooperation of many bees. Just because few bees not capable to build place where queen have to be born
- actually they can't build anything without cooperation - that's long story. But as example how they determine vertical lines to build honeycomb(they have to be vertical because honey is fluid and if they do that horizontally it will flow down, and they have to build twice more to store same amount of honey) they use them self's swarm as level, they making hanging group of them self, and that determines vertical direction.
To make system more robust, they have to be depended on each other - it's made by specializing types of work one units can do or by their size and such, or by age, or just because they really can't do something complex alone from things that are vital for them. As example bee do different critical tasks at different ages - and that helps them be small and dumb, they are just too dumb to be able do different tasks in same time and it makes them be more efficient and allows to learn stuff first etc etc.
- I would say like humans, same stuff here
In real live with bees there is nothing that happens just because or for fun - literary anything is done because of some important reason, and there is a lot of such reasons, but goal of them is one - survive and prosper.
All that reasons forming system of equation which allows them to stay one hive, work as hive. But this is also weak point in their adaptation mechanism, it's very hard to solve that system of equation in another and same time useful way.
- single unit important for coordination, it do not makes orders, it is just signal : All Quiet in Baghdad
- vital living structures, which can be build together only
- reproduction rate of one unit may and can be regulated according food availability and size of swarm or swarm needs, system needs
- differentiate specializations
- fight other colonies (star systems planet etc)
- if no wish to fight, just feel boundary of other hives, it will be border of expansion.
as note any multicellular organism is perfect example of system which regulates itselfs, it's just a question of implementations of negative feedback to the system and it's members. And being collective consciousness does not mean absence of such regulation. And it's not important made this regulation naturally by evolution or vitro. As Ultimate factor for resticting size is just lack of energy and matter for growth.
Determining size of hive does not need one center-queen-etc, it can be done p2p style.
But if they can reproduce until natural boundary, there is no reason, at least at space stage, for them to limit themselves. On planet stage may be, it may help selection work faster, so for speedup reasons, but space stage not really.
Make hive be not a one big homogeneous structure, but collective consciousness from collective consciousness - binary tree, grid, p2p - it will help to optimize structure as whole