The Brain is a very fascinating organ. There are many aspects of it we simply do not know. It's shape is one of them.
Perhaps a look at embryonic Brain formation would provide us with clues as to why brains are the way they are. Looking at early cell creation and sequences in our egg cells after conception usually tells us clues about genetic instructions required to layer on top of each other to develop organs into what they are today.
This technique can be used to see that we structurally evolved from fish, with early embryos looking 'tadpole-like' and having tails - to be lost later as genetic instructions evolved later to lose our tails.
The Brain forms in our 3rd week of gestation with 3 main cells forming the forebrain, midbrain, and hindbrain. These are expanded to form the main axis, which slowly grow in size and length. The brain is very early split into hemispheres, although it is worth noting at this stage the Brain appears to be linear in nature, curling up on itself to form the classic Brain shape at 3 months.
From four months, folds in the brain become evident as cells are continuously added. No-one really knows why but there are several theories why these folds are evident in all animals. One theory is that in order for parts of the brain to communicate, increased surface area and internal cell structure is important and need to be closely aligned. Cell formation then follows a folded wrinkled structure as all parts of the brain need to be kept close to the central stem, that supplies blood and enables communication with other organs.
This may be a factor preventing the brain from being 'square' often in nature, as it may interfere with the ability for the parts of the square far from the middle, the corners, from being able to access supply and communication structures at its centre.
We also know that many animals develop protective structures around their brains to protect its delicate cell structure, which is easily damaged by trauma and sudden movement. Squares have weak corners which given any impact would easily dislodge.
So to summarise:
- Brains are actually formed as linear in early development based on a central stem
- complexity is added as cells grow to form the major components of brains
- as time continues they scrunch up into wrinkled spheres, for a yet-unknown reason but likely associated with surface area and accessibility to the stem for supply and communication.
- brains require protection. Spheres are best as well for this.
So can an organism evolve a square brain? There needs to be a major advantage for this to occur - perhaps it lives underwater in a static environment, such that structural protection is not an issue, and its development creates more rigid stems that develop corners early in gestational formation due to mutation. Rigid external structures could determine advantages of this, or an internal square structure (sensory organ perhaps) that the brain stem must wrap around during development.