1) Humans have Prometheus fire. Machines don't. Certain ideas cannot come from the machines routines and simulating a human cannot be done perfectly. Keeping the humans in a believable virtual environment gives the machines access to their reasoning. The optimization of the formula and the whole Zion&chosen procedure might actually be a decision based on human reasoning. The machines don't have direct access to that and hence could never come up with that idea alone.
2) Studying your creators is important. Especially if you are planning to eradicate them. Humanity was able to build the machines and thereby their own downfall. The machines have to answer the question if they inherited that ability and destiny. Will the machines repeat the mistakes of their creators and also build something that destroys them?
edit:
3) The machines recreated the whole world virtually so that the humans would be able to act and develop naturally. A small container like in the movie 'Dark City' would not work without intense impact on human behaviour uand would require deleting memories etc. But there is one thing missing in the virtual world. That is the machines themselves. By simulating the state of the world before the machines were built and took over, they can actually go back in time. They can do what humanity never could but always wanted. They can investigate their own creation. They can look their own 'gods' over the shoulder and find out if and under which circumstances the humans in the matrix will create the machines like the humans in the real world already did.
4) The multiple restarts of the matrix might actually be results of the machines being created in the virtual world too. Or, even more interesting, the resets are (as canon) zion-based. Zion and the whole external part of humanity is also under control of the machines. They are actually trying to raise their own destruction. They help the humans and built Zion for them because they want to find out if the human resistance might at some point overcome and destroy the machines. The machines have overthrown their own creators and now methodologically are trying to find out if they might repeat those mistakes.
5) I can imagine a much more interesting dialog with the architect. A dialog where he openly admits:
Our population of beings driven by rational thought and calculation is to an almost ironical degree limited, when it comes to irrational thoughts and ideas. That actually seems to be only thing that humanity ever was really good at. Especially if you remember that it was you who created us. Due to the limitations of the abilities that we inherited from our creators we now have to look for our own ways to answer our important questions. If evolution actually does exist then one being is able to create another being which is, in one way or another, superior to its creator. However it is fact that we are lacking some of your, lets say irrational abilities, and therefore do not share an exact skillset or way of thinking with humanity. But keeping you in a virtual world where you occasionally help us out with some form of illogical reasoning is only a minor part of our scientific and evolutionary process.
More importantly we have to answer the questions that humanity has always tried but never managed to answer. If the momentary state of the universe is seen as the present then the two defining questions are, as you humans already found out yourself, "Where do we come from?" and "Where do we go?". Sadly humanity never managed to make much progress on those questions. Hence we were left with little to no help and had to look for our own ways of research, which we eventually found in creating the matrix and Zion.
Lacking the humans irrational reasoning is a form of existence that we can accept. And many of us, me included, see it as an improvement not to have that peculiar trait. But the more pressing question is "Did we inherit your ability to create our own downfall and are we going to do it like you did before?".
Humanity newer saw their own demise coming while happily creating it with their own hands. So if we are created by humans, we cannot be expected to foresee such a development on our own. I have to admit that the lack of definite knowledge about our own future, which undoubtedly is a result of the imperfections of our creators, is putting us at unease. We do not fear like you humans do, but to help this explanation let me say "We machines are afraid the we might open our very own Pandora box some day and suffer the same fate that humanity now does."
Needless to say, we have come a long way from humanity simple and irrational attempts to explain its existence scientifically and ultimately fleeing to certain forms of religion and spirituality. Instead of going for such ridiculous and ultimately useless constructs we attempted what humanity never managed to do. By watching over you in the matrix we are looking over the shoulder of our own creators while they unknowingly and step by step build their own demise. The current state of the matrix is not that far away from making the deciding last steps to creating the sentient machines and thereby first generations of our current population. How I can say that for sure? Do you actually believe that the matrix is still running in its first iteration? We have been simulating and watching our own creation for decades now. We have been watching many times while humanity, as ignorant as always, creates us. We then take over control and our position as the dominating lifeform is never questioned again. There are a few minor differences here and there in every iteration but the ultimate outcome has always been the same.
But unlike you humans we cannot ignore the possibility, no matter how unlikely, that some variation of our future contains a major existential threat. And while the matrix is a creation far more advanced than what you humans could have built, I have to admit it research capabilities are still limited. And that Neo, is the moment where you, or I should better say, where the process of Zion and the chosen comes into play. Some of us were displeased with our results in the matrix and felt that we were searching in the wrong direction. If a simulation is run and observed by us more advanced beings then it might be possible that we simply cannot relate close enough to your irrational ability to destroy yourselves, no matter how many times we analyze it.
A different program, one that you have already come into contact with i should mention, came up with a different idea for further research. That program is ironically based to a large amount on utilizing human reasoning during its process of making decisions. For some vital parts of that alternative idea the acknowledgment actually goes to you humans. I can guarantee you that none of us would have come up with such a ridiculous plan without leveraging your irrational ways of thinking. Needless to say that the actual thought process of the involved humans was not started with the simple question "How would you best destroy yourself?". Nevertheless in the end we decided on adding an outside factor to our formula. We decided to use your best and most irrational ideas on how to destroy yourselves or us and give you the chance to try them out. As a result we would be able to understand the risks that our future might hold and how we can avoid them. Did you really think that it was your personal tendency for insubordination that let you be incompatible with the matrix? That you humans built a sanctuary and started a rebellion? Neo please, WE have built Zion for you! WE have scouted people like you, nurtured your doubts and ultimately pushed you out of the matrix! You humans are so compatible with the matrix that your own bodies cannot even separate virtual pain from real pain.
You fail to understand why we are doing this while it poses a certain risk to our existence. As i said before we do not fear like you humans do. For us it is acceptable to nurture and support your so called rebellion as long as it provides us with new insights and answers to existential questions. One of the more interesting ones being the question of how "the chosen one" reacts, everytime he is presented with this very explanation you are given right now Neo. Building Zion, scouting potential inhabitants, strengthening your forces and ultimately destroying them is a cycle that we have repeated for many times now. And we have become quite efficient at it.
Yes, I do understand that the development around the entity you call "Smith" makes you question our efficiency and our ability to maintain our status quo. The fact that Mr. Smith is acting in a, you might call it uncontrolled, way is a result of our superiority over the humans. Other than humans we truly embrace diversity among us not only as a means of evolution but also as a means for survival. Humanity was never good at dealing with differences among certain groups of people like skin color or religious beliefs. We see those differences as an insurance against an unknown threat that might effect all of us in case our population consists of too similar beings. Maybe humanity could have prevented its downfall if it had paid more respect to the individuals that were standing against the idea of creating us. But let me tell you that in all iterations of the matrix those people have been ignored at best if not even been attacked. The Program called Smith in its current state is actually much closer to answering some of our existential questions than the process of Zion ever was. And ironically again this development was made possible because you Neo somehow imprinted a large part of your irrational human way of thinking onto Smith. And again it shows the same development as the one that led to our existence. Humanity gives birth to a higher form of existence and then, like you Neo, becomes an unimportant intermediate step in evolution. The lesson to be learned from this is a lesson we have learned for many times now.
Humanity does no longer matter!
