How could a civilization create a dangerous autonomous mechanical ecosystem that will not be overly dangerous to its creators?
Militaries are expensive; military research doubly so. So what's a small colony from a small civilization supposed to do to defend itself? It needs all the resources it can get to bootstrap its own industrial base, so it can't create much more than a token defensive force unless it wants to cripple its own growth. Nanofabrication and robotic mining can help, but everything you produce is going to be inferior to the high-end military technology of the people who might invade you.
So why not evolve new threats? Organisms can reproduce and evolve without your direct involvement, crating new and completely unexpected threats. It's hard to engage a conventional military when strange monsters are swarming your position. Biological organisms probably won't do: they're too vulnerable and unsuited for war. Claws and teeth have limited use against combat armor. You need something that can't be easily handled with fire or chemical weapons, and can do real damage.
So let's make the organisms artificial. A mechanical ecosystem of autonomous (https://en.wikipedia.org/wiki/Self-replicating_machine) robots that uses evolutionary robotics to become ever-better killing machines.
Now we're getting somewhere: armor-piercing guns and diamond-toothed saws beat keratin shards and teeth any day, and robots evolve much faster and with more purpose than organic life ever could. Programmed evolutionary imperatives and and behaviors mean that this 'mechology' can devote itself to the destruction of intruders without endangering the colonists or engaging in wasteful perpetual war against itself.
A simple ecology will suffice to start. Plant-equivalent robots mine for materials and convert them into refined forms and power sources. Animal-equivalent robots prey on them and each other. Their AIs are simple neural networks. Their genetically-specified controllers ensure that they are evaluated based on military suitability, and ensure that the least fit don't get to breed. All are designed with an artificial version of chromosomes which are merged and mutated during reproduction. The resulting template is then assembled into a robot and set loose. All elements of the robots are subject to mutation, to ensure that preconceived notions don't limit their evolution too much. The only thing that's immutable is instructions not to attack colonists or their infrastructure.
Now comes the tough part: making sure it stays safe to you. Sure, you can enforce behaviors and try to direct the evolution as best you can, but evolution has a nasty habit of doing the truly unexpected, so sooner or later you'll discover that your safeguards aren't as safe as you think. And while an uncontrolled militarized mechology that indiscriminately kills everything in sight is great if you want to make an uninhabitable death world, it's significantly less great if you're trying to live there.
So what can you do to prevent this dangerous mechanical ecosystem from evolving out of your control and becoming a danger to you? How can you create safeguards that are strong enough to not be subverted while still not being intrusive enough to force your new army to evolve predictable traits and tactics? Or is there no way to reduce the risk to an acceptable level?