It depends on the technologies they have. Different technologies lead to different solutions and different possibilities.
The knowing of nuances of those technologies is one of the problems in attempts to figure out the optimal fleet.
It starts from ISP of engines they use(or properties of FTL if there is one) to how efficiency weapon can be scaled (bang for energy buck and energy/resource buck to build the thing)
As an example Death Star was a crappy ship technology wise, build wise, but it had a good main weapon with good bang for energy buck.
But then we also have to understand the goals of the combat, because the destruction of the planets which DS was capable of, it is good and impressive, but what if no one gives a crap about planets as places to live and considering them just as a resource of matter for their constructions. With space habitats, it is a real alternative, and destructions of a planet will make them just happy, as the way which spends them time and efforts to lift the matter from the space body.
- yes, in your question they seem to care about the planet, but in which fashion they do that. They will prevent its destruction, prevent a landing of the troops to the planet, prevent nuclear/whatever bombardment, etc.
All those and other details are important to answer the question.
Another example - if a big ship can wipe a star system in dust in 5 seconds, jump in 5 seconds in any place with FTL and you choose the big one or the fleet of the millions of small ships which are capable(any of them) of destroying the big ship in 5 seconds - go with big one, it will destroy millions of star system before it will be destroyed itself.
But if there are countermeasures for FTL - then answer is not so easy, it depends on the efficiency of those countermeasures. If you can see the enemy in FTL, predict his destination, attack while in FTL then the picture is different(and btw similar to the noFTL setting).
If we take noFTL setting or one which is just similar, by the properties, to noFTL one, then distributed systems seems to be preferable than a single big ship.
Missiles are preferable(but it depends on engines, energy sources, how efficient they really are against the big ship etc)
The distribution by size in the fleet depends on goals/task they have to solve and how mass efficient are those solutions.
As an example, a carrier in the real world have the displacement of about 100'000 tons but can be destroyed with a single rocket which weight of about 3 tons. If the probability of destruction is 1%, then you better have 1000 of such rockets, it will be just 3% by mass, and the probability of survival for the bigger ship will be about 0.004%. (in real live there will be a problem to launch that many rockets, but in space, you can have them floating, and starting on command; but IRL carrier solves another task of offering different capabilities - that is the part of the task for a fleet problem)
But if the efficiency of the bigger ship to be destroyed is not 3% by mass, and you need the same amount(by mass) of rockets, then it might depend on efforts needed to produce those rockets compared to efforts to produce the big ship.
Again the AI thing - current rockets do not have AI(as super intelligent thing), they are capable of winning the intelligence race with a rock but that's all they can - still, they are efficient in the destruction of their targets.
There are other ways to make them smarter not including AI in the equation, including an operator far far away which chooses the initial strategy of the attack.
At some point of nonmagical technology, as far as I can see myself, there is no difference in a big ship or a small ship. You just have more than your opponent and expect that what will be left is that advantage you had, so basically 1:1 ratio by mass.
As an attempt to figure things out you might read the answer, the OP in the question was more extensive in describing the available technologies, but it still not enough to answer the question.(TL;DR; missiles rocks)
Recently I saw a conclusion for Children of a Dead Earth, game, simulation that missiles again rocks, but I know a lot of people who will disagree.
But the conclusion that missiles rocks, again is not directly applicable to your situation, as it means that big ships might be not the best thing to have, but it does not make small ships automatically better than middle sized.
Advances in detection systems play a significant role in the situation but depend on the speed of travel of detected target and the goals it might have. If there is possible some sort of fog of the war situation, then having a fleet of smaller ships can be advantageous to having the big one.
etc etc etc etc
As a recommendation, go with million or few hundred millions of average size ships(100m - 1km diameter ships). It's adequate force per star system with no defense, which will allow establishing the defense and controlling the system over the time. 10 years and the system will be not easy to take, without magical technologies or overwhelming force.
As for a planet, take smaller ships and use them as missiles.