There are a few reasonably plausible ways to design a fictional solar system with planetary orbits quite close together.
Part One: Crazy Co-orbitals.
Epimetheus, a moon of Saturn, orbits the center of Saturn with a semi-major axis of 151,410 kilometers, plus or minus 10 kilometers.
Janus, another moon of Saturn, orbits the center of Saturn with a semi-major axis of 151,460 kilometers, plus or minus 10 kilometers.
That is a difference of approximately 30 to 70 kilometers.
Epimetheus has dimensions of about 129.8 by 114 by 106.2 kilometers.
Janus has dimensions of about 203 by 185 by 152.6 kilometers.
So the radii of the two moons in their orbits largely overlap, and when the inner moon Epimetheus catches up with Janus they should collide and destroy each other.
But that doesn't happen to those co-orrbital moons.
Janus's orbit is co-orbital with that of Epimetheus. Janus's mean orbital radius from Saturn was, as of 2006, only 50 km less than that of Epimetheus, a distance smaller than either moon's mean radius. In accordance with Kepler's laws of planetary motion, the closer orbit is completed more quickly. Because of the small difference it is completed in only about 30 seconds less. Each day, the inner moon is an additional 0.25° farther around Saturn than the outer moon. As the inner moon catches up to the outer moon, their mutual gravitational attraction increases the inner moon's momentum and decreases that of the outer moon. This added momentum means that the inner moon's distance from Saturn and orbital period are increased, and the outer moon's are decreased. The timing and magnitude of the momentum exchange is such that the moons effectively swap orbits, never approaching closer than about 10,000 km. At each encounter Janus's orbital radius changes by ~20 km and Epimetheus's by ~80 km: Janus's orbit is less affected because it is four times as massive as Epimetheus. The exchange takes place close to every four years; the last close approaches occurred in January 2006, 2010, 2014, and 2018, and the next in 2022. This is the only such orbital configuration known in the Solar System.
And it seems possible that the co-orbital status of Janus and Epithemeus, with them switching orbits appoximately every four years, may have existed for tens of millions or hundreds of millions of years.
So a science fiction writer might consider designing a solar system where two planets orbit the star in such an orbit about a thousand times larger with about 50,000 kilometers betweeen the two planetary orbits. And if they have a program that can run orbital similations, they might possibly determine how long such an orbital configuration would be stable for.
Of course, the orbital periods of the two planets around the star would be about one Earth year long, and it would take thousands of such orbits for the inner planet to catch up with the outer planet and for them to switch orbits.
So in the millennia between such events, a civilization could arise on one of the planets and develop astronomy and discover that the planets orbited around their star, and notice that the inner planet was catching up with the other planet. And if their math wasn't up to calculating what would happen, there might be widespread fear that the planets would collide or that one would be ejected from its orbit.
Part Two: Forbidden Zones.
Stephen H. Dole, in Habitable Planets for Man, 1964, discusses the spacing of the planets in our solar system among many other factors involved with planetary habitability.
On pages 48 to 52 Dole discusses the forbidden regions around the orbits of planets. The size of the forbidden region of a planet is calculated from the planet's mass, the mass of the star, and the semi-major axis of the planet's orbit. Within the forbidden region, smaller objects can not have stable orbits and so can not clump together to form planets.
According to Dole's calculation, about half of the Solar System is within the forbitten zones of various planets. Thus at most about twice as many planets of similar size could exist within that distance of the Sun. Of course other planet could have stable orbits farther out than the known planets.
I note that small changes in the mass of main sequence stars cause greater changes in their luminosity. If star A is one percent more or less massive than star B, its luminosity will me more than an one percent higher or lower than the luminosity of star B.
So a planet in the habitable zone of a more massive and luminous star will be relatively farther out in the star's gravity field than a planet in the habitable zone of a less massive star. Thus the gravity of the planet should be stronger relative to the gravity of the star at its orbital distance, and the planet should ahve a larger forbidden region.
And a planet in the habitable zone of a less massive and luminous star should orbit deeper within the gravity of the star and thus the gravity of the star will be stronger relative to the planet's gravity, and that should make the planet's forbidden region smaller.
Or maybe it goes the other way around. I'm not sure.
The famous TRAPPIST-1 star system has 7 planets orbiting very close to a small dim star, TRAPPIST-1.
The planetary obits thus have semi-major axis with small differences between them, hundreds of thousands of kilometers instead of tens or hundreds of millions.
All the planets would be visible from each other and would in many cases appear larger than the Moon in the sky of Earth
Three or four planets – e, f, and g or d, e, and f – are located inside the habitable zone.[x]
Because they are so close to their star, all the TRAPPIST-1 planets are probably tidally locked to it, and it is uncertain whether tidally locked planets can have life.
TRAPPIST-1 is believed to be older than the Sun, so the planets should have had their present orbits for billions of years.
The Kepler-36 system has the smallest known ratio between planetary orbits.
Kepler-36 b has an orbital semi-major axis of 0.1153 AU,and Kepler-36 c has an orbital Semi-major axis of 0.1283 AU, a difference of 0.013 AU or 1,944,772.3 kilometers. The ratio between the semi-major axis of the orbits is 1.1127493.
If a planet orbited its star at a distance of 1,000,000 kilometers, a planet orbiting 1.1127493 times as far out would be at a distance of 1,112,749.3 kilometers, 112,749.3 kilometers farther.
If a planet orbited its star at a distance of 1,000,000,000 kilometers, a planet orbiting 1.1127493 times as far out would be at a distance of 1,112,749,300 kilometers, 112,749,300 kilometers farther.
So one can imagine a solar system with a habitable planet at 0.898 AU, another at 1 AU, another at 1.1127493 AU, a fourth at 1.238211 AU, a fifth at 1.377818 Au, and so on. Though I don't know if such close orbits actually would be stable.
And I don't know how well those close orbits in the TRAPPIST-1 and Kepler-36 system agree with Dole's calculations of planetary forbidden regions.
Part Three: Trojan Planets.
One possible planetary arrangement would be for two planets to share the same orbit around their star, separated by 60d egrees, a trojan orbit.
HOwever, all know trojan orbits in our solar system involve objects with vast mass differences between them.
For example, the mass of the Sun is about 330,000 times the mass of Earth, and thus about 6,000,000 times the mass of Mercury, and about 1,038.3889 times the mass of Jupiter. The largest asteroid in a trojan orbit is 624 Hektor, about 200 kilometers wide, about 0.0157 the diameter of Earth, and thus about 0.0000038 the volume of earth, and presumably having less than 0.0000038 the mass of Earth, which would be less than 0.000000011 the mass of jupiter.
As a rule of thumb, the system is likely to be long-lived if m1 > 100m2 > 10,000m3 (in which m1, m2, and m3 are the masses of the star, planet, and trojan).
The dividing line between planets and brown dwarfs is about 13 Jupiter masses or about 4,131.4 Earth masses. A system with a highest mass planet and an Earth mass planet would be unlikely to be stable.
Similarly a system where the larger planet was Earth mass would probably not be stable unless the smaller object had less than 0.0001 Earth mass. And the smallest gravitationally rounded bodies in the solar system that could be called planets and not asteroids or other minor bodies have mass around 0.0001 Earth mass.
The mass range for habitable planets would probably be only about 10 or 100, certainly not enough for a larger planet and its smaller trojan planet to both be habitable.
But something even better than trojan obits has been proposed.
Part Four: Co-orbital Rings.
Astrophysicist Sean Raymond in his PlanetPlanet blog has a section devoted to designing imaginary solar systems with as many planets, preferably habitable, as possible.
In "The Ultimate Retrograde Solar System", Raymond found a paper by Smith and lisseur saying that if alternate planetary orbits were in opposite directions, planets could be packed closer together with stable orbits than if they all orbited in the same direction.
Raymond said that about four planets could orbit the Sun in the habitable zone if they all orbited in the same direction, but about eight could orbit the sun if they orbited in alternating directions.
But Raymond warns science fiction writers:
With the Retrograde Ultimate Solar System we are now swimming in impossible waters. Two planets can end up orbiting the same star in opposite directions, but only if their orbits are widely separated. I don’t know of any way that nature could produce a system of tightly-packed planets with each set of planets orbiting in the exact opposite direction of its immediate neighbors.
This means that the Ultimate Retrograde Solar System would have to be engineered. Created on purpose by some very intelligent and powerful beings.
Such a solar system with closely packed orbits alternating between prograde and retrograde would have to be artificial, and not natural in your story. Characters who know much about planetary formation would have to know that system was artificial.
The good part is coming.
in "The Ultimate Engineered Solar System" Raymond references another paper by Smith and Lissauer.
Smith and Lissauer show that a ring of co-orbital planets can be stable, if the planets all have the same mass and are all equally spaced along the orbit.
So Raymond designed a system with 42 Earth mass planets sharing the same orbit 1 AU equally spaced. Since an orbit with a radius of 1 AU would have a circumference of about 1,022,022,733 kilometers, 42 equally spaced planets would be spaced about 24,333,874.6 kilometers apart on the orbit.
Then Raymond designed a system with six rings of 42 Earths apiece within the Sun's habitable zone, for a total of 252 Earth like planets.
If the planets were smaller, with about 0.1 times the mass of Earth (about the mass of Mars) there could be 13 rings of 89 such planets each for a total of 1,157 mars like planets in the habitable zone.
Then Raymond designs a system with planets with half of Earth's mass, and so 52 planets in each ring, and with the rings of planets alternating their orbital directions. That gives eight orbits with 52 planets per ring, a total of 416 planets.
But of course such a system could never form naturally.
I can only think of one way our 416-planet system could form. It must have been purposely engineered by a super-intelligent advanced civilization. I’m calling it the Ultimate Engineered Solar System.
We can make such a system millions and billions and trillions of times more plausible by making all the planetary rings orbiting in the same direction, which will reduce the number of rings and the total number of planets.
So that would make the system much more likely to form naturally. But you would probably still have to search millions and billions and trillions and quadrillions of star systems to find one like that which had formed naturally.
So any such star system in fiction would have been made artificially by an advanced civilization.