You need a universe where:
- Everything is MUCH closer. The Moon needs to be only a few thousand miles away, not hundreds of thousands. The other planets can't be much farther. Ancient folks had no way to build rockets or heat shields, nor to astrogate effectively, so the Universe must shrink to a small, calm pond they can handle.
- Escape velocity is much, much lower, so gravity works quite differently. Low enough for a kite or sailed vessel to not return under some circumstances. (We'll conveniently overlook that this prevents our form of life from existing)
- Air is everywhere. Space must be breathable. Ancients have no way to compress or store air for weeks of travel, so the Universe must be tweaked a bit.
- Water and Food must be available on most celestial bodies (not the Sun, of course. That would be silly)
This means that your ancients can take voyages quite similar to sea voyages, using their nifty-but-similar ether-ships or air-ships. Like sea voyages, they can use the winds and currents of space, astrogate using the sun and stars, and replenish their provisions at their destination while they deal with traders or monsters or whatever.
This also means, by the way, that monsters can jump off planets and become roving space monsters preying upon voyagers, and that space-pirates may swarm near wealthier planets.
This has been done, of course, in literature and cinema already: Baron Munchausen's trips to the Moon and dalliance with the Queenqueen there (The Adventures of Baron Muchausen, 1988), are merely one example.