This idea has been explored before. One example is David Weber's Safehold series, which is based around a society established on another world as a refuge against aliens which have destroyed the remainder of humanity.
The feasibility depends on the presence or absence of a useful FTL drive. Without one, humanity would have to have already started sending out sleeper or generation ships, with one, it may be possible to send out a secret mission to start a new colony under the alien's noses (Figuratively, of course, they may not even have noses).
If an alien species advanced enough to travel between the stars arrives and decides to wipe us out, it is unlikely that we'll be able to escape anywhere in our own solar system, and it is also a little unlikely that simply having skipped off to the next remotely habitable planet will work either - the aliens may have thought of that already. To work, a strategy like this would require a redoubt a long way away, not just a few light years. We're not up to doing anything like this yet.