How could a vertebrate animal evolve for extended flight before evolving land adaptations? Is that even possible at all?
By extended flight, I mean breathing air and spending most of their time in the air, similar to sea birds.
Hmm. This is an interesting one. Unfortunately, my best offer is speculation, at best, but I'll try to give some very real scenarios where this could play out.
If you were to take the flying fish (https://en.wikipedia.org/wiki/Flying_fish), and over time they find more and more evolutionary reason to spend more time out of water. Say, for example, there's a more robust source of food in the air (insects, small birds, etc), there might well be reason for any strains that show adaptation to prolonged time in the air (breathing air without gills). Or perhaps their normal food source is depleted, for whatever reason? Then the only option is to get food from above water. Then they either adapt or die.
Another, and in my opinion far more likely option, would be if (for example) dolphins or orcas (far more likely for dolphins in this instance, purely based on weight and size) were to have a subspecies evolve from their numbers with elongated, or at least more pronounced, pectoral fins which could, in time, evolve into more wing-like appendages, should there be an evolutionary advantage to this.
It really all depends on how the environment changes, and how the species changes to survive that.
Yes.. or No... Kinda
The early predesetor of flight is gliding. Now there is a family of marine animals that glide over the water, sometimes for up to 40 ft! Now while a bird will not happen, we can explain how a gliding fish can evolve to an actual flying fish.
Though keep in mind that the new flying fish will be amphibious in a way, having to fly into water every once in a while.
How about rolling your evolution backstory forward from a creature like the flying fish? A prey animal that avoids predators by launching out of water into the air and gliding for a time. Evolutionary pressure on this animal could continue to make it a better and better glider until it actually evolves powered flight. Along the way, its also adapting to air based vision and breathing. Eventually, you can arrive at a whole ecosystem evolved from this starting creature. Some can certainly live by flying, essentially, forever. Maybe they still spawn in the ocean before they take to the air for the rest of their lives, or maybe they live on their parent's back until they launch off on their own.
You start with Flying Fish. This is a family of bony fish (about 64 species) who leap out of the water and glide at low altitude, using adapted fins as wings. This seems to have evolved as a way of evading predators. They can cover distances of about 50 metres on their own power, or use updrafts at the leading edges of waves to go considerably further.
Then you need them to evolve lungs, and flight muscles. Neither of these is totally implausible, given the right pressures, which might well start with a predator fish that learned to see in air and leap out of the water to catch them. You'll need a long time for these adaptations to evolve, but it seems workable.
Ok, while an interesting mind game, there are some inherently bothersome issues with your question... The question you ask is 'How could, if possible, an vertabrate...(etc)'. First issue is 'How could?', as the answer is 'through evolution, due to pressures caused by changes to the living environment of said vertebrate creature.
The thing to understand about evolution is that one creature NEVER turns into another creature. The evolutionary process is an extremely slow and random process with many failures (in fact, most changes fail- even the good ones) what makes a given evolutionary change a 'success' is the failure of the previous strain of the beginning creature (at least locally). I say it that way because biodiversity occurs due to the differences of locality stresses vs total environmental stresses. For example, if a volcanic vent forms on one side of a mountain, then the area around that vent will suffer the biological stresses of the volcanic gasses, liquids (water infused with said gasses), increased temperature, etc. If species in the area are unable to adapt fast enough, the entire local ecosystem can collapse, creating a dead zone. But as you get further away, other species (or other families of the same species that had lived in the now dead zone) may experience a reduced stress form due to diffusion, and may still suffer loss of health, food supply, etc. but not quite enough to actually KILL the lot as in the dead zone.
As life always attempts to spread into any empty area, in an ongoing effort to survive in the new, more hostile dead zone plants without resistance to the new stresses (acid, alkaline, heat, etc.) will be unable to grow in the remaining dead zone, but if several generations accidentally happens to create a genetic version that is just a LITTLE more resistant, then THAT plant will start to encroach into the dead zone a little further than the others. This creates an ideal area for that variant to survive and THRIVE in where perhaps it would have been pushed out in the more hospitable environment, but here it has a chance to grow and dominate, giving future generations the chance (once maximum survival range is reached) to possibly enhance (through chance, generally) the resistance to the still excessive stresses further into the dead zone.
Assuming there are no airborne seeds of plants that already have some ability to adapt to the hostile dead zone stresses, (creating a new scattering of living plants in the dead zone with new species or variants encroaching from the edges) this process will continue in the same way until the dead zone is reclaimed by totally resistant strains. The same process applies to animals, but tends to be slower. I know I'm being long winded, but it's important to say, as evolution is constantly occurring- even today, but it takes a failure of a current eco-niche species, or a hugely beneficial change occurring randomly that allows an evolutionary change to advance into the future.
This means that the biggest issue with your hypothesized flying vertebrate evolutionary process is that once lungs (or otherwise thin air breathing structures) begin to evolve, land living (or tree living) changes rapidly become needful things, as the fragility of wing structures means that collisions with land will become a dramatic survival pressure. Just my opinion, of course, but trying to explore all angles, and the only thing I could come up with was if the world did not have ANY land for a time, then survival through LONG time flight might become an effective survival trait, but I don't know that otherwise the evolving 'flying fish' could survive both ocean going threats and evolving land-based threats (or worse, pre-existing threats).
Also, the evolving species would need to find an acceptable food supply that does not require them to return to the depths, and that they can obtain even if followed by predators in the water. All in all, while such evolutionary adaptations are theoretically possible, the likelihood of it becoming successful enough to thrive is pretty small, compared to the possible predatory species that might see such flyers as lunch on the wing, especially if they do not become predators in their own rights... Odds are that such evolutionary 'experiments' have occurred in the past, but that they simply couldn't compete with other dominant species and predators, and died out (failed), rather than thriving. That, or they DID develop land-based aspects and survived that way. Or not. It's evolution after all, no guarantee of survival. Never know when you'll be hit by a huge meteor, you know. Or lose your food supply, or encounter another catastrophic environmental change...
A swimming species could be forced out of water to stay on top of it by a predator. First step in evolution would be a duck like creature, but in the middle of the ocean, still needing to dive to keep wet. Over time they evolve to have lungs. In the meanwhile their predator evolves to swim close to surface, and this creature evolves wings.
As others have said - start with a flying fish. It jumps out of the water and flies to escape predators, or to impress mates, perhaps both.
Some of these flying fish have a wing configuration which lets them do a primitive version of dynamic soaring - the non-flapping, very energy efficient kind of flight which albatrosses use.
Air is less dense and less viscous than water, so it takes less energy to move through it. A fish that is flying from point A to point B is using less energy than a fish which is swimming the same distance. Especially if it is using dynamic soaring. If one shoal of fish is swimming to that big juicy swarm of krill and another shoal is flying, the fliers will get there first and grab the food. So natural selection will favour the fish that are better at dynamic soaring, until you get a creature as good at it as an albatross. (Unless there is a bigger selection pressure to stay underwater, such as huge flocks of hungry gulls trying to pick you off as you fly by).
Like the albatrosses, the fish will dive back into the water to feed. It can also wet its gills and skin. It'll need to lay its eggs or give birth to its babies in the water. And unless it develops the swift's ability to sleep in the air, it will need to return to the water to rest (like seabirds do when they are out to sea far from any land).
So a fish-albatross-swift may spend most of its time in the air, apart from feeding dives and reproduction.
Not sure what selection pressures would drive it onto land. Foraging on the beach or in estuaries, perhaps? A fish will be rubbish at taking off from land. The tail is designed to generate thrust in water.
EDIT TO ADD INFO ON TAKE OFF ON LAND: You need a propulsion system that works on land (legs for instance) and which can accelerate you forward fast enough to generate lift as air flows over your wings. Fish use their tails as propulsion - waggling them to push against the water. Air is not dense enough for this, plus on land you also have to overcome friction of the fish belly against the ground.
The fish-albatross CAN take off from land by spreading its wings and facing into a stiff breeze. But no breeze = no take off. Watch this albatross chicks vs sharks clip and imagine there are also predators on land! Those albatrosses pick isolated islands to breed on for a reason.
This possibility was discussed in Jack Cohen and Ian Stewart's What Does a Martian Look Like?: The science of extraterrestrial life (2002). They suggested this could occur on a higher-gravity planet because air density would be greater at sealevel than here on Earth. Also, winds are more powerful as the air density increases. For the same speed, a volume of air has greater mass and therefore has greater momentum. From there is a small step to imagine an ecosystem that takes advantage of this free lift.
First marine creatures adapted as gliders and if further adaptation occurred this might turn fins or equivalent structures into wings. Not exactly flying fish, this will depend on what marine creatures are on the evolutionary pathway(s) to flight.
For example, some of the large colonial jellyfish might allow themselves to pushed by windpower and grow sail-like structures to assist and eventually skin above the waves while trawling for prey with long skeins of their tentacles. There also could, possibly, be soaring sting rays.
In an environment like this, the seas and oceans of high-gravity planet there would be most likely many sail-powered species. Flying sea creatures would be a bonus.
High-gravity planets are also expected to have extensive sea coverage due to the larger amount of water present of their surfaces. I have seen a suggestion that some Super-Earths might have oceans a thousand kilometres deep. No land masses there. So if there was flight on this planet it would have to be marine creatures that evolved it.
PS: Cohen and Stewart's xenobiology book is essential reading for anyone in the creature design business or with an interest in alien biology. The title given above is the paperback edition's as the hardback had the original title of Evolving the Alien. They also wrote two science fiction novels that have some of the best aliens. Look for Wheelers and Heaven. Jack Cohen was an adviser to Anne McCaffrey for Pern dragons and Larry Niven & Jerry Pournelle for the grendels in their Heorot series.
It's not only possible, it's relatively probable!
This unique-looking fish is the sole member of the family Pantodontidae. It is an obligate air-breathing, surface-dwelling fish that flaps it pectoral fins while jumping and may be capable of some gliding. Ah, just "some" and just "gliding", you say. Well, allow me to introduce...
This remarkable relative of tetras is thought by some to be the only fish capable of true, powered flight. Even those who protest that its fins are not capable of a proper stroke agree that it is an accomplished glider that pursues flying insects through the air.
So, on the one hand, you have a possible glider that breathes air and flaps it's fins while out of water, and on the other you have what may already be capable of actual flight, and neither of them have ventured onto land at any point in their evolutionary history. It isn't remotely far-fetched to imagine an aquatic creature that breathes air and is able to take flight, but lacks any ability to move on land and has no ancestors capable of moving on land either.
One avenue of research you need to look into and incorporate with the great suggestions above is surface loading ratio, and wing shape.