NIMBYism
The thing about airplanes, they need airports. Airports have everything that nobody wants to have as a neighbor:
Toxic chemicals? We've got jet fuel, tetraethyl lead in the avgas, de-icing agents, plus lubricants and hydrolic fluids galore. Check.
Noise? Jet aircraft are literally the exemplar of damaging levels of sound.
Traffic? A nightmare.
To make matters worse, the very existence of an airport forces physical limits to what all neighboring plots of land are allowed to build, in order to clear the airway for planes to fly with sufficient separation minima.
Seaports have a lot of the same issues, sure - but the noise is profoundly reduced (ships don't go blasting overhead my house... at least not until climate change has had a few more centuries to work) for individuals, and the majority of neighboring real estate is the ocean... which nobody's building on anyway.
Unless you're in a spectacular hurry, sea travel is friendlier.
Environmental performance
Remember what I said about Tetraethyl lead? Before turbines, it's piston-driven props running on leaded gasoline. This is NOT optional. We still use it today because nothing else* has been proven to effectively get Octane levels up above 100. 100+ Octane is NOT optional for aircraft - ultra high compression is THE key to aircraft engine power.
The only other way to get that kind of Octane rating is with ethanol, but ethanol is aggressively volatile it gleefully boils away as you climb, creating a condition known to engineers as "vapor lock" - and known to the rest of us an "engine failure emergency."
(* There's one promising candidate in the market now, but it's struggling to get adopted.)
If people want to get lead out of gasoline, including aviation fuel, this limits the power available to aircraft and prevents airline infrastructure from being developed anywhere near as robustly. Seat prices go way up, and aircraft must fly lower - meaning less efficiently.
I know you said speed over comfort but...
Comfort is relative. An ocean liner enjoys tremendous economies of scale, and so may include all sorts of amenities that even modern long-haul airliners can only dream of. Compared to one of those noisy, filthy, cramped airplanes? It might as well be a cruise ship. Maybe we're not taking in concerts, gambling, or stopping at every tropical port with a tourist trap... but I can at least breathe fresh air, walk around, stretch my legs, eat at a proper table like a civilized person - and at a fraction the per-seat price. I can also bring whatever luggage I please, and I didn't have to go WAY out into the boonies to get to that utterly remote airport nobody would let be built near them.
Air travel, to abuse a quote, ends up 'nasty, brutish, but short.' It's reserved for military/emergency purposes only - or the eccentric but spectacularly wealthy. High-speed rail replaces the ocean liner inland, but air travel across large spans of LAND is more common than trying to fly the things over the oceans.