How could an auditory organ perceive sound waves with a completely different anatomy from the vertebrate ear (e.g. canal, typanum, bones) at the same effectiveness?
The eardrum and bones are an impedance matching device that conveys the sound in air to the underlying sense organ, the cochlea.
The cochlea works with stiff hairs reacting to pressure waves. Do you want a different way for that to work? If hearing started underwater, that seems the most natural evolution.
The added parts to make it work efficiently in air could be different. Could a coclea (primary sensor) be devised to naturally work in air? That would eliminate the need for the rest of the stuff which basically serves as an adaptor.
Then you'll get different adaptations to allow the primary pressure sensor to work in air.
Another drastic difference would be to consider how a microphone works. The cochlea mechanically does a frequency spectrum analysis. An eardrum that had a direct high-frequency sampling of its deflection would be utterly different. The nerve signal would get a waveform, which then needs to be processed by the nervous system to notice features (if not doing a spectrum). So more like how the optic nerve takes raw data and finds lines, patters, color space transform etc. The audio nerve would take a waveform either as a single point or a small window of time that the wave moves through, and does the analysis in software. It may be able to use different algorithms than we get by starting from the frequency decomposition.
Another answer (dunc123) mentioned piezoelectric crystals. That makes me consider that a microphone system as I speculated on above may evolve from an earlier mechanism used for sensing acceleration (like our own otoliths) rather than from something like a lateral-line sense.
It looks like stiff antennae or hairs are your best best; the most well-studied invertebrate ear seems to be the Johnston's organ, a multipurpose mechanoreceptor found in the joints of insect anatennae. A larger organism which needed to listen to a wide frequency range might require an array of different-length hairs.
An auditory organ's job is to detect tiny movements. If we want something truly alien then we need to think of a different way to detect those movements. First we should look at how the ears we have work.
The vertebrate ear is a very complicated organ. Sound waves enter the outer ear and cause the eardrum to vibrate. This causes some small bones to vibrate which in turn cause a fluid to vibrate. The vibrating fluid causes some hairs to vibrate. These are attached to cells which detect the vibrations using mechanoreceptors. In fact, these are the most sensitive mechanoreceptors in our body. Most of our mechanoreceptors are in our skin, contributing to our sense of touch. We often feel sounds with our skin, especially low frequency ones.
Insects and other arthropods detect sound using a chordotonal organ which can be located in all sorts of places within their body. The basic operation of it is similar to our ear: it uses mechanoreceptors to detect vibrations.
Hearing and touch both use mechanoreceptors to detect the movements we call sound. However we have another sense that we can use to detect sound. We can see sound when, for example, we look at the cone of a speaker. Eyes don't use mechanoreceptors; they use photoreceptors.
Tiny hairs surrounding the alien's eyes would vibrate in response to sound. Different hairs may respond to different frequencies. The eyes would detect the movement and transmit this to the brain. Their eyes would need to be at least bifocal, allowing them to clearly perceive both the exceptionally close hairs and events occurring in the outside world. Refocusing of the smaller lens would allow it to focus on different ranges of hairs, providing a band-pass filter for the alien.
Instead of observing the hairs directly vibrating the alien eye may respond to the diffraction patterns caused by light shining through their array of fine hairs. As the hairs move, so the patterns change.
Of course, they wouldn't be able to hear in the dark. At least, not until they evolved to have ultraviolet bioluminescence in the fine eye hairs. Or behind them in the case of the diffraction detection method. This would prevent the light from interfering with vision in other wavelengths and vice versa.
"Those youngsters today never seem to take their eyebuds out. My brood play their music so loud I can see the whole house vibrating! And the amount of glowing eye makeup they wear..."
Taste and smell are harder to use to detect the influence of sound waves. Some type of symbiotic bacteria living within massively flared nostrils perhaps. They would emit chemicals in response to high frequency stimulation. An alien using this method would require exceptionally accurate response to smells and tastes.
On the other hand, deposits of piezoelectric crystals across the alien's body may generate suitable electrical signals to interface directly with their nervous system.
Possibly by detecting resonance on a membrane through nerve signals.
Sound is vibrations in the air, and when those waves run into things you get vibrations.
Human skin is sensitive to sound to some extent, especially low frequencies.
If the aliens membrane had a lot of nerves running through it, it could be pretty sensitive to vibrations at higher frequencies. This is similar to the human tympanum, but any structure designed to detect sound vibrations is going to work in a similar way, and so parallel evolution is to be expected.