What is IPA
IPA is the International Phonetic Alphabet. In principle, it has letters and diacritics which enable the representation of all sounds that the human phonatory apparatus can produce when used in normal speech. It also includes symbols which enable the representation of some (but not necessarily all) sound variations, such as pitch, tone, stress or phonetic emphasis. It cannot represent the sounds made by our close cousins the chimpanzees, or by our more distant cousins the cats and the dogs, although the chimpanzees, the cats and the dogs have lungs, larynges, vocal chords, tongues, teeth, lips and noses just as we do: but theirs are different enough from ours to make different sounds. It cannot represent music. It cannot represent (or at best can only very awkwardly represent) various noises, such as the noises we make to imitate kissing or blowing wind.
How IPA is used
IPA can be used in two ways, known generally as broad and narrow transcription. (The representation of speech in IPA is called transcription and not writing because it operates at a much less abstract level.) In general, dictionaries give a broad transcription, which conflates the sounds which are allophones of the same phoneme. For example, the English phoneme /l/ has two allophones, a "light" [l] as in lent and a "dark" (velarized) [ɫ] as in fiddle, which are both shown as /l/ in most dictionaries. (Note that the broad transcription representing phonemes is usually written between slashes and the narrow transcription representing phones is written in brackets.) There is no pair of words in English which are distinguished by having a light [l] as opposed to a dark [ɫ], and this is why the two kinds of /l/ are said to be allophones, that is, they are just different realisations of the same phoneme.
An alien IPA for an alien species
How would one build a phonetic alphabet for an alien species? First of all, one must begin by understanding how the aliens species makes sounds. There are many ways to make sounds, and humans use only some of them (at least for speech). Do the aliens make speech sounds like us, by inducing vibrations in a column of air and then modifying the pure tones by various means, such as adding vocal chord vibrations, changing pitch, coloring the sound by rounding the lips or releasing the air through the nose, adding friction sounds, or stopping and releasing the air with the lips or the back of the toungue? Or do they use alien mechanisms, for example the stridulation used by the grasshoppers of Earth?
After considering the phonatory apparatus of the aliens one should decide what sounds are considered distinct by the aliens. For example, English (at least RP and General American Englishes) distinguishes the front mid vowel /ɛ/ (as in ten) from the front open vowel /æ/ (and in tan); a Romanian needs quite a lot of effort to learn to hear this distinction, because the Romanian language has only one phoneme /e/; or consider Mandarin Chinese, where there are four different phonemes which all sound like ch /tʃ/ to a European. (The four Chinese sounds are /ʈʂ/, /tɕ/, /ʈʂʰ/ and /tɕʰ/, transcribed
zh, j, ch and q in Pinyin.) That is to say, just because the phonatory apparatus can make certain sounds does not mean that they are meaningfully distinct in any given language.
For an example of human yet alien-like acoustic language consider the whistled languages, which are of course impossible to transcribe using IPA, but are somewhat representable using musical notation.
Then how would we write the alien language?
When alien language is to be represented in a book one must necessarily use some sort of tranliteration or transcription scheme using Latin letters, possibly with diacritics, or with a variation in font (to remind the readers that the letters do not stand for their usual sounds), or both.
Chinese Pinyin is a (relatively) widely known transcription scheme, which uses Latin letters in a most un-Latin way. For example, let's say that you see quanxu; a naive English reader might naively imagine a pronounciation like /kwænksu/ whereas a savvy reader will know that it should be /tɕʰwanɕy/ (sort of like chʰwahnshü, with ah the vowel in car and ü as in German).
Egyptologists use a special transcription system (complete with a conventional pronounciation) to represent ancient Egyptian words -- Twt-ʕnḫ-ı͗mn, pronounced conventionally /tuːtənˈkɑːmən/ is much easier to show in print than the actual hieroglyphs. (Wikipedia says that the original pronounciation may have been something like /taˈwaːt ˈʕaːnxu ʔaˈmaːn/.)
So to write the alien language use Latin letters, in some sort of systematic way, trying to represent not the sounds of the alien speech but the meaningful distinctions and roles of those sounds. Use vowels for sounds which can be pronounced (by the aliens) continuously and can form the center of a syllable; you may use the human front-to-back order /i/-/e/-/a/-/o/-/u/ to represent some alien natural order, for example high pitch to low pitch; use continuant consonants such as f, v, s and z to represent sounds which can be pronounced continuously but are not syllable centers; l, r, m and n can be used in both roles; use stops such as p, t, k to represent momentary sounds which cannot be sustained; use h and possibly also n (and even m) to indicate some sort of variation corresponding for example to human aspiration or nazalization.
