The International Phonetic Alphabet is used to describe sounds present in human languages.
It is organised by place and manner of articulation in the vocal tract. I.e. the sound [p] is a bilabial voiceless stop, meaning it is made using both lips, the vocal cords are not vibrating, and as a stop the lips fully touch stopping the airflow entirely before releasing it again.
All this organisation is only possible because we have a good understanding on the human vocal tract. Similar tables could be constructed for any given phonetic system, but a system that aims to include all possible vocal tracts would have to be so large as to be completely unusable. The only way to objectively represent the sounds produced by an arbitrary vocal tract in a way that isn't incredibly unwieldy would be to use a spectrogram.
The fact that language need not be based on sound at all makes it even less plausible. Even dealing solely with human sign languages the array of possible handshapes, locations, movements, and other gestures is so vast that no equivalent of the IPA (that can meaningfully distinguish any two signs distinguished in any sign language) has been constructed, and even transcriptions dedicated to one specific sign language are difficult or impossible to interpret without detailed knowledge of the language in advance (making them more like English spelling than a phonetic alphabet). If your aliens have drastically different body plans, encoding their sign languages would make it even more complicated.
Other modes of communication e.g. using chromatophores, radio, scent, etc will fall somewhere between these extremes (one where it's practical to transcribe within a given species but not across all possible species without just giving the raw sensory data received by the "listener", and one where even phonetic transcription of a single language is impractical).
So wanting to also include all modes of communication would make the implausible utterly impossible.
Your best bet is to avoid attempting phonetic transcription entirely.
In general I would recommend simply translating the speech of aliens, with some note as to the language and mode of communication, especially when the language is not based on sound. Where quoting verbatim is absolutely necessary (e.g. for proper nouns, words with no equivalent in human language, and potentially poetry being recited) I would give the text in the alien language, either followed immediately by a translation, or in a context that makes it unambiguously clear what the meaning is. Note that this is the approach Tolkien (the father of constructed languages in fiction) takes and even then some readers find the amount of his languages he includes off-putting so attempting to include more is almost certain to backfire.
When you do quote alien speech verbatim, I would rely on an ad-hoc transcription. You should not attempt to include all the phonetic detail of the aliens' actual speech, but instead give an impressionistic understanding of how your audience would describe the sounds. Your aliens may have very different vocal tracts, but if they have a sound (or multiple sounds) that sound similar to a [p], just write them as "p". I would avoid using diacritics (accent marks) or apostrophes as these tend to make (English-speaking) readers more conscious of the artificiality of what they're reading (even though of course these are used in various languages to denote specific sounds).
Your ad-hoc transcription should probably reflect those used historically by English-speakers to write words in foreign languages. These typically use vowel letters similarly to their use in Spanish or Italian, use consonants in their usual English sense, and generally avoid c in favour of either k or s. Sounds not present in English should generally be transcribed according to the closest sounding English sound (or combination of sounds), although certain foreign sounds do have fairly common transcriptions (e.g. the German "ch" sound, the same as the Spanish "j" sound, or Russian "х" sound is usually transcribed ad-hoc as either "kh" or "ch", although the latter may be confused with the English "ch" sound [t͡ʃ]).
As an example, this is article 1 of the Universal Declaration of Human Rights in Hindi, first written in the native script, then in an ad-hoc transcription as I would write it if it were an alien language in a novel, in a scientific romanisation you'd see in a grammar of Hindi, and lastly in the IPA:
- अनुच्छेद 1(एक): सभी मनुष्य जन्म से स्वतंत्र और मर्यादा और अधिकारों में समान होते हैं। वे तर्क और विवेक से संपन्न हैं तथा उन्हें भ्रातृत्व की भावना से परस्पर के प्रति कार्य करना चाहिए।
- Anuched 1 (ek): Sabhi manushya janma se svetantra or meryada or adhikaro me saman hote heng. Ve terk or vivek se sampana heng tatha unhe bhratritva ki bhavena se perasper ke prati karya kerna chahie.
- Anucchēd 1 (ēk): Sabhī manuṣya janma sē svatantra aur maryādā aur adhikārō̃ mē̃ samān hōtē haĩ. Vē tark aur vivēk sē sampanna haĩ tathā unhē̃ bhrātr̥tva kī bhāvanā sē paraspar kē pratī kārya karnā cāhiē.
- [ənʊtːʃʰeːd eːk | səbʰiː mənʊʂjə dʒənmə seː sʋət̪ənt̪ɾə ɔːɾ məɾjaːd̪aː ɔːɾ əd̪ʰɪkaːɾõː mẽː səmaːn hoːteː hɛ̃ː‖ ʋeː t̪əɾk ɔːɾ ʋɪʋeːk seː səmpənːə hɛ̃ː t̪ətʰaː ʊnʰẽː bʰɾaːtɾɪt̪ʋə kiː bʰaːʋənaː seː pəɾəspəɾ keː pɾət̪iː kaːɾjə kəɾnaː tʃaːhɪeː‖]
I would be extremely hesitant to attempt to transcribe any language based on a medium other than sound. Rather than describing a sign or pattern of lights etc, you're almost always going to be better off relying on the best translation you can give (together with a note that they are communicating through signs or lights etc), even for proper names and words with no clear equivalent in English, as descriptions of these are likely to get overly cumbersome and draw a reader out of the story.
If you want to show off your conlanging beyond this, I would suggest doing as Tolkien did and leaving it for the appendices.
