Communication Basics
By this, I mean the primary sensory channel of communication. For example, we humans communicate through two channels: sound (speaking) and sight (reading). Before deciding how to form a common language, we must decide which sensory channels are available to aliens for communication.
By our knowledge and experience so far, there are four possible channels for this purpose: sight, sound, smell (through pheromones as a lot of animals do) and esp (aka telepathy). We would need to determine which of these channels do the aliens utilize and if hybrid communication systems are possible for faster information exchange.
The Easiest Channel
If the aliens can use multiple channels for communication like us, the easiest channel for us to use would be sight. We can use paper signs for this, hand gestures and computer screens to utilize this channel. This would require the least coding and decoding effort required. It may or may not be the fastest channel, considering that electromagnetic waves can carry a huge amount of information, but almost everyone on Earth prefers using sound for communication instead of sight.
The next easiest channel would be sound. This would probably require some sensitive sound recording and transmitting equipment. Plus it would require more technical coding and decoding which would limit the number of scientists who can work on this type of data (it is easier for most of us, humans, to code and decode visual signs than audio communication which involves infra and ultra sonic ranges).
Smell would be a quite complex channel as we aren't so good with quickly releasing and then absorbing chemical compounds. This would be slow, tedious and complex.
ESP (telepathy) would probably be the hardest of all, considering how many esp hoaxes and wannabes are busted regularly in science documentaries and researches.
After Channel Decision
Once the channel has been decided and it has been known what frequency ranges the aliens operate at, the next step would be the communication of some primary realities which would function as breaking the ice and informing each other of the extent of knowledge we have.
Here again, it would be much easier to use sight channel than sound. For example, we would start with counting from 0 to 99. This would inform them about the signs we use for our number system. Place one bead on the table and write 1. Place two beads next to it and write 2 and so on.
Once the number system has been established, we would show them number tables (sequences). As in:
2 x 1 = 2
2 x 2 = 4
2 x 3 = 6
2 x 4 = 8
and so on. At this point, we would leave some blanks in our tables and expect the aliens to fill them. If they do it successfully, it would mean that they have successfully understood our number system and we can progress.
We would not continue towards complex mathematical equations, factorials, differentials, integrals and that sort of thing.
Non Mathematical Communication
This would involve teaching them abc from the very start and then progressing with images. If we succeed in teaching them our written language form, we can communication to them as we communicate with each other in text. If not, we would have to restrict ourselves to images and signs which would be a slower method, but still functional.
If the aliens only use audio channel, it would require us to learn their audio communication basics instead of teaching them ours. This would go by understanding what reactions do they show against which audio messages.