I'm making a sophont species of insect people with compound eyes and mandibles. They can't smile (no lips) and they can't wink (no eyelids). How would they express themselves if they wanted to communicate with humans on friendly terms?
$\begingroup$ Can you rephrase your question to make less opinion based? As it is now I feel it might be closed soon. $\endgroup$– L.Dutch ♦Jul 28, 2017 at 5:42
$\begingroup$ Opinions aside, I'm not clear about what this question is actually about. How would they communicate? How would they underline their friendly attitude? When? During initial meeting? Later? $\endgroup$– MaciejJul 28, 2017 at 8:58
$\begingroup$ The movie "Enders game" had such an insect species. They couldn't communicate with humans, which led to humans thinking they needed to exterminate the insects, when the insects were just trying to make a peaceful alliance. $\endgroup$– NavJul 28, 2017 at 12:31
Any kind of insect-like alien species (or, really, any kind of plausibly-alien alien species) will be unable to easily mimic subtle body language humans can naturally relate to. Likewise, humans will be quite alien to the other species.
So the aliens will communicate their friendliness via statements like "I come in peace", translated as best as possible. And by not wholesale slaughtering you upon arrival.
It's possible both species will eventually learn to make certain gestures or facial expressions the other species interprets as non-threatening, but there's really no guarantee this works. You'd really have to draw or model your species with different expressions until you find something that has the right intuitive meaning, but you'll likely find that what you think of as "super smile", someone else will interpret as "he's going to eat me!".
As far as how they communicate, let's start by looking at how normal insects communicate with each other.
As this site sheds a little information on, insects communicate in a variety of ways.
Bees do elaborate waggle dances to tell other bees where the food is.
Ants can leave trails of pheromones to tell other ants where the food is. (More information in this biology.SE answer.
Ants can also tap each other each other on the leg to signify "I'm following you".
At least some ants also communicate via sound, rubbing a thing on their bellies to produce different noises. (And another link).
Several other forms of insect (and some spiders), including crickets and grasshoppers, use stridulation, a fancy word for "rubbing body parts together", to make sounds for mating and warning calls. (The ant sounds count as stridulation too.)
Fireflies use Bioluminescence to communicate for mating purposes.
While not technically insects, the male jumping spiders communicate during mating rituals with very complex vibrations akin to music, as well as waving their arms around to get attention.
Tarantulas (another form of spider) will attempt to bluff an opponent into thinking the spider is more powerful by forcefully slamming the ground, creating larger vibrations than a critter normally would for the tarantula's size.
There are, I'm sure, myriads of ways other insects communicate with each other. But this should show you that insects are generally capable of a very wide variety of communication methods.
It's up to you to decide which of these methods (or others not listed) you want your alien species to use. Whatever the methods used, it will be up to both alien and human exobiologists to learn how the other species communicates. From there, linguists and engineers will help them learn to understand the other language (whatever its form) and replicate that language so the other species can understand them.
Humans express their emotions with more than just facial expressions. Emotions can be expressed with body language, gestures and postures, through the tone of our voices, and the way we move.
It is not unreasonable for insectoid sophonts to do the same. Their emotions can be expressed with movement, body language, gestures and postures, their voices and any sounds they might make. Some insects can make vibrations with their body parts, for example, wings or mandibles. Insectoid sophonts might make good vibrations or sounds to show their happiness. This might not be restricted to just happiness it could encompass their entire emotional range.
Terrestrial insects can produce chemicals to facilitate various functions and activities. There are pheromones to attract mates for reproduction. Ants leave trails to aid their foraging. Possibly insectoid sophont might do something similar with their emotions.
Of course, they might communication their emotions via language in the same as humans do. This is, after all, what intelligent organisms do, they can communicate with each other.
However, insectoid sophonts expression their emotions it is improbable they would only use one of the communications suggested above. It seems reasonable they would use a repertoire of behaviour, speech, vibrations and possible chemical aromas to show their emotions. This is not unlike humans. A person shouting vehemently and stamping their feet is obviously showing their anger. An insectoid sophont might make a hostile buzzing sound and snap its mandibles to show its anger.
In The revolution of the ants from Bernard Werber, some ants search an emotion called "laughter".
The fact is, well, ants live for food, servitude and mating so there is no place for humor in their lives. But when they discover it, the author tells something like this (I'll get the book this evening to have better quotes):
The ant was shaking with spasms
It has to be noted that in a previous book (The ants), ants already knew how to laugh:
A lot of spiced pheromone bubbles went into the air.
Elsewhere, an ant was talking with humans and expressed anger. I recall that the ant was using a translating machine, but a transcription was in FULL CAPS. I don't remember how the machine detected this.
Of course it is a Sci-fi book and the ants' expression is generally described ("103th was sad...") and I don't know if one day Werber saw a laughing ant...