If an advanced alien life form were to visit Earth, given the hypothesis that baring/showing one's teeth might be universally considered a sign of aggression -- would it make sense to avoid smiling at them?
Yes, but you've got to wonder what sort of a species they're going to be if they take it personally when you do.
In your question you make the very clear assumption that baring teeth at this alien species is to be considered a sign of aggression, so you clearly know this in advance of meeting the aliens. You've also stated they're 'advanced' and I'm assuming they're spacefaring, which means that if they think we're starting a fight, they can hit back with enough force to autoclave the planet.
In that context, giving them a big, cheesy grin would be akin to deliberately trying to piss off a foreign dignitary with the power to level your country. You shouldn't smile, not because it's going to do bad things, but because it's polite.
On the other hand: these aliens are advanced. If they're coming to us, then they should be able to realise that our social norms are not going to be the same as theirs, and that some customs and behaviours that they take for granted are going to be different here. Any race that's developed enough to have spaceflight should be able to take a breath and realise 'oh, it's OK, he isn't actually trying to start a war, he's just saying hello'. If they actually do take offence, they aren't the kind of aliens we want to be associating with!
I wouldn't expect us to start nuking ships in orbit if an alien ambassador descended into the White House and took a leak on the floor of the oval office. I'd expect we'd try to work out if we should do the same when we visited their houses first. I hope a more advanced race would feel the same way.
TL;DR: Don't smile, but don't expect the aliens to go thermonuclear if you do.
Of course, all of the above is based upon the predicate of the aliens actually being in any way like us. If they're just... well, alien, then it's anyone's guess.
I think smiling at aliens is a good idea, if the aliens have observed humans for a while before the contact. In that case, they are well aware of a correlation between a smile and non-aggressive behavior, and should expect us smiling.
If they just have crashed and you're the first human they've ever seen, smiling is probably ok too. The aliens must realize that their cultural norms do not apply to you. Human teeth (generally) do not look too dangerous, so the aliens are likely to guess that you're not going to attack them with your teeth.
I would go with the corny phrase, "you have to be true to yourself." If you feel like smiling with them, smile with them.
Human gestures are amazing things. We use them to communicate things at a velocity that our more conscious communication mechanisms cannot keep up with. These gestures are the ones that we claim allow one to "peer into the soul."
It makes sense that you would want to approach smiling at an alien accordingly. If you believe it is a good idea to show these aliens that which makes us human, go for it. If you feel the need to protect that bit of humanity from the aliens, don't smile.
I could see a case where one has started to smile and subconsciously realizes the alien is mimicking you, perhaps not with their mouth (do they have mouths?), but with some other set of muscle movements. You have hundreds of thousands of years of instincts bound up inside you. Are those alien muscle movements friendly or hostile? You may not get the right answer, but given that you may have a scant few milliseconds to either hide the smile smoothly under another gesture or smile broadly, it's those instincts I'd put my faith in. They're the best we've got for that sort of thing.
I would, however, avoid a false smile. Either smile genuinely, or not at all. A false smile not only doesn't show the soul of our species, but it also bars the teeth. Now you have all the disadvantages of a smile, but none of the advantages. That's a poor trade in my book. They didn't come all the way to our uncharted backwaters of the Western Spiral to observe enamel and dentin. They made contact because they feel there's something to be gained from this contact, and it's certainly not our technology. We have found over thousands of years that humanity is something we can give freely. We should offer it, and see if they accept.
This recording was made shortly before the author was involved in an unfortunate accident involving a bull in musth. It would appear that giving the sign of the horns to a bull does not have the same effect as giving the sign of the horns at a Black Sabbath concert. Many post-historians surmise that, in retrospect, heavy metal concerts may not have been the best places to find gestures that inspire a "connection between kindred souls."
Baring teeth is a threat in the animal kingdom because teeth are weapons. For a gorilla it is similar to brandishing a gun. However, human teeth are no longer effective weapons and so I find it unlikely that aliens would be intimidated by a show of human teeth, or even realise that they used to be weapons.
Overall I would say it is a good idea to smile at aliens. If they have studied human behaviour then they will expect to see smiling as a sign of friendliness and if they haven't they are unlikely to see it as a threat.
It seems to me that any organism (or machine for that matter) capable of traversing interstellar distances will be acutely aware of the difficulties of communication (even between individuals that speak a common language, let alone between species or civilizations). Consequently I would expect them to be very forgiving when it comes to an inadvertent faux pas.
Imagine the most brilliant people you know getting upset because a chimpanzee or a bonobo flipped them the middle finger, or called them a name in sign language — it just doesn't seem likely.
It depends on how much the aliens know about humans. Say, for example, that they know the anatomy of a human, but not our culture. They know that we have somewhat sharp, bony, bacteria-clad outcroppings in a cavity that we can open and close at will with some force. No matter how "alien" they are, if they are fleshy on the outside, that's threatening. If they know nothing about people, they have no idea teeth are hard. If they know that a baring of the teeth (ie, smiling) is a polite thing to do in most human cultures, no sweat.
Smiling may be misinterpreted by an alien culture - the difference between a smile and the aggressive baring of teeth might not be clear (if the latter is indeed something the aliens would assume). However, I would expect that the possibility of misunderstandings will occur to both parties, and so each side will be correspondingly cautious about reacting prematurely.
When humans need to communicate and lack a common language, or have any other communicative difficulty, we tend to default to nonverbal interpretation (body language, crude gestures, facial expressions). These are treated as more basic, and more translatable than verbal language. This tends to be true for dealing with animals, as well as other human cultures, since body language and especially aggression and threat tend to be fairly intuitive, and reading them (for those on our own planet) are a survival mechanism.
So, even though we know, really really know, that aliens might not share body language or nonverbal signals, we will probably be trying to fit whatever they are doing into our expectations, as they will be trying to translate our own signals. It is a survival mechanism, to try and figure out the threat level as best we can. I would expect this kind of thing to be a possible problem mostly during first contact, when both sides are trying to figure out how to communicate and if the other side is hostile or friendly. Once there is some common ground for communicating, it would make more sense to ask before taking a gesture as a threat, although miscommunications may still occur.
Smiling is an easy point to pick at, since it is already close to an aggressive signal (baring one's teeth). There might be other signals that translate poorly, though - showing hands is a peaceful move for us (look, no weapons), but may be an aggressive one to a species that uses their hands to fight with directly - ironically, something that occasionally comes up in fantasy in reference to mages. Something I read a long time ago had a species try to signal non-aggression by showing their unarmored bellies - which backfired a bit, since they did so by rearing up, not rolling over, and so scared the people they were trying to communicate with. Eye contact is sometimes aggressive, something like a challenge, and sometimes a friendly connection. Crouching down might mean non-aggressively trying to seem smaller, or a prelude to pouncing. There are a lot of ways to misunderstand body language and gestures, even when we have previous experience to guess from - and with aliens, we won't.
So where does that leave us? We should probably not try to anticipate how body language will be received, it's too hard to anticipate without more knowledge. It's probably better to act as normally as we can, and adjust our behavior depending on how it is received. We will probably also try any signals we know of from a species that looks similar to said aliens (if there are any that look even vaguely similar). Pausing, or backing away are rarely seen as aggressive moves, and moving slowly and stopping any gesture that's received poorly (or reacted to strongly) may also help diffuse tensions. Mimicking is probably also a possible avenue of communication, indicating that someone is paying attention and wants to communicate.
As long as both sides keep the possibility of miscommunication in mind, any accidental mistranslation can probably be worked out peaceably.
(tl;dr: probably safest to not smile, but you could try to figure out a way to demonstrate first that smiling is a good thing, perhaps with some video of human interactions)
This is a late answer, but others are going off in many directions with assumptions that I don't think can be made, so let's try to rein this in.
Even an advanced, space-faring, extra-terrestrial race which has catalogued and studied numerous other extra-terrestrial organisms very well could view teeth baring aggressively. I would assume that you have a roughly equal chance of it going either way.
The following situations have been well covered by other answers and therefore I will skip them: the visitors 1) do not assume aggression based on our teeth or 2) believe that our teeth are aggressive but also realize they might be mistaken and so carry on cautiously. These are possibilities, but these cases do not present the problem of your question, so let's move on to the opposite reaction.
There are a few more cases to consider..
We are compared to the animals around us
Among animals, showing teeth often is a sign of aggression. If I wolf shows me its teeth, I will likely take a defensive posture or run.
If our visitors are studying the creatures of our planet, and they see "teeth = aggression" is the norm, they have no reason to assume we are any different. In fact, if you look at many real, modern documentation of supposed E.T. encounters, you will find that some people have claimed to be abducted by aliens, that the abduction-subject attempted to greet the aliens peacefully with a smile, offered hand, or a "Greetings!" only to have the aliens appear frightened and back off.
That's right! We already have claims in our real history that people have encountered this exact problem; we try to greet aliens with our show of respect to have it taken offensively.
I am still undecided about the whole UFO sightings and ET abductions thing, so obviously take this with a (few) grain(s) of salt. Unfortunately I have no citation link for this information since I am recalling it from TV news and science/history channel stuff from many years ago.
Still, I can easily see ET study a rabbit, get bit, learn a lesson. Study a wolf or bear, see their reaction before they tear another animal apart, learn the lesson further. Then try to use the knowledge they've gained to try and communicate with humans; human then smiles - oh no they are going to try to bite us too!
One more note on this case before I move on: note that we could also be misinterpreting the visitors' "offended-ness" in the same way. We smile, their eyes go wide and they jump back so we think they are offended or scared, but really in their culture wide eyes are a sign of "I'm paying extra attention to you because I'm being polite" and a jump back is nothing more than "our greeting is complete, so now I'm getting out of your personal space." You never know.
ET may have a bad temper
There is no reason to assume advanced = level-headed.
There are plenty of humans who will take offense at any slight cause, whether real or imagined. This does not address the teeth specifically, rather is a general case which includes the teeth issue among others. I have known many intelligent people to ignore basic wisdom. It is possible that the visitors might all be wise and level-headed, but let's not make the assumption that they are. Let's start by treating their attitudes and reactions similar to humans; no reason not to, especially since humans are extremely varied.
Even if teeth are not taken aggressively, showing the teeth in a smile could still get negative reaction anyway. Let's go over a specific example case...
I am the commander of the E.T. visitors. I have a high rank and do not tolerate anyone who does not show me the respect I deserve. My subordinates have studied your planet for decades now, ever since the first encounter at Area 51 which your government still denies (whether I know, or care, that your government kept this secret is irrelevant). Now, after going over all the paperwork flowed up to me of yet another planet containing intelligent life, I have arrived to supply an offer and paperwork for annexing Earth to the Galactic Empire. I arrive and head to the U.N. headquarters. Never mind that the previous encounters with us were done discreetly with only the U.S. government, I'm not aware since I only skimmed quickly through a tenth of the paperwork on my desk. I arrive and meet a U.N. ambassador who believes this is First Contact. I quickly state "I am Sir Arg, here to offer you the opportunity for Earth to become Galactic Empire Planet Number 3274598. Please note section 43263; you will be expected to provide us with half of your planet's entire raw resource production starting no later than 2 of your years after signing." Being the wrong governing body and having no way to translate, the U.N. ambassador has no clue what I said, smiles at me and responds "We accept you in peace." The ambassador waves.
Now, take your pick, it doesn't really matter: the smile/teeth, the wave (seems more likely), whatever. I'll pick the hand wave.
"How dare you give me the hand gesture that means 'Get out of my office at once, and I will call you back when I am ready to speak with you'! I retract the offer, and I will be filing a lawsuit against you for harassment since this is a violation punishable even without your admittance into our empire. The damages I will be seeking are 100Kg of antimatter for the cost of my fuel to get here and for your publish apology, punishable by death."
I could give another similar response but for the teeth since that was your initial question: "How dare you suggest that you might eat me! I will be filing a grievance against this planet immediately to have your kind exterminated for your violent tendencies."
Now the folks who are actually supposed to interact with the visitor arrive. "Sorry sir, please forgive these people. They did not even know you exist. We are ready to begin negotiations. (gives a hand wave, except with the arm going down toward the ground instead of up overhead since that is the proper gesture the visitor expected)".
"The deal is off. (insert angry ranting here, whatever you might expect an unreasonable human to say who refuses to be civil)"
It is possible, even likely, that the visitors will be level-headed, understanding, and forgiving. The odds are still quite high that they may not be. I have had people be angry at me for something I did even when my action was an attempt at being helpful or polite, and I have had people refuse to see my action from the point of view that I conducted it with. You probably have too.
I think the question is difficult to boil down to merely "should we smile at them or not?" and requires a lot more planning, including contingencies. However, if the question needs to be answered as simply and generically as possible, I would do it this way: if you can somehow demonstrate that smiling is a good thing then demonstrate and then smile at them, but if you cannot reasonably demonstrate this ahead of time it is probably safest to avoid the smile.