I was wondering whether a planet might exist that has life forms with up to five sexes, with a single organism having three different and independent, functional sets of sexual organs. Presuming intelligent life develops on this planet, what type of social structure/roles may exist?
There's no biological reason that multiple sexes couldn't exist, but there are biological reasons to expect only two sexes in most situations. It can be tricky enough for a male to find a female to mate with, throwing in another factor as well could make reproduction harder.
This is a concept that's been explored a number of times in sci-fi already. My favourite is The Gods Themselves by Asimov, which describes a species in another universe that consists of three sexes which mate by 'melting' into a single entity, with one impregnating another while the third provides the energy for the process.
EDITED TO ADD:
With the clarification of what we're looking for - a species with five sexes, and each individual having three of them - it's hard to imagine how such a species could possibly evolve. It's just too complex and requires too much careful control over getting individuals together to breed. We'd need a reason for why such a quirky system emerged, and that reason is likely to inform the social structures as well. It's also going to depend a lot on what these five 'sexes' actually do, biologically speaking.
It also seems very likely that these organisms would be able to change part of their sexual equipment, the way some complex animals can on earth, to suit their surroundings. That's going to change things as well.
We could imagine something of a caste system in place; rather than having fairly soft gender roles, as most human societies have, we might see that an individual is fitted into a caste and lives and operates according to those rules.
We might find that all individuals are born with the arrangement of, say, 1, 2, 3, and start out in the lowest caste, doing manual labour. Those who rise to the top of the social strata in that caste might find themselves morphing into 2, 3, 4s, and able to climb to the next caste. Or perhaps those who are particularly prone to fighting will become 2, 3, 4s, while those who gain social rank by negotiation would become 1, 3, 4s, and those who learn their way out might be 1, 2, 4s, and each would change into the appropriate caste (Soldiers, Leaders, Scientists). Repeat for each new level, until you have a single 5 at the top of everything.
What Werf said is true, but I want to follow up on it more. There are two main reasons this is unlikely.
Threesomes are overrated, multi gendered species won't evolve
Part of the reason why there are unlikely to be more then 3 sexes is because of gene inheritance. Evolution is all about making sure as many of your genes as possible exist in the next generation. When you mate with another sex the resulting child has your mates genes as well, meaning it has fewer of your genes. This is a big sacrifice with sexual reproduction, putting effort into making a child which won't have all your genes.
Now it turns out sex has many powerful uses, the ability to produce young that can quickly adapt & evolve is very important, enough that it's worth the heavy sacrifice of passing only half your genes down to each child in order to have sex. However, once you have a source of extra genes there isn't much advantage in having a third sex. Having 3 sexes means that you only pass on 1/3 of your genes per child, with two sexes you pass on 1/2. That means that each child has 1/6 less of your genes. The extra genetic diversity of another sex helps minutely, but not enough to justify the sacrifice of passing on less of your genes per child. The end result is sexual reproduction is likely to be limited to two sexes to maximize the number of genes passed on while still having some means of recombining genes.
If you want to have three (or more) sexes you pretty much need to ensure either one sex is a relative of one of the other sexes, benefiting from helping relatives to pass on their genes, or that each sex puts in exactly the same amount of effort, and pass on the same amount of their genes, into raising the children and more children are produced, so that you end up with the same amount of genes passed on by having more children (due to the added contribution of a third parent) to compensate for each child having fewer of each parent's genes. I discussed some options here: How to handle a tri-gendered race
No hermaphrodites allowed, Pick a sex and stick to it!
Second there is the hermaphrodite state, that one individual has more then one sex. This is also unlikely in more complex species, particularly sapient. Generally speaking it's better to specialize at being the best at one sex then to generalize to being competent at both when you can only play one role at a time.
To give an idea why lets look at the male/female dynamic and imagine a hermaphrodite mammal, assume they have no difficulty finding others of their species for now. Generally it takes far less energy to play the role of 'male' then it does 'female', it costs very little to produce sperm, quite a bit to be pregnant, birth a child, and raise it. Thus your hermaphrodite would prefer to play the male role then the female role.
Thus every one of our theoretical mammals will compete to be the 'male'. The 'stronger' ones, the ones able to win whatever competition is used to decide which individual plays male or female role, will be a male more often and spread their genes far more then the ones that always 'lose' and end up the female, since the ones that win will be able to mate with lots of others while the ones that lose are pregnant and only manage to bear a single young. Thus the ones that are good at being males will produce lots of children that are also good at being males.
For those that are best at being males they will not want to be females at all, being pregnant will hinder their ability to compete as a male (try wining any competition while 9 months pregnant), thus the inconvenience of pregnancy could cost them multiple mating opportunities as a male. Given enough time some mutation will come along that actually makes it easier to compete at being a male while making playing a female role even harder. For instance imagine a mutation causing more testosterone to be released, increasing muscle production but making it harder to carry a child to term. For those of the species that are already strong males this will be a welcome mutation, making them much better at out competing others for the preferred male role now that their stronger while sacrificing the ability to get pregnant that they don't really want if their good enough at being males. Those with this mutation will father more young that also have this mutation, causing the mutation to quickly spread.
As time goes on similar mutations will keep happening, those that already have mutations that make them better at the male role will keep looking for mutations to help them at this role, increasingly accepting sacrifices in their ability to play female role to be better at the male role.
Meanwhile those who aren't specialized as males will be unable to compete and win matings as a male. With such a low chance of successfully mating as male they won't bother competing if the competition requires expenditure of resources (as most competitions do!) since they will just waste resources to lose. Instead they will start evolving the other way, sacrificing their ability to be a good male to be even better at being a female, producing more young, having them faster etc.
The net result is you end up evolving towards two separate sexes, male and female, because it's better to be the best at your chosen sex then to be adequate at either, which results in never managing to mate as a male since you can't compete with those specializing at it and also being a subpar female.
Generally hermaphrodites only exist in species that are heavily isolated and/or it's difficult to find another of their species, where ensuring that you can mate with any you find to ensure you can have children at all is more important then specializing in one sex only to fail to find a member of the opposite sex to mate with. These species also tend to be less complex (read unlikely to be sapient) since their isolated nature, lack of competition for mates, and occasional self-fertilization all slow the rate/spread of novel adaptions a bit. Here is a question of mine looking at a way of encouraging sapient hermaphrodite species, though with limited success in finding a good answer: How to justify evolution of a sapient hermaphrodite species
My best handwave justification The net result is that your species seems quite unlikely, since both multiple sexes and being more then one sex at a time are unlikely to evolve making the evolution of both traits quite unlikely. In addition you have the difficulty of finding mates with the appropriate sexes as mentioned by Werrf.
If you want to justify this...well the best justification is a designer race, something created rather then naturally evolved. Failing that I would say to make this happen you would need to ensure that every sex is equally involved in producing the child (unlike most species where females usually put in far more effort then the male), likely with either all parents collaborating to raise the young in a monogamous relationship (most likely to lead to sapience), or each parent taking a young away to raise individually. You could then claim that each species has multiple sexes as a way to make it easier to produces a mating 'set' of parents by helping to ensure each individual can mate in a number of different roles to fill whatever role is needed in a mating set.
However, to be frank it's quite improbable this would every occur. The above helps to justify it slightly, but I doubt I could ever really accept anything other then some sort of genetically engineered creature having this trait, it's just way to complicated and counter to some basic concepts of evolution.
Ian M Banks explores a similar idea in his book The Player of Games, which has a race of humanoid with three sexes. The "Female" produces the eggs and carries the fetus to term, the "Male" provides the sperm, and the "Apex" essentially acts as an interface between the two with an invert-able set of genitalia, and is essential for fertilization.
The society is caste-like Empire, with the "Males" working as soldier and manual labourers, "Females" as housewives and service jobs, and "Apex" as the ruling class, priests and officers and suchlike. However the author makes it clear that these roles are not necessarily genetic but have their roots in the historical culture of the Empire (I highly recommend the reading this great book for more details).