TL;DR - Make your protagonist think in ways that are clearly alien thought processes, but are still logically self-consistent on some level.
Mental illness is defined by society, and tends to be very subjective. Just a few decades ago, homosexuality was listed in the DSM as a "mental illness." Even today certain sexual fetishes are listed as "mental illness" that seem more along the lines of societal taboos than objectively "bad."
However, an objective definition of mental illness is that it is a deviation in an otherwise normally functioning mind that is detrimental to or causes dysfunction in the possessor of that mind or his/her peers. Being depressed is objectively a mental illness, it causes pain to the person suffering it; gender dysphoria is objectively a mental illness, it is a thought process that causes pain to the person suffering it; schizophrenia is objectively a mental illness, as the lack of a proper understanding of reality creates dysfunction for the sufferer. These all have the common characteristic that they present as a person showing no outward or obvious pathologies of their physical bodies, but their mental states show something is clearly wrong.
Previously, it was assumed that mental illnesses did not have a physical cause. Ultimately, we are likely to find that this is not the case, perhaps even in situations where it is caused by environment stresses rather than genetics (for example, schizophrenia has been linked to brain dysfunction, it is possible to induce pathological apathy in a person through traumatic brain injury, gender dysphoria is likely caused by brain anatomy not being appropriate for the gender of the person's body, etc, etc...).
Humans are intelligent creatures, and we are not strictly bound by our genetics. In addition, the way a human is raised is likely to cause physical manifestations that affect their behavior.
It is then perhaps more appropriate to say that mental illness is merely a deviation from a physical "norm" that we are unable to detect as pathological. This is unfortunate as it likely implies all mental illnesses as we define them are idiopathic (unknown cause), which means it is all the more difficult for you to say your character's traits are nature or nurture.
The only way you're going to get any real traction on your question is to try to re-frame it in more objective terms. Look at how the human thought process works, and how you can make it more alien but still self-consistent. A common theme with mental illness is that there is a logical contradiction of sorts at play. I feel sad, but I clearly have no reason to, I feel as though I am male/female but my body is clearly not, I feel like the government is spying on me, even though it can be shown empirically that they aren't, etc...
Viewed this way, it is much easier to define it as a mental illness when one's perceptions do not match an objective, physical reality. They all represent dysfunction in what we would call common sense human thinking.
However, if you want your character to be "off" and seem "off" without being "crazy," you need to have him/her think from a point of view that is not human, but still consistent with reality.
One of the biggest problems that faces astrobiologists and xenolinguistic scholars is how we would communicate with an alien species. The entirety of our experience in communicating with others has been with our own species, and it is reasonable to assume that, since we all evolved together, that no matter how different our languages are, our thought process is the same. This is evidenced by the fact that a child can pick up any language, even languages as incredibly different as say, Japanese and English, or Chinese, and Swahili.
Contrast this with, say, a dolphin. It's generally assumed that dolphins are intelligent enough to have some rudimentary concept of culture and language, yet in all of the millions of years we've shared the Earth, no human has ever been able to "speak" with a dolphin, nor vice-versa. No matter how young a human or dolphin child is raised with the other species, it will never learn to speak their language, and is unlikely to understand their culture. This is the problem that xenolinguists face: Is it possible for two species who have evolved under incredibly different circumstances to have a frame of reference similar enough to enable communication.
All human languages, without exception, have the concepts of verbs and nouns. All human cultures have a very similar underlying concept of morality, which may even be objective. This is likely due in no small part to our genetic programming and our evolution as a communal species.
However, it is entirely possible, and perhaps even likely, that a species evolving differently may not have the "universal vocabulary" to communicate with humans, or we may find their way of thinking so different from ours we are unable to make sense of concepts they take for granted.
So if you want your protagonist to seem less human in an objective way that isn't tied to mental illness, look at how you can change his/her thought processes in a way that is still self-consistent and obviously not dysfunctional, but in a way that causes someone to say "I would have never expected a human being to think that way."
Have him/her reach conclusions in a way that seems chaotic and/or roundabout compared to how a "normal" person would reach them. Have him/her be able to think so many steps ahead that he/she seems to be clairvoyant. Think of ways to "break common sense." All of these will make your protagonist seem more "alien" and less human. A pig can never fly no matter how well it jumps, but if you see a pig flying one day, it's not necessarily inconsistent with itself or reality, it just means that it's not a normal pig, or that it's not a pig at all.