From a purely biological perspective, an offshoot of H. Sapiens is a distinct species when the two groups can no longer interbreed. A species can also be defined as the largest possible gene pool under a given set of circumstances.
From a psychological perspective, it's a bit fuzzier. "What does it mean to be human" is certainly one of the oldest questions that humans have asked and it continues on today. However, I think that psychologically a lay-human can identify another human based on certain characteristics of physical form or life experience. If a living thing fails to meet these criteria, it may be classified as "inhuman" or a creepy "something else".
Note that these norms are based on fairly open and accepting definition of what makes something human. There have been plenty of times in history where a particular group of people is defined as not-human then oppressed/slaughtered, for example, the German classification of the Jews as not human and therefore worthy of slaughter.
Physical form
The body needs to look something like this or the female equivalent along the path from baby to elderly. Missing limbs is okay because a human being can still live if it's missing an arm. However, missing a head while maintaining the ability to move is distinctly inhuman.
Physical weakness
An individual who does not behave in the way that humans do, in terms of endurance, strength or (in)destructability can be classified as not human. The further outside the bounds of normal characteristics an individual goes, the less human they become. Being very weak does not preclude classification as human since there are many very weak humans.
Mental weakness
It's a common thing to hear "...all too human..." when a character in a book makes a huge mistake or succumbs to a typical human frailty such as despair, madness, anger, or fear. Something that looks like a human but does not display any of these these emotional/mental characteristics doesn't usually qualify. Example: Super soldiers who fear nothing and feel no pain at slaughtering other humans could qualify as "not human".
Comparable Intelligence
Society is pretty accepting over very low intelligence to normal to even high intelligence but extremely high intelligence has an alienating quality that precludes definition as human.
Mortality
In Bicentennial Man, a robot slowly gains more and more human traits over the course of his 200 year life. All his organs and body become biologically based with the exception of his positronic brain. For the better part of his life he lobbies to be accepted as human but the global society continually denies him that designation until he permits himself to die. With that act of mortality he places himself within the bounds of human mortality.
If you can't die, you aren't human.
Progression through Life Stages
An individual who doesn't have a childhood because they were created in adult form could be considered kind-of human but kind-of not. It would depend on the specifics of the story to determine how "kind-of not" the individual may be. The movie, "The Curious Case of Benjamin Button" where Benjamin Button is born as an old man and ages backwards might satisfy the "not human" definition because his life is so radically different than anyone else's. Terminating a life before reaching a particular life stage is not enough to preclude defining an individual as human.
If there are substantial deviations from the normal characteristics in any of these categories, an individual or group could easily be classified as "not human".