Torture is the last thing you should resort to.
Mutilation should never come into play.
People can and will submit to quite a few rules in the interest of a certain goal, or ideal, however you're shooting yourself in the foot by mutilating your own crew.
Initial Crew
The original crew of your ship will have several things in common:
- They are trained professionals
- They understand the importance of their mission
- They all accept that they are making a decision with far reaching implications for themselves, and for their children
- They understand that whatever their lives were like on Earth, life on the ship will be nothing like it
In order to safeguard the success of the mission it's only natural that much stricter rules, procedures, and laws would be enforced than anyone had ever lived under before. These people will understand this.
However, the longer the mission drags on, and the more their situation sinks in, the more people might start misbehaving. After all, the first 19 generations are nothing more than caretakers meant to get the "bus" to its intended destination. It's their great-great-great-great-....-great grandchildren who will reap the rewards of their work.
And to that effect, some pretty heavy duty indoctrination, and manipulation will be necessary.
Punishment
Each and every person is a valuable asset on board your ship. They have specific training - they are not easily replaced. You have no pool of "spare" professionals from which to draw new crew members either - you have to wait for kids to grow up. Furthermore, that crew member will most likely mentor his/her replacement.
Locking people up because they lose their marbles and pose a danger to themselves and others should be perfectly acceptable. However, maiming that crew member, or killing him should be an absolutely last resort.
Instead, the focus should be on rehabilitating that crew member, with brainwashing, and other psychological manipulation being perfectly acceptable means of dealing with the psychoses which will invariably arise.
Human Nature
Frankly, I think you're pretty screwed as far as maintaining control of your crew is concerned. It's human nature to seek privacy, to buck authority, etc.
There will be places on the ship where people will build their own little unsupervised spaces. Where they will have hidden contraband, or meet to mate with someone other than their assigned partners, etc.
Teenagers will rebel against their assigned roles on the ship, and the injustice of having that future chosen for them.
Power-hungry ######## will try to over-throw the leadership and gain command & control of the mission (practically guaranteed to happen sooner or later).
Contraband will develop, with people who have access to the stores of supplies skimming off the top and trading it for various items or favors.
Additionally, you will also have to deal with Command crew members using their perks and privileges in illegitimate ways - such as detaining people they perceive to be their competition in some way (for their position, or for the attention of a mate, etc.)
Maintaining Control
You have to be able to monitor these people at all times. I would suggest implants, with a monitoring AI who acts as people' psychologist and conscience at all times.
This AI would not report all transgressions to Command ... only the serious ones, so that people would feel that they can trust and confide in it. The AI could also be a master manipulator, being able to either delay or confuse people when they have dark thoughts, or simply convince them that their course of action is not a good one.
The reason I advise such an approach is because a human crew cannot realistically monitor everyone's implant feed - you'd need more watchers than crew. Humans are also prone to corruption. But an AI, as long as it itself does not go nuts, is more or less the "deity" that generation after generation can look to for guidance and enlightenment.