Im a Detention officer currently. I can tell you right now that positive reenforcement (aka paying or rewarding good behavior) is infiniteley more effective than brutality. 90% of our control over inmates and maintaining of order within our facility comes down to 4 simple factors.
1: Equitable consistent treatment. Nobody is treated worse or better than anyone else unless they earn it. If a prisoner is in isolation and has lost comissary privledges he knows its due directly to his poor actions. If a prisoner is being afforded extra privledges it is directly linked to his demonstration of sustained good behavior. Nobody is punished or rewarded simply because theyre liked or manipulated extra favors. At all times inmates know exactly what thier status is and how thier actions have directly earned them that place.
2: The Inmates know they can rely on your word. You do what you say you will do when you say you will do it. Inmates know that when a guard says no its final, and when he says he will allow something or will do something on behalf of an inmate it is going to be done momentarily. If an inmate knows that there is zero variation in the reliability of the guard's word he will be much less likley to challenge the authority of the guards. The moment a guard fails to keep his word order begin breaking down because now the boundaries are no longer clearly defined and more assertive inmates will begin trying to re-define said boundaries in a way that is advantageous to themself.
3: Food. Seriously, humans do an incredible amount of thinking with thier stomachs. Providing meals that are nutritionally and proportionally adequate on time is probably our biggest security method. Virtually any fights or disturbances we have not related to mental illness in our facility are going to occur when theres some disruption to our food service. If a favorite and expected menu item cannot be delivered that day, or some breakdown in the system means food is late, lower than usual quality, or less than adequate portions we can pretty much begin counting down to the minute when the next violent outburst will occur.
4: Granting inmates who demonstrate obiediance, compliance, and dependability work privledges. Prison is boring, a chance to not only get out of your cell but also possibly earn some extra treats like more rec time, snacks, and bigger meals is an opportunity most strive for. Also, posessing access to areas most inmates do not such as the kitchen or janitorial areas is a status symbol. Inmates like to feel unique and trusted and tend to be seen as higher status simply because we allow them to wash dishes or clean toilets in a guard break area. They are trusted and allowed to go places other inmates arent and after a few years of "youre the same as everybody else" its seen by the institutionalized as a very big way to individualize themselves from the rest.
The idea that paranoia, brutality, coercion, and atrocity can be used as control methods are false. You will always be only one riot away from having your system collapse. Humans naturally resist such efforts to control them, in order to control humans (and quite frankly, I practice state sanctioned brain washing and mass-pacification on a daily basis) your charges need to feel like thier actions matter and that atleast the possibility of advancement exists. Seriously, by the time im done with most of these guys just the chance that they might not be allowed to spend an extra 3 hours a day out of the cell washing underwear or scrubbing toilets if they act out is enough to keep most of them in line. Zero violence, intimidation, or menacing behavior required. If you practice proper regimentation and de-individuation consistently and long-term enough you can literally lord scrubbing toilets over somebody as a treasured privledge to be earned or taken away and you dont even gotta beat anybody up.