Bullet proof vest are effective because they dissipate energy, not because they are tough.
Toughness is the material's ability to resist being fractured.
Lets say you have armor that is very tough but doesn't dissipate energy, like plate mail. That will stop most attacks, but eventually the attacks will break the toughness of the material and go straight through. Most bullets can do this to human sized armor.
If the energy dissipates then the bullet needs more energy to move the vest more to have enough energy to break through. Because of this lighter weight polymers with good dispersion are preferred to tougher materials that don't disperse energy as well.
However, you don't need every part of the vest to be strong enough to stop the bullet on its own, since the force is pushed over a larger area. If an average bullet exerts 1 kilonewtons and Kevlar is tough enough to withstand 2 kilonewtons of force exerted over it, making armor that can withstand 5 kilonewtons out of materials that don't dissipate as well isn't worth it. Both vests will survive, but one will dissipate worse and be more expensive of heavy.
Weight and safety concerns
Furthermore, Kevlar is half the density of diamond, with more tensile strength. Replacing Kevlar with diamond will increase the weight of the bullet proof jacket to potentially more that double its current weight. The armor could be cut down on weight reducing the dispersion even more, which would lead to other problems. Furthermore, with less dissipation the jacket will still cause more damage to the wearer as the same force is applied to the user over a smaller area.
All in all, making armor tougher doesn't always make it better.