Lord Woodesey and I seem to have had the same idea, great minds think alike, so they say. There are several ways to have a deer antler helm without it being detrimental.
The easiest, and by far the simplest, idea is to fix the antlers to the helm and then cut through them near the base, leaving you with two nubs attached to the side of the helmet. Then, you glue the antlers back on to those nubs (glue did exist in Medieval times but you may also be able to use wax if you prefer).
This may seem redundant but, by cutting and gluing the antler, you have created a structural weakness. This means that, when force is applied from someone pulling it or a weapon strike, the antler breaks off where we glued it to the base. This prevents someone being able to use it against you but still allows for you to wear your antlers into battle. The plus side of this is you may also be able to re-glue the antler back on after the battle if you pick it up and its not too damaged. A similar thing can also be done with horns, using the exact same process as described above.
Metal though is a little bit different. Due to its weight, it would be a lot harder to use glue to achieve the same effect with steel or bronze, possible with modern glues but less so with Medieval ones. Instead, what we can do is make these pieces out of a lighter material such as aluminium or tin and weld the pieces together. This means they do not add too much extra weight, can be broken off easily due to the structural weakness and can possibly be repaired or replaced after the fight.
Rather than risk Lord Woodesey damaging the antlers (or whatever else he chooses to decorate his helmet with), we can instead have detachable horns. Essentially we can have something similar to arrow or spear heads. A metal fixture goes onto the side of the helmet and the antler or horn is pushed into that fixture (it has already been filed down to make sure it is the right size and a snug fit).
This way your antlers can easily be pulled out of their fixtures before battle, so they are not a hinderance, and put back in when not in combat. This also prevents them from being damaged or lost in a fight. You could even do a similar thing with metal.