Yes, you need to bring down the hammer with sufficient speed. But if there is no limit, not a problem.
I will explain this in terms of using a fusion initiated fission reaction (the reverse of the normal situation in H-bombs).
Step 1. cause a Deuterium, Tritium fusion reaction. The necessary step is to bring the atoms together with sufficient energy to overcome the Coulomb Barrier. Due to quantum tunneling effects, this is considerable easier to achieve that would otherwise. However, this still results in fusion temperatures on the order of 15 million degrees C in the sun's core.
But the energy required to overcome the barrier, is considerably higher than a mere 15 million degrees in the core. This works because gas particles follow a distribution of high and low speed particles, so the few particles that are 100 million degrees are fast enough to sustain the fusion reaction.
What you really want to know though is velocity, not temperature. Someone has already done the calculation and 3.15 x 106 m/s is enough for Protium/Protium fusion, which is about 0.01c, Deuterium/Tritium fusion is easier, requiring only about 70% as much velocity.
When Dueterium/Tritium fuse, the result of the reaction is in several forms, this includes a neutron at 14.1 MeV.
When this neutron strikes a U-235 atom, it will split (only about 7-8 MeV needed by the neutron to cause this fission). Mission accomplished.
What does our brawny hero notice? Nothing at all. The resulting 200 MeV output is about 3.2 x 10-11 Joules. i.e., energy equal to 3.2 microwatts for a span of 10 microseconds. The energy from a grain of sand falling 1 micrometer is larger.
The tremendous thunderclap from moving the hammer at relativistic velocities will be far more significant. Not to mention the newly generated mushroom cloud from all of the other fusion reactions that occur at the same time.
Note that the fusion reactions will be occurring in the air as the hammer approaches the target. Fine tuning the hammers speed so that only a single fission occurs at the target is unrealistic in the extreme, you are moving billions of billions of atoms around at relativistic speeds. The chance of only a single fission event is essentially zero.
Compare with the classic treatise, relativistic baseball.
This answer simply assumed the hammer though made of natural materials was magically accelerated (mixing fantasy and science is based on messy assumptions)
No hammer can survive the necessary acceleration. Assuming a generous 1 meter path for the accelerating hammer, it will require 45 trillion gravities of acceleration. For 0.6 meters, 75 trillion gravities.
Well, a quick calc, as a limiting case assume the hammer is simply a 10 cm cube of perfect carbon nanotubes. Mass = 1.6 kg, tensile load is about 1.2x1015 pascals (1.6 kg * 7.5x1014 m/s2 / 0.01 m2). Theoretical tensile limit of carbon nanotubes is about 300 GigaPascals. I.e, the tensile strength required is at least 4000 times the theoretical tensile strength. And this is based on extremely optimistic hammer design, i.e., the bending strength of a normal hammer is much less than tensile strength of the material.