What you describe is the properties of a large rock. Actually an infinitely large infinitely hard rock, but if your shield has any imperfections due to implementations, it probably would act like a large rock.
Any hard surface which does not move will cause the effects you are seeing. Rock is hard, and a heavy rock wont move very much. Accordingly, a sword striking this armor would have exactly the same effect of trying to fight a boulder with a sword. An arrow would be like an arrow fired at a boulder.
How do you defeat it? The answer to this is a question: "how do you defeat any armor based on some non-realistic physics?" It's trivial to create an armor which has no weaknesses in the presence of only physically realizable weapons. If you want to make an invulnerable armor, you can.
I can see a few issues. Unless you violate thermodynamics by having heat that can only travel one way, your armor is going to be the hottest wool blanket you've ever felt. Heat is high speed particle movement, so if it reflects collisions perfectly, its going to reflect all heat -- in both directions. The wearer will feel like someone wearing full Alaskan winter survival gear on a desert island.
Also, breathing is nice. Its not nice to reflect all of the oxygen molecules away from the wearer. This is actually a big deal for the force shields of Frank Herbert's Dune, which you may which to use as a source. He had shields which only turned on when something fast came at them. Below a threshold, the shield did little to nothing, letting oxygen and carbon dioxide diffuse across the border.
As a result, in Dune, there was an entire set of skills regarding how to fight with a shield. You learned how to move so slowly and smoothly that the shield didn't realize it was being penetrated.