When we talk about "manipulating time" from a physics perspective, typically we are referring to things that are well modeled using a model that permits manipulating time. Science is not very good at making statements about the truth of reality, merely statements which appear so astonishingly close to the truth that one can believe they are true and get away with it. One of the easiest ways to weave magic with science like this is to define a mechanic which refutes some fundamental assumption of science, but does so gently enough not to break the world.
A "time resistant" material might simply be a material which is not well modeled using our usual approaches. Because we're working with time manipulation, it would be fun to create a material which has properties which are not time invariant. Time invariance is a very important feature of science: it says that the properties science is studying do not vary with time. The results may change with time, such as an electric stove that gets hot as you turn it on, but the fundamental scientific principles of electricity and heat do not change with time.
You might have a material where most of its properties are time invariant, like any other material science is comfortable with. However, when put near a time manipulation field like your character has, it exhibits non-time-invariant behavior which makes it very difficult to move this field through the material. Think of it like an ooblick, whose properties appear to change when stressed in the right way.
Thanks to Nother's theorem, we know that every symmetry in Lagrangian equations has a corresponding conserved variable. That sentence is a mouthful, but the effects of the theorem are easier to understand. If you have time-invariance (the rules of physics do not change with time), you naturally have conservation of energy as a consequence of the equations. If you have position-invariance (the rules of physics are the same at all positions in the universe), you naturally have conservation of momentum. Let us assume that your time manipulator cannot violate the conservation of energy. Why? Because if you can violate the conservation of energy with your ability, all of science falls on its face*, and you said you want some resemblance of science. This means, as far as the time manipulating character understands, their abilities must be time-invariant. The two go hand in hand, thanks to the mathematics and Nother's theorem. If the time-resistant armor exhibits non-time-invariant behaviors when subjected to time manipulation, the conservation of energy appears to fly out the window. This means the character will have trouble understand what is happening near the armor when they use their time manipulation effects. The fundamental assumptions of the universe that they are making fail.
Of course, there's no reason to say this actually has to violate the conservation of energy, it just needs to appear so from the perspective of the time manipulator. Perhaps they feel the armor sucking them dry when they use the field near it, when in reality all it's doing is taking advantage of the fact that their understanding of time manipulation is not actually the real ontological rules of time manipulation. This would lead them to feel like it takes work to use the field near this armor.
You could pull a wave/particle duality stunt with this as well. If, for instance, the way the characters manipulate time is described using wave behavior, the armor may be manipulated in a way that is far better modeled using particle physics. This would be the same as wave/particle duality for photons or electrons: its not that an electron is a wave or is a particle, its that in some circumstances you can model it best as a wave, and some circumstances you can model it best as a particle. If the armor can exhibit behaviors which are best modeled as particles, anyone who assumes time manipulation effects work as waves will be sorely displeased with how things work. The results may be as counter intuitive for your characters as the Delayed Choice Quantum Eraser experiments, whose results are completely and utterly baffling if you assume photons are either waves or particles.
Interestingly, this also suggests a way for the time manipulators to overcome the armor. If the wearer of the armor starts to assume the armor always causes time-manipulation effects to behave as though marshaled by particles, there may be regions where they get confused as well. Perhaps there is a way to manipulate time such that the manipulation is poorly modeled by both waves and particles. In this case, one who understands the time-manipulation analogue to quantum physics would be able to do things that the armor wearer could never expect. Thus, the armor wearer might leave openings in their defenses, assuming the armor can do more than it really can. For an example of where things like this can go crazy, take a look at Bell's Inequalities. If you are assuming all of the funny behavior of quantum mechanics is merely us not knowing enough about the particles we are observing, you believe in a "local variable hypothesis," which has been proven to be an inadequate model of quantum behaviors.
Now none of what I have suggested here for the armor is actual physics. There's no QM involved . The links are all valid QM, but the applications with respect to time manipulation are all made up. However, these suggestions do take up the flavor of the problems tackled by QM researchers, so that flavor may lend credibility to your pseudoscience.
* Most of modern scientific theories are modeled using Lagrangian equations. Even in General Relativity, there is a conserved value we can call energy, though admittedly it is more complicated than our typical formulations of it.