Boxcartenant has given quite a concise answer that is essentially correct; I'm just going to provide a different perspective on the same idea.
If you look at the laws of relativity as they were described by Einstein (and refined by many others after him), you come across a concept called Spacetime. The idea is that the space we inhabit is really 4D, not 3D and that we are all moving through time as a physical dimension. Once you get your head sufficiently around this idea, other things leap out at you.
For one, the only reason we can remember in only one 'direction' is because of the second law of thermodynamics, or entropy. The universe decreases in order globally every time we lay down a memory by increasing order locally in doing so; the extra disorder being the heat we generate by using our brains. Entropy is the only physical law that does not seem to be symmetrical through time, but that is a topic for another question.
The point is, you're travelling through time right now; you're just traveling forward at 1x speed like us all. Can you change that speed? Sure. people who are travelling fast (higher level of kinetic energy) go through time slower than those with low kinetic energy (travelling at exactly the speed of light would slow it to zero, theoretically). Can you reverse it? We don't believe so as it would violate relativity.
Another point to consider. Many people believe that Spacetime could be a static construct; that is to say that nothing in 4D changes. We only experience it linearly because we are designed to do so. If that is the case, time travel would not only be impossible, but it would be pointless. You'd be completely unable to change anything.
Regardless of what you see on sci fi shows, a time machine would have to be able to move in all 4 dimensions for the reasons mentioned in comments and the first answer above; the Earth rotates to create day/night cycle, and orbits the sun. More than that, the Sun orbits Milky Way, which in turn orbits the local group...
Bottom line is that even if you stayed static in space relative to (say) the big bang and traveled a couple of hours, you could end up in space, in the core of the Earth. Either one would be a touch less than pleasant.
If (on the other hand) you're talking about travelling through time and staying in place relative to (say) the core of the Earth, it's still best to think of that as movement through a physical dimension. The best description I've ever read of this concept for beginners is the last chapter of '6 not so easy pieces' by Richard Feynmann, which talks about spacetime and how it works as a physical dimension (but better to read the complete book first so you get a feel for the underlying principles).