This is a classic philosophical question. You can probably find all sorts of interesting discussion of it around the internet, so I'll just talk about my view. I'll largely ignore travel into the future, because travelling into the future is something we already know how to do - just build a spaceship, get on board and accelerate as close to the speed of light as you can manage and let time dilation do the work. Building such a spaceship is left as an exercise for the reader.
Certainly, the lack of time travellers from the future has been seen as an indication that time travel will never be invented. Douglas Adams wrote in one of the Hitchhiker's Guide To The Galaxy books or radio episodes (or both), something to the effect of "time travel, by its very nature, is invented in all eras simultaneously". So either there is time travel, or there isn't, but you can't have a point where time travel is invented because once it's been invented people can travel back in time and someone in the past is going to figure it out and it just carries on from there.
That only works, though, if you look at one particular model of how time travel works. In Adams' work, time travel technology allows the user to move freely backwards and forwards in time, from the beginning of the universe (where they built a restaurant called the Big Bang Burger Chef), to the end of the universe (where you can dine at Milliway's, the Restaurant at the End of the Universe). But it doesn't necessarily have to work like that.
In the television series Quantum Leap, a time travel device is invented which only allows the user to travel into the past within their own lifetime. Mutation of history is entirely possible in the show's model, and indeed that's primarily what the main character engages in for largely altruistic reasons. In addition, the "leaper" actually swaps places with a targeted person in the past, taking on their appearance during the leap and basically impersonating them.
So in this model, not only is time travel limited to a human lifespan before its invention, time travellers are effectively undetectable (provided they impersonate people sufficiently well). Obviously this is a storytelling device to give the show interesting scenarios each week (they had a lot of fun sending Sam into people with no legs, pregnant women etc) but it does provide an answer to your question.
There are other such limitations. Rather than the lifetime of the traveller, the machine may only be able to enable time travel within a time period where the machine itself exists - so you can never travel back to before the machine was invented (or forward to after it was destroyed, if it allows you to travel forward).
This is shown in several things, but the film Primer stands out as an example of this for me. In that film, the time machine has to be built, turned on and left. Later on, you can get inside it and wait the same amount of time before getting out, and you'll emerge just after it was built (it only travels backwards at the same speed you usually go forward, so travelling back two months takes two months of sitting in an airtight box). So in that world, again, once the machine is invented you might see time travellers everywhere, but never beforehand, and because you travel back at 1 second per second you're severely limited by both your lifespan, your ability to cope with the experience of the journey, and the availability of a machine which someone else isn't already using and which retains power and integrity for the necessary time.
There are also the numerous possibilities of multiple timelines, such as the idea that travelling backwards in time immediately makes a new timeline which isn't the same as the one you originally came from, and that this limits the appearance of time travellers somewhat because they don't all go back to the same one and neither do they all come from the same future timeline.
Of course, there is also the possibility that there is a fair bit of time travel activity but that various people try to cover it up, regulate it, police it and possibly utilise more time travel to prevent the first time travel incident from ever occurring. This model is much in evidence in the Star Trek: Department of Temporal Investigations books, where a variety of "uptime" agencies are seen intervening to mitigate, cover up or outright erase from history various time travel incidents. However, in that setting many people are aware that time travel is possible and that it happens sometimes. They're just usually sworn to secrecy if it happens, or their memories are altered, or they end up dead.
