I will try to draw together the best answers as I see it from the posts made to date.
The main barriers facing the authorities would be detection, capture and conviction. The fact that teleportation is so unlikely would at least seriously hamper each of these.
Two very important aspects that need to be clarified are the ability of the Teleporter and the seriousness / regularity of the crimes.
Petty crimes
Assuming the crimes are petty such as avoiding immigration controls or small scale theft, there is little or no chance that the Teleporter would ever be caught out side of the most serious blunders on his part (see below).
Major crimes
Assuming his crimes are major like rape, murder and grand larceny and are carried out on a regular basis, then the situation might be different. Given the enormity of the discovery of teleportation and the imbecilic use that he is making of it, it is reasonable to suppose that although he might be a genius, he also has serious mental deficiencies and might be expected to lack common sense, restraint and be generally careless. In this case there is a chance that he would eventually be detected and caught. Serious blunders might include:
Indirect prosecution
Stealing traceable property such as paintings, new bank notes, high value gem stones or similar. In these cases the teleportation would be immaterial as he would eventually stand trial for handling stolen goods rather than stealing the goods and would have difficulty in explaining how the goods had come into his possession.
Capture and prosecution by reason of teleportation failure
He might foolishly visit the same location repeatedly and be caught red handed before he could teleport. Or he might carelessly wander too far from the device only to find that it had been stolen or damaged while he was away or he might be unable to reach it if he was injured in car accident (for instance). In these cases he would also likely be prosecuted as the teleportation device is again irrelevant.
Capture and prosecution by aided by notoriety
He might simply be so careless that even if he can’t be caught he accidentally gives away his secret. After enough people and news teams have captured footage of his teleportation “trick” there would be a media sensation and his notoriety would be such that he would eventually be identified.
Having been identified he will either have to go on the run or stand trial. This is perhaps the hardest case and the most interesting one. If he stands trial then it is likely there would be conflicting evidence, however because of his notoriety (as the vanishing man or whatever) he might find it hard to get people to vouch for his being at a locality and any jury might be more willing to reject any claims based on sightings given the strength of the other evidence.
Going on the run
If he went on the run he would have access to anywhere on earth in seconds so might escape capture for some time, but would eventually fall foul of the teleportation failure case as described above and would be convicted.
Government involvement
Initially any Government would be dismissive, but at some point the level of his notoriety would reach a level sufficient to trigger alarm bells and National Security fears. At such a point the Government would be forced to take action. They might just assassinate him (out of scope) or try to capture him to learn his secrets which would happen faster with coordinated Government action and faster again if he had a criminal record already. When he was caught the Government would have serious issues to consider in terms of how to proceed. They would want to obtain the device and understand how it worked. But they might also be concerned that he was working for a foreign Government or had an accomplice. Such a situation would complicate matters further as it might prover impossible to hold him in any prison if a third party could open a portal into his prison cell and release him.
The evil genius with restraint and common sense
Assuming he did have common sense, restraint and was very careful, but still insisted on committing serious crimes (highly unlikely as described above), it would be very difficult or impossible to stop him, catch him or successfully convict him. He would always avoid repeat visits, would always take precautions concerning an alibi, would go in different disguises, would avoid leaving finger prints or DNA and would teleport to selected locations that minimised the risk of capture by reason of teleportation failure.
There are many other potential situations involving various combinations and degrees of motivation and evidence. The outcomes would be even harder to distinguish and even more open to opinion based answers.