You want to turn to Russia, but not because they have a similar population density. No, you want to borrow their playbook because they are very good at area denial.
You said it yourself - the country has way too much land, and for the most part it's a barren wasteland. So it's not going to bother your citizens to make the wasteland even more barren. Dial up scorched earth tactics one thousand percent and lay waste to the perimeter of your country barring some controlled areas. Cobalt bombing your borders would be a good start.
Let's see those smugglers and poachers deal with 50-100 miles of irradiated, water-less, food-less, shelter-less, utterly inhospitable hell.
Beyond the Zone, and in the permitted corridors, your regular border guards pick off whoever survives.
The more resourceful criminals will try to bypass your Zone with vehicles. Air-based vehicles will be beyond the means of most, and are easy to detect, engage, and destroy with conventional military weapons, so we won't worry about those.
The biggest issue will be all-terrain vehicles. But Russia has that covered too - General Winter's most reliable subordinate is Colonel Mud. He defeated Napoleon's artillery, Hitler's tanks, and he will do the same to the trucks of your criminals. He does not need to destroy them, merely delay them until your border guards can detect the party and deploy a unit. The Russians' own strategy for dealing with muddy roads is to make enormous specialized vehicles but the bigger they build 'em, the easier they are to detect (and the more expensive they are).
Ultimately, no border control is 100% effective, but layering a nuclear wasteland, an impassable mud pit, and your regular guys with guns will make it hard for all but the most determined and wealthy criminals to get through.
Once they get through, though, they will still need to actually do their business and leave. Ensuring that your nation is a poor environment for doing crimes will help deter people from even attempting to pass through. If all mining sites are properly secured, there will be no illegal mining. Same goes for grazing (or you can just torch the grass), smuggling (they have to get the goods to market, catch them there), poaching (keep an eye on your animals), etc.