A bunch of people have given great answers revolving around how there is "no stealth in space," but I feel like they might be slightly missing the core of the question.
How would, if possible, a space "radar" work?
Pretty much the same as on Earth. RAdio Detection And Ranging (aka radar) is a system that bounces radio waves off distant objects to see them. Radio waves, being just a specific slice of the light (EM) spectrum, can travel through space just fine. Actually, light travels slightly faster in space than in air.
Or how would you scan for Ships further away than your Optical sensors can see.
As I mentioned above, radio waves are light (outside the "visible" range for human eyes, but still light). Assuming you mean ships that are too far because not enough light will make it into your detector, then the answer probably is "you can't scan for them." Light is a pretty good way to see things. It travels at the cosmic speed limit. It's a wave that's also its own medium (so it goes through open space without a problem). The right frequencies of light easily interacts with most things (it's reflected or deflected by most things), so it's great for seeing things. And, it's easy to detect across a broad range of frequencies.
There really aren't any other mediums of detection as good as light. So if light isn't good enough for seeing something, you probably can't see it. For example:
- W and Z bosons are force carrier particles, like the photon. Maybe that means they could play in the same weightclass as light ...if their range wasn't so limited.
- Neutrinos, have a very long range, travel at about the speed of light, and pass through miles/kilometres of rock like it's nothing. Does this mean we can use neutrinos for some super penetrating form of sight? Nope. It passes through things so thoroughly, the Super-K neutrino detection experiment had to be built 1 km underground, as a 40 m by 40 m cylindrical stainless steel tank holding 50000 tons of ultra-pure water, etc. Here's a picture. It takes the analysis of super computers around the world to determine if any single neutrino particles have been detected.
- Gravity waves, a far reaching disturbance in the very fabric of space, is nearly impossible to detect, especially when caused by small objects. It took building multiple 4 km by 4 km observitories just to detect the gravity waves from colliding black holes. Those waves caused disturbances shorter than the diameter of an atomic nucleon.
The big space makes is semi impossible to look for the reflected radar rays. 0-100 Eyeball also does not work because of the hugeness of Space, same as all other optical installments.
You've just intuitively stumbled over the inverse square law. As Wikipedia puts it, "a specified physical quantity or intensity is inversely proportional to the square of the distance from the source of that physical quantity. The fundamental cause for this can be understood as geometric dilution corresponding to point-source radiation into three-dimensional space."
For light sources, this means means visible intensity rapidly drops off. This is why stars far brighter than the Sun are only specks in the night sky (at best). The principle behind this law is also why, as you noticed, it becomes very difficult (very fast) to survey everything within a 3 dimensional space as you expand the spherical area you're interested in looking at.
The inverse square law also means things can, in fact, hide in space, but only at great distances (several AUs at the very least). This is why we might have a proper ninth planet which no one ever noticed. Light traveling out to those distances and back would be extraordinarily diluted. Even the heat generated by a super-Earth to Neptune-sized planets at those distances would not be detected. This is why it'll be years before we confirm or falsify Planet Nine.
So, what does space-based detection look like?
- Radar (and other light-based detection systems) works in space but decreases in usefulness with distance.
- Active omnidirectional systems are only good for one's immediate area. Precisely how immediate/large that area is depends on the power output of the antennae and the sensitivity of the detectors. (I.e. you'd need to come up with numbers before someone could calculate the distances.) Whatever the case, we are still talking about how many km.
- Passive omnidirectional systems would be pretty good at seeing fairly distant ships if the sensors are designed to pick up heat. Even with the inverse square law, a computer-based detection system could spot above average heat spots moving through space. This means it's nearly impossible to sneak up on someone unless there is a large object between them (e.g. a planet or large moon). However, the inverse square law does mean you probably won't passively detect ships more than a few AUs out. As before, the exact distance will vary based on detector sensitivity.
- Directional systems (active and passive) will have far greater ranges, but you need to know where to look for them to be useful. (Note: active directional systems only work to distances in the thousands of km. Even the best Earth-based lasers are multiple km wide by the time the get to the Moon.)
- Given that most people in even the most advanced spacefaring society will live on or around planets, moons, and space stations. Directional detection systems will probably be useful for distant areas known to be populated. In that case, ships would have a difficult time sneaking out from home since a directional system could keep an eye on them as the move out into open space.
- Ships could try to (somewhat) counteract heat based detection by coasting as cold as they can through areas they're likely to be detected. They could also have a design that attempts to direct a good deal of their excess heat in a single direction (one where a detection system is not). Like conventional stealth, neither of these techniques would completely hide a ship.
- Conventional stealth would still be useful for things like radar. This is more important in close range, when active detection becomes an issue.
- At any significant distance, light travel time becomes an issue. Mars, for example, is many light minutes away. This means keeping an eye on ships a few planetary orbits out could mean you're looking dozens of minutes to several hours in the past. (Correspondingly, things take months to travel any significant distance.)
- Detection at great distances and behind obstacles like planets could be dealt with by putting orbital radar systems in place. (That wont help the transmission time, of course.)