Depending on the limitations or advantages of your particular FTL/hyper travel method, there are a number of different tactics that can be employed.
Parallax and astrometry
Provided that you can travel through the system at about 4-5 c, and that you have a way of looking out through the canopy while at low FTL speeds (similar to the Frame Shift Drive or Supercruise mode of the Elite universe), you can observe some of the planets against the stellar background by eye. No additional sensor systems will be required for this tactic. This tactic was the main method of discovering new planets in the initial versions of Elite: Dangerous.
Provided that travel exceeding the speed of light is not available within a star system, travelling at a significant percentage of c, combined with cameras and a relatively simple AI, you'll still be able to discern most, if not all planets in the system.
In this case, you need an array of 360 degree cameras, as well as an AI/computer system capable of discerning the moving planets against the stellar background. Even without initial telescopic observation, it can aid in pinpointing the approximate position and distance of bodies in the solar system for further observation.
The main advantages of this tactic, is that the ship does not need to remain stationary while scanning for bodies (in fact, movement is required to speed up observation). Depending on the image resolution and computational power of the ship's hardware, a preliminary scan can also be performed relatively quickly, whether the ship is positioned inside the star system or outside.
Additionally, the star system can be scanned using multiple wavelengths, including infrared, ultraviolet, etc.
Traditional RADARs work by bouncing bursts of radio signals off of nearby objects and listening for the "echo" or reflection of the same signal. This can allow the user to approximate the distance of an object, as well as determine the approximate position of the object.
The main advantage of this, is that existing (hyper)comms systems aboard the ship may be (re)used for this purpose, which can help save on ship mass (antenna systems are usually lighter than complex camera systems), and you'll have the additional benefit of being able to "filter" away more distant objects (eg. other stars).
Traditional radio is, however, limited to the speed of light, meaning that each scan/ping might take minutes or even hours. In fiction, this could be counteracted by using a means of FTL communication with similar RADAR-like properties in that the signals are reflected from objects and planetary bodies.
Utilizing signals, waves or other effects created by the act of dropping out of FTL could also be leveraged for initial positioning of objects in the star system in a similar manner. Whether the exit from FTL travel is marked by quick deceleration or the emergence of the vessel from a higher dimension, the process is bound to create some sort of shockwave resulting from the loss of kinetic energy or the displacement of spacetime. Provided the shockwave bounces off of objects in the star system or can provide other effects, it can be used to aid in an initial mapping of the system.
Similar to (re)using the comms systems as a sort of radar, it can also plausibly be used in a similar manner to traditional radiotelescopy. This method doesn't require waiting for signals to reflect from the objects in the system, but instead on their ambient radio noise.
As with the RADAR tactic, radio telescopy could also possibly be implemented using FTL communications, provided that it has somewhat similar properties to normal radio waves.
Both the RADAR and radiotelescope tactic can be implemented on the ship in a more complex radio array of multiple antennae that can be configured for various purposes, whether it's basic listening and positioning purposes by pointing the antennae independently, or more direct and focused observation.
Other advantages include the ability to filter away other phenomena such as dust clouds, as well as the ability to spot intelligent/communicating lifeforms rather early.
Radio observation also has the added benefit of being able to reveal information of an object's composition that isn't revealed by visual observation.
The main tactic
My main tactic would consider that a complex telescope and camera array would add a lot of mass to the ship itself, so the main observation and pinpointing would likely be performed using radio techniques. Utilizing any sort of phenomena that could result from your FTL method could also be part of the routine.
Such a procedure could happen in a manner similar to this:
- While still travelling in FTL/hyperspace, the onboard (hyper)radio systems are preconfigured to deploy in a rough "observation/pinpointing" mode and pretuned to reflections of the FTL "shockwave" that occurs when entering normal space.
Depending on where the ship will exit FTL in the system (whether on the edge of the system or inside, eg. in the habitable zone), the onboard (hyper)radio antennae can be configured to "listen" in a broad "window" (think a 3D sphere segment) or a "full sphere" configuration.
- (Hyper)Radio systems immediately deploy upon exit from FTL, and the ship computer/crew await reflections from the FTL "shockwave" for the initial pinpointing of planetary objects. The ship computer will attempt to approximate relative directions of planets etc. and attempt to map out other valuable and helpful information such as the location of the ecliptic plane of the star system.
- Additional preliminary (hyper)radio scans are carried out on different (hyper)bands to refine the readings from the initial "burst" observation. Depending on where in the star system the ship is located at this point, as well as the technology employed, the initial observation can be done in anything from a few minutes to a few days. The main objective will still be to locate approximately where the ecliptic plane is in relation to the ship. This will be the most likely place where planets may reside.
- Sections of the star system are then more finely mapped according to priority, eg. by starting to map out the area of the habitable zone. The onboard telescope(s) may also be deployed in order to observe objects using parallax observations. The onboard computer continues to refine the "map" of the system.
- As objects are found, the crew or computer may opt for more intensive observation of an object or planet. The ship is rotated and (hyper)radio systems are configured in an "array" configuration similar to radiotelescope arrays here on Earth, and onboard visual telescopes, spectrometers etc. are focused on the object to determine all the parameters of the object/planet.
- Rinse and repeat the 5th step until most or all of the valuable objects are located and the onboard computer has a decent map of the solar system.
- Optionally opt for more detailed scans when the ship is closer to objects of interest.
Again, depending on the technology used, the crew could map out a star system in anywhere from a few minutes or hours, up to a few weeks or months. There wouldn't be a need to have extremely powerful equipment or vast amounts of complicated sensor equipment on board, and some existing systems (like the comms array) could potentially be reused or repurposed for observation and mapping.
The tactic is somewhat similar to current iterations of Elite: Dangerous, where the player initially scans the approximate position of planets and asteroids using a discovery scanner upon arrival in an unmapped system (which charges and sends out a sort of hyper-radio burst). The player ship then generates an initial map of the star system that also contains an initial approximation of the ecliptic plane, as well as star positions (in case of binaries or more stars in the system).
Afterwards, the player then conducts more detailed scans of planets, asteroid clusters etc. using a mix of radiotelescopy and visual observation. This detailed mapping is implemented as a "mini-game".
As the planets are mapped in detail regarding composition and type, the player may opt to perform surface scans from up close in order to fully map planets and their resources. This step is conducted by using probes in game, but other means using various sensor and camera equipment on board could also yield useful information from orbit without using disposable hardware.
Depending on how you implement FTL travel, there may be additional methods of quick initial observation or mapping on the star system. Mass and gravity may have an observable effect on your version of hyperspace/FTL, or an effect on the ship or FTL drive itself if it passes close by to bodies of a given mass. By reading and measuring those effects while in FTL, the crew may be able to at least deduct that something is present.
If operating on a "jump point" mechanic of FTL, there may also be conditions that are favorable to the formation of jump- and exit points, eg. the relation of the positions of the host star or one or more gas giants or other large bodies in the system.
Though FTL/hyperspace observation may not provide direct evidence of the presence of planets and objects, it might certainly be enough to provide a hint on where to start looking, which might further reduce the time needed to look for planets and other objects.