between three and five kilometers under the surface of the sea
Well, that will cover up almost everything, I'd say.
There was a US nuclear test, operation wigwam, that involved a modestly sized warhead (~30kT) at 600m depth. There's some footage of the test on youtube. There was a big splash on the surface:
with disrupted water in an area a couple of hundred metres across, but no mushroom cloud, fireball, etc. A bigger bomb will be more dramatic, but you're talking about tests in several times the depth of water.
What you'll see from space will be some big, brief splashes, and some transiently warmed water. Unless you have seismic sensors or hydrophones on the planet itself, or radiation sensors in the atmosphere, there's no guarantee that you'd be able to identify that the event was caused by a nuclear blast even if you did manage to see it. You could probably tell that it wasn't caused by a meteorite strike, but I'm not sure you could rule out submarine volcanism.
Lets assume you have a nice diffraction-limited telescope with a collecting element of size $d_t$ that's looking at the right place at the right time in good weather. According to Rayleigh's criterion, you could resolve a feature with diameter $d_f$ in light with wavelength $\lambda$ at range $r = {d_td_f \over 2.44\lambda}$. A 20m aperture telescope would see a 200m feature in 500nm light about 3 million kilometres away, which is about 8.5x the distance between the Earth and the moon. A 100m aperture scope could see a 10km feature from half an AU away. That ain't too far on the grand scale of space... chances are good that the bomb-builders will see your starship braking into the system before you see their deepwater blasts, unless you have reactionless drives and fancy heatsinking. Remember also that if the thing you're searching for is about the same size as the resolution of your scope, then you're going to end up with "a blue pixel turned white for a few seconds and then turned blue". Not quite a smoking gun!
With atmospheric and surface tests there's a reasonable chance you'd be able to see the flash which will be quite distinctively unnatural. Humans had things like Project Vela that could detect nuke x-ray flashes from high earth orbit (100000km up) in 1963. Anyone capable of building and operating an interstellar spaceflight will likely have the ability to detect those flashes from much further away... the mere presence of a signal would be enough to raise eyebrows (or the spacefaring species equivalent thereof), even if it couldn't be localised to anything more precise than "probably that planet over there".
edit
As a footnote on short-wavelength (eg. gamma and x-ray) detection:
- X-ray telescopes do exist, as focussing x-rays is possible! We already have the technology to make x-ray optics that can work with 80keV x-rays (17.7pm).
- Gamma ray astronomy is a thing, but focussing gamma rays is impractical, probably even for starfarers, absent some magical new unobtanium or technology which is out of the scope of this answer. Shielding gamma rays is possible though, which means that observing a narrow region of sky can be done, and larger pictures built up by changing the orientation of the detector.
A gamma-ray detector on the CGRO (launched in '91, so it is old tech!) took this image of the moon from low Earth orbit using only photons with energies >20MeV:
It is an open question as to how far away one could be and detect the gamma-ray signature of a nuclear blast, but localising a signal to a particular planet would not be impossible, so long as that signature could be distinguished from background noise.
The Vela satellites used x-ray and gamma ray detectors in tandem as part of their nuke detection strategy, so clearly enough radiation can escape the atmosphere to be detected 100000km away by 1960s technology. You can extrapolate modern and future detection capabilities as you wish. Your spacefarers might not be able to detect the gamma ray signature, but a combination of x-ray telescope observations and visible-light bhangmetrology seems like it should do the job in spotting nuclear fireballs, or things suspiciously like them.
Nuke blasts that do not generate a visible fireball will not be producing detectable amounts of short-wavelength radiation. This includes sufficiently deep underwater blasts and buried blasts.
edit 2
A footnote on antineutrino detection. The fission primary of a Ulam-Teller style staged thermonuclear bomb generates a hefty dose of electron antineutrinos. These are somewhat unusual on Earth, and as such seem like they'd be an interesting way to detect a nuclear blast.
Unfortunately for your spacefarers:
- neutrino detection is extremely hard, and detectors need to be very large.
- false positive antineutrino signals are generated by cosmic ray interactions, and require massive shielding to reduce even on Earth.
- anyone using nukes can (or soon will be able to) build nuclear reactors, which generate similar antineutrino fluxes to bombs going off
- the neutrino detector on Earth is not going to be more than ~12700km away from a blast on or under the surface. The inverse square law means that distant detectors will receive vastly fewer neutrinos from the event.
- the earth itself is hella radioactive, and produces a lot of antineutrinos. The sun, too, produces some. This noise makes it even harder to detect small distant sources like a small nuke going off.
- the starfarer's own spacecraft likely has some kind of nuclear propulsion and power supply, which will inevitably generate its own neutrinos of various kinds.
There's more information to be had if you read Antineutrino Detectors Remain Impractical for Nuclear Explosion Monitoring.
Between the first and last bullet points, you can see you'd have to build some huge detectors (maybe out of comet ice) and then fly away from them to let them work. And if you're doing that sort of thing, you'd probably be better off just building some bigger telescopes to observe the inner system instead.
