Let's suppose you are traveling at 10% of light speed, and you need at least 0.1 seconds to react to a potential collision.  That means you need to be able to spot a cold rock at a distance of (0.1 x 3x10<sup>8</sup>ms<sup>-1</sup>/0.1s) = 3x10<sup>8</sup>m, or three hundred thousand kilometers.

The equilibrium temperature of objects in deep space is 2.7K.  But for argument's sake, let's imagine the rock you're about to hit is at a relatively warm 5K.  Let's also assume it has a surface area of 10<sup>5</sup>m<sup>2</sup>.

Per the math on the [Wikipedia page][1], the rock will have a black-body emission peak of 294GHz, and its power output will be 3.54W.

If you have a giant <strike>infrared</strike> microwave sensor with an area of 100m<sup>2</sup>, then at a distance of 3x10<sup>8</sup>m it can detect, at most, 3.13x10<sup>-16</sup>W of power from the rock.  To put it another way, at 294GHz, that means you're detecting just 1.6 million _individual photons_ per second.  Which is not impossible, but very difficult (thermal cameras have serious noise problems because the camera itself is not at absolute zero).

So, it might be _just about_ possible to detect objects passively, if they were the size of a city block, and fairly warm, and you could destroy or avoid them in a tenth of a second.

But at 10% of light speed, you could be obliterated by an object the size of a peanut, and if an object had been in deep space long enough it would have cooled to the temperature of the cosmic background, making it thermally invisible no matter how big it was.

Personally, if I were on an interstellar ship, I would insist on it having active sensors (radar or lidar).  If that's absolutely forbidden for narrative reasons, then I don't think you could avoid all collisions, so you'd need to be able to survive them.  You could have an "icebreaker" ship flying just ahead of you, basically a huge metal asteroid with engines, and if something hits it you have time to hit the brakes before you rear-end it.

  [1]: https://en.wikipedia.org/wiki/Black-body_radiation#Equations