Let's suppose you are traveling at 10% of light speed, and you need at least 0.1 seconds to react to a potential collision. That means you need to be able to spot a cold rock at a distance of (0.1 x 3x108ms-1/0.1s) = 3x108m, or three hundred thousand kilometers.
The equilibrium temperature of objects in deep space is 2.7K. But for argument's sake, let's imagine the rock you're about to hit is at a relatively warm 5K. Let's also assume it has a surface area of 105m2.
Per the math on the Wikipedia page, the rock will have a black-body emission peak of 294GHz, and its power output will be 3.54W.
If you have a giant infrared microwave sensor with an area of 100m2, then at a distance of 3x108m it can detect, at most, 3.13x10-16W of power from the rock. To put it another way, at 294GHz, that means you're detecting just 1.6 million individual photons per second. Which is not impossible, but very difficult (thermal cameras have serious noise problems because the camera itself is not at absolute zero).
So, it might be just about possible to detect objects passively, if they were the size of a city block, and fairly warm, and you could destroy or avoid them in a tenth of a second.
But at 10% of light speed, you could be obliterated by an object the size of a peanut*, and if an object had been in deep space long enough it would have cooled to the temperature of the cosmic background, making it thermally invisible no matter how big it was.
Personally, if I were on an interstellar ship, I would insist on it having active sensors (radar or lidar). If that's absolutely forbidden for narrative reasons, then I don't think you could avoid all collisions, so you'd need to be able to survive them. You could have an "icebreaker" ship flying just ahead of you, basically a huge metal asteroid with engines, and if something hits it you have time to hit the brakes before you rear-end it.
* Edit re: relativistic collisions
Why do I assume a micrometeoroid will not pass right through the ship, as it would at orbital speeds? For objects in Earth orbit, we're talking about rail-gun speeds (~105ms-1), but for interstellar travel we're talking about speeds a thousand times greater, closer to what you'd see in a particle accelerator.
Suppose a pebble passes through a spacecraft at 10km/s, and its temperature increases by 100K (which I think is a low estimate). It might melt and fly apart, but at that speed it'll be out the other side of the spacecraft by the time that happens. Now, suppose the same pebble was traveling a thousand times faster. It hits the same number of molecules on the way, but each of those collisions is a thousand times more energetic. Loosely speaking, by the time it passed through the spacecraft its temperature would have increased by 100,000K. At that temperature, it's not held together by chemical bonds; it's a collection of highly energetic charged particles radiating in all directions, so most of the (considerable) kinetic energy is going to end up transferred to your spacecraft as heat.
In some ways that is good news; if you did see a large obstacle in advance, you could pretty much blow it to atoms with a .22 rifle. Although, given the constraints above, the only way that'd happen is if you were constantly accelerating towards a big bright nebula or something, and could see objects occluding it.