The requirement of a compass behaving sensibly is that a planet has a reasonably smooth bi-polar magnetic field. Under such circumstances the lines of force always point towards one of the magnetic poles on the planet's surface, and therefore the compass always points in the same direction.
You also mentioned fauna. The Earth's magnetic field is a vital part of maintaining the biosphere because it acts to deflect molecule-shredding cosmic rays of varying sorts. Maintaining life without having feasible compasses thus leaves us with a problem to solve.
Non-working compasses and living fauna require either:
- No proper magnetic field, and highly radiation resistant lifeforms
- No proper magnetic field, and subterranean lifeforms
- Something funky about the magnetic field that maintains deflection of cosmic radiation & solar winds, but messes up the compass.
Things that could interfere with the a compass:
- Large, frequent, random deposits of ferromagnetic material causing localised distortions in the EM field (compass won't reliably point in a single direction)
- A multi-polar magnetic field. That said, this is a tough sell: we don't know if this is possible in the long term, or if the magnetohydrodynamics planetary bodies force a dipole. The planets in the solar system with magnetic fields are not like this
- The planet is iron-poor. There are no ferromagnetic materials readily available, so no society can invent the compass.