The main problem is mass. Black holes are massive. The black hole engine will be more massive than the spacecraft itself. Also, while BHs are powerful the massive they become the less useful power can be extracted.
Since the holes mass millions of tonnes, any associated starship should likewise mass similar amounts, so the acceleration can be ~halved. The very smallest black holes might prove difficult to feed at the indicated rates, since they’re smaller than protons, so Crane & Westmoreland suggested using the black hole as a “battery” – a finite store of energy – and letting it push self and payload until just before its final explosive last few seconds. One problem is that the hole becomes very energetic indeed as it loses mass, so just when the appropriate time to EJECT is an interesting question. For every 10-fold decrease in mass, the self-acceleration increases 1000-fold, so a crewed starship would need either acceleration mitigation or would need to eject once the black-hole was under one million tonnes.
A useful paper on BH powered spaceships is this on authored by [Crane and Westmoreland][1].
The interstellar travel and space buff Adam Cowl in his review of black hole starships came to the following conclusion.
In conclusion the concept needs considerable work before it can be considered an interstellar drive option. The radiation intensities that need to be handled boggle the mind. However coupling our particle theories to black holes is not without problems – quantum gravity may well alter the intensity once the hole is small enough and we have no clear idea of the fate of the multitudinous particles produced. Does a super dense ball of quagma result, “stuck” to the ball by gluons dragged out of the vacuum of space? The related idea, of quark matter, might present the option of embedding a Kugelblitz inside a quark nugget. A more developed understanding of the quantum chromodynamic (QCD) vacuum and quantum gravity needs developing.
For now, like the original Photon Rocket, this idea goes back on the
shelf, until our physics catches up.
Basically BH powered starships sound like a nifty idea, but on closer examination they're more like a luxury vehicle that cost an arm and an leg to run, and spend most of their time in repair shop. While ordinary vehicles have much, much better performance characteristics and you can get some useful mileage out of them. Like fusion rockets, for example.