On its own, a rocket just goes. It doesn't go in a straight line, and it particularly doesn't go on the line you want it to go on. In order to control it, you need a control system that does three things: determines where it is, determines where it wants to go, and positions the rocket nozzle(s) accordingly. (It's also possible to use thrust vectoring - steering by throttling back one of your several rocket engines. Assuming you have several rocket engines. But since this is more complex in pretty much every way, it's probably not going to save you.)
None of these steps are going to be easy, or really even feasible, with Victorian technology.
Step 1 is slightly plausible, in that the fundamental tool used to determine the rocket's orientation - the gyrocompass - was invented in the Victorian era. (The tail end of it, but never mind that.) However, these were relatively large compasses designed to be used aboard ships; they were not hardened to survive the rigors of rocket launch. (In terms of acceleration and vibration, I'm not sure the Victorian era had anything comparable to the rigors of spaceflight, outside of the most powerful of natural disasters.) They were also prohibitively large and heavy for missile work. But they did exist.
Step 2 is less plausible. Once the rocket's orientation is determined, you need to compare it to the desired orientation (which changes during ascent) and then translate the difference into precise commands for each of the thruster actuators. And you have no electronics with which to do the job.
Mechanical computers did exist for solving somewhat similar (though less complex) problems in artillery. In the 1910s, there were computers that, given the range and orientation to another ship, could translate them into approximate firing solutions for the ship's guns. (But they couldn't score hits on computed solutions alone until the development of electronics and radar; considerable manual work was needed "correcting" shots.) Your mechanical computer would need to be much more precise than this, because a human trying to correct it by hand will invariably overcorrect and you will not go to space today.
It goes without saying that like the gyrocompasses, early naval firing computers weren't up to the rigor of space travel (they weren't always up to the rigors of ocean travel, in practice) and were much too large and heavy to fit on a practical rocket. So, you'd need to be much more advanced than the state of the art in multiple ways.
That gets you to step 3. Hydraulic controls are available, but will they be durable enough and accurate enough for the purpose? A few millimeters left and right can spell the difference between getting to space and getting the Range Safety Officer to press the big red button. (Oh, you don't have radar-detonated explosives? Hope you don't land anywhere important...)
Having a rocket engine, although not trivial in itself, is a far cry from having a rocket.