If a rocket ship with limited fuel has fallen into a decades-long elliptical orbit around the Sun, where in the orbit should it fire its engines in order to achieve escape velocity with a minimum of fuel? Likewise, where would the most efficient place in the orbit be to fire the engines in order to stay in-system but with a shorter orbital period?
Fire the engines closer to the Sun for maximum efficiency. This is due to the Oberth effect.
Assume the spacecraft undergoes a burn when it is farther away from the Sun. The expelled propellant will have a certain amount of kinetic and potential energy, $E_p$. If an energy of $\Delta E$ is released during the burn, the spacecraft gains an energy of $\Delta E-E_p$.
Now, if the burn is done when the spacecraft is close to the Sun, $E_p$ is lower because the propellant has a lower potential energy. Therefore, the quantity $\Delta E-E_p$ is higher, and the spacecraft has more energy. This makes it a lot more efficient maneuver for transferring to a variety of orbits - or, in this case, leaving the Solar System entirely.
As SF. pointed out, the Oberth effect holds for all massive bodies. It therefore makes sense that you can use other planets, not just the Sun, to reach a higher orbit or escape velocity.
As a rule of thumb, in orbital mechanics, actions you take on one side of an orbit take effect on the other side of an orbit.
So, in order to create an infinitely eccentric orbit (i.e. to break free of the Sun), you burn along the velocity vector at the point of closest approach to the Sun (periapsis or perehelion).
Similarly, you can make your orbit smaller by burning against the velocity vector at the same point.
What I've described is the Hohmann Transfer, which is the simplistic two burn system to move from one circular orbit to another, by moving through an elliptic orbit. In this case, you're starting in an elliptic orbit, so you only do the second stage.
There are some other things to consider. There are other types of orbital transfer with different burn characteristics that use less propellant.
Also, if you want to really get fuel efficient, you use a gravitational boost. So, for example, missions to the outer planets like Jupiter and Saturn typically boost down to use Venus, Earth and Mars (if they can) to slingshot themselves to the right orbit. It's counter-intuitive, but it's true.
As a great example of it, someone posted a challenge in the Kerbal Space Program forums to get to the Jupiter Analogue and back using the minimum size of space-craft. Someone used this technique to do it with obscenely little material and their calculations are documented in a spreadsheet.
If that's not enough, there's also the Interplanetary Transport Network, a series of paths through our Solar System using Lagrangian and other such points to really minimise fuel burn, at the cost of travel time.
A famous example of this was a Japanese spacecraft that malfunctioned (I think it was called Hiten: https://en.wikipedia.org/wiki/Hiten), and it still managed to get to the Moon using this technique.