TCAT117 suggests a pulsed fission engine, but these are horribly contaminating and therefore have never been tested.
https://en.wikipedia.org/wiki/Nuclear_thermal_rocket gives another alternative. This consists of a nuclear reactor as a source of heat, through which liquid hydrogen is heated and used as a propellant in a nozzle similar to a conventional rocket nozzle. This design was actually given some consideration and some engine tests carried out. It's much less hazarous than a pulsed fission engine, but chemical rockets are less hazardous than any of the nuclear options, so in the real world they won out.
Hydrogen is the preferred propellant as its light molecules give the highest exhaust velocity at any given temperature.
The following are highlights from the comparison in the Wikipedia article, which I have copied in here as requested:
Specific impulse 850-1000 seconds, more than double that typical for a oxygen/hydrogen powered engine. Specific impulse is the number of seconds a stage can produce a thrust equal to its initial fuel weight before fuel runs out. It is proportional to exhaust velocity. Thus the simple solid core nuclear thermal rocket is capable of double the efficiency of a chemical one.
Thrust - weight ratio achieved in apollo era (about 5:1 on a 1.5g planet.) This is much less than a chemical rocket, and means that nuclear thermal rockets are more suited to being used in upper stages where burn times are longer. The first stage (only) of a rocket needs high thrust-weight ratio as vertical takeoff means initially a lot of fuel is used fighting gravity. The sooner you can build some speed and get into a near-horizontal trajectory the better. Once this is achieved longer burn times at lower acceleration is not such a disadvantage. SNTP era (separate article) reached 30:1, a thrust-weight ratio at which engine mass ceases to be any real issue. https://en.wikipedia.org/wiki/Project_Timberwind#Space_Nuclear_Thermal_Propulsion_Program
NASA actually considered replacing the 3rd stage of Saturn V (known as Saturn IV-B) with a nuclear thermal rocket for enhanced performance.
The wikipedia article has a worked example based on the Saturn IV-B and I present a summary below. Delta V is the standard measure of efficiency of rocket in space, equal to the speed difference it is able to depart before it is depleted.
The author seems to have neglected the mass of the upper stages. If factored in, this will further favour the Nuclear Thermal Rocket on the mass/mass comparison, as the engine mass will be less significant.
Standard Saturn VI-B Hydrogen-Oxygen fueled
Fueled Mass 119800kg, dry mass 13400kg, specific impulse 475s.
Delta V (414 s × 9.81) ln(119,900/13,311), = 8900m/s
Nuclear thermal rocket, drop-in replacement matching volume/volume
Fueled Mass 38600kg, dry mass (due to increased engine mass) 17300kg, specific impulse 850s
Delta V (850×9.81) ln(38,600/17,300) = 6,700 m/s.
While the Delta V is lower, the mass of the stage is much lighter due to the hydrogen propellant being lighter than the hydrogen/oxygen bipropellant of the original stage, so the stages below will compensate.
Nuclear thermal rocket, replacement matching mass/mass
Fueled Mass 19000kg, dry mass (due to increased tankage) 38300kg, specific impulse 850s
(850 s×9.81) ln(119,900/38,300), or 9,500m/s
NASA considered an even smaller stage due to constraints of the Vehicle Assembly Building : 10,429 kg empty and 53,694 kg fueled. This would improve the payload capacity of the Saturn Vf from 127,000 kg delivered to low earth orbit (LEO) to 155,000 kg.
This is a moderate improvement on chemical rockets, based on Apollo era technology and far from optimised. An example based on project Timberwind would be a much greater improvment, 1.5 to 4 times payload increase. https://en.wikipedia.org/wiki/Project_Timberwind#/media/File:SNTP_Upper_Stage_Applications.png
Note that the Space Shuttle's second stage (the main engines) fired from liftoff, though most of the initial thrust was provided by the first stage boosters. I would foresee a similar arrangement with chemical boosters around a nuclear thermal rocket core, to keep the heavy nuclear thermal rocket engine burning for the longest possible time.
An issue mentioned is that the specific impulse of nuclear thermal rockets is limited by the maximum temperature the reactor can withstand. I think a hybrid engine with a nuclear thermal core followed by oxygen injection into the hydrogen stream in an afterburner for liftoff could improve this issue to give even higher specific impulse, and would have great potential as a first stage.