Short Answer:
Planets with up to about 2 times the mass of Earth should probably be habitable. With normal planetary materials, the surface gravity of worlds with such masses should get up to about 1.6 g.
Long asnwer in Five Parts.
A planet might be able to retain a considerable atmosphere for geological period sof time without the protection of a magnetosphere. However, there are other possible upper mass limits for a habitable planet.
Part One:
The escape velocity of a world is the most important facter which determines how long it can retain an atmosphere.
The two main classes of astronomical bodies are objects with escape velocity too low to retain significant atmosphere for any time, and objects with escape velocity so high they can retain significant atmosphere for billions and trillions of years. Objects of intermediate escape velocity that will retain atmospheric gases for periods of time intermediate between zero time and infinite time are a minority.
The most important factor in a world's ability to retain any atmosphere that it might have is its escape velocity. And a world's escape velocity is not the same as its surface gavity, and does not increase and decrease in strict proportiona to the surface gravity.
Habitable Planets for Man, Stephen H. Dole, 1964, discusses the required escape velocity on pages 34 & 35.
https://www.rand.org/content/dam/rand/pubs/commercial_books/2007/RAND_CB179-1.pdf
It gives a rough rule of thumb for the period of time it will take a planet for its original atmosphere to decrese to 1/e of its original aamount (if there is no process of atmospheric replacement). Since e is 2.718, 1/e is 0.3679 of the original amount.
After twice that period, the remaining atmosphere will be 0.13535of the original amount, after three periods the remaining atmosphere will be 0.04989 of the original amount, after four periods the remaingin atmospehre will be 0.0183 of the original amount, and so on.
Atmospheric gases escape from the outermost atmosphere layer, the exosphere. Their velocity depends on the temperature in the exosphere, which is usually considerably higher than the temperature at the surface and the lower atmosphere.
According to table 5 on page 35, the period for a gas in a world's atmosphere to be reduced to 0.3679 of the original amount is proportional to the radio of the world's escape velocity dividd by the root-mean-square velocity of that gas in the temperatures at the exosphere.
If the ratio is 1 or 2, the period is 0. If the ratio is 3 the period is a few weeks. If the ratio is 4, the period is several thousand years. If the ratio is 4, thepirod is aobut 100 million years. If the ratio is 6, the priod is infinity.
So by increasing the escape velocity of a world by 3 times, from 2 times the root-mean-square velocity of a gas to 6 times, the period it will take for the atmosphere to descrease to 0.3679 of the original amount will change from 0 to infinite.
So the ability of a world to retain an atmosphere depends in the first place on its escape velocity compared to the velocities of gas particles in its exosphere. The temperature in the exosphere will depend on heat from the star. And the escape velocity of the world will depend on its mass, radius, and density.
So the escape velocity of world is the first thing that determines its atmosphere. Other factors may decrease a world's ability to retain an atmosphere but nothing can possibly permit a world with insufficient escape velocity to retain an atmosphere for geologic eras of time, unless some advanced civilization builds roof over that world - out of matter or forcefields - to keep the atmosphere in.
I don't know what sort of biochemestry you want for the lifeforms on your high gravity planet or what sort of surface temperatures they will need, so I will assume that they use liquid water, and have Earth like biochemestry at Earth like temperatures.
As we all know, the planet Earth has 1 Earth mass, 1 Earth radius, 1 Earth averge density (5.514 grams per cubic centimeter), a surface gravity of 1 g (9.80665 meters per second per second), an escape velocity of 1 Earth escape velocity (11.186 kilometers per second), with Earth like temperatures, and has retained an Earth like atmopshere for sufficiently long geologic eras. Earth has managed to retain enough oxygen, carbon dizoide, nitrogen, and water vapor for life.
So thus it seems logical to assume that any with a surface gravity equal to or greater than Earth's equal to or greater than 1 g, will have a sufficient escape velocity to retain a dense atmosphere of the gases necessary for Earth life.
Part Two:
Must all planets with a higher surface gravity than Earth's have an escape velocity high enough to retain an atmosphere long enough for advanced oxygen breathing lifeforms to evolve?
But escape velocity doesn't change at the same rate as surface gravity. Only 2 planets in our solar system, Jupiter & Neptune, have surface gravity greater than Earth's, about 2.357 g and 1.122 g. But Jupiter, Saturn, Uranus, and Neptune have escape velocities higher than Earth's - 5.3, 3.1, 1.9, & 2.09.
Similarly, it may be possible to design a planet with a higher surface gravity than Earth and a lower escape velocity.
For example, a world with 0.25 the mass of Earth and a radius of 3,000 kilometers (0.47088 Earth), would have a surface gravity of 1.13 g and an escape velocity of 8.15163 km/s (0.7287 of Earth).
https://philip-p-ide.uk/doku.php/blog/articles/software/surface_gravity_calc
http://calctool.org/CALC/phys/astronomy/escape_velocity
It would have 0.25 the mass of Earth in 0.103823 the volume and thus would have about 2.4078 times the density of Earth.
A world with 0.2 times the mass of Earth with a radius of 2,500 kilometers (0.392 Earth ) would have a surface gravity of 1.31 g and an escape velocity of 7.986 km/s, (0.7129 Earth). It would have a density 3.32 times that of Earth.
A world with 0.15 times the mass of Earth with a radius of 2,000 kilometers (0.3129224 Earth) would have a surface gravity of 1.52 g and an escape velocity of 7.733 km/sec (0.69 of Earth). It would have a density 4.9 times Earth's.
Thus the escape velocities of those worlds have been getting lower even while their surface gravities got higher.
On page 54 Dole says that the surface temperatures in Earth's exosphere are about 1000 K to 2000 K, several times the average surface temperature of 288 degrees K.
Dole says that if the maximum exosphere temperature of a world with a surface temperature as high as Earth's could be as low as 1000 degrees K, the root-men-square velocity of atomic oxygen in the exosphere would be as low as 1.25 kilometers per second. Thus a planet with an escape velocity 5 times 1.25 km/s, or 6.25 km/s, would be able to retain oxygen for geologic periods of time.
Above I designed several worlds which had surface gravities higher than Earth's but escape velocities lower than Earth's, the lowest with an escape velocity only 7.733 kilometers per second, jut a little bit above 6.25 km/s.a colulated the N
But if the maximum exosphere temperature of such a world was 1250 K, or 1500 K, or 1750 K, or 2000 K, the velocities of oxygen atoms in the exosphere would be higher, and possibly significantly higher than the escape velocities I calculated above, and possibly high enough for the world to lose oxygen at a much faster rate, too fast for large lifeforms needing oxygen to ever evolve.
So that shows that it is possible to design a world to have a higher surface gravity than Earth while also having an escape velocity too low to retain an oxygen atmosphere for long enough.
The two main classes of astronomical bodies are objects with escape velocity too low to retain significant atmosphere for any time, and objects with escape velocity so high they can retain significant atmosphere for billions and trillions of years. Objects of intermediate escape velocity that will retain atmospheric gases for periods of time intermediate between zero time and infinite time are a minority of astronomical objects.
Part Three:
Other factors can make a world lose atmosphere faster.
A world can lose atmosphere by atmospheric gases escaping itno space, and also by atmospheric gases becoming liquified or solidified and leaving the atmosphere.
Here is a link to a list of process of atmospheric loss:
https://en.wikipedia.org/wiki/Atmospheric_escape
And one of those processes is charged particles from the solar or stellar wind knocking particles out of the atmosphere. So the OP worries about whether massive Earth like planets would have solid cores and so lack magnetic fields to deflect charged particles from the Sun.
In our solar system Earth, Jupiter, Saturn, Uranus, and Neptune have strong magnetic fields whil Mercury, Venus, and Mars have very weak or no magnetic fields.
As a general rule, it is believed that more massive worlds will tend to have stronger magnetic fields than less massive ones, and that rapidly rotating worlds will tend to have stronger magnetic fields than slower rotating ones.
Despite its small size and slow 59-day-long rotation, Mercury has a significant, and apparently global, magnetic field. According to measurements taken by Mariner 10, it is about 1.1% the strength of Earth's. The magnetic-field strength at Mercury's equator is about 300 nT.[98][99] Like that of Earth, Mercury's magnetic field is dipolar.[97] Unlike Earth's, Mercury's poles are nearly aligned with the planet's spin axis.[100] Measurements from both the Mariner 10 and MESSENGER space probes have indicated that the strength and shape of the magnetic field are stable.[100]
It is likely that this magnetic field is generated by a dynamo effect, in a manner similar to the magnetic field of Earth.[101][102] This dynamo effect would result from the circulation of the planet's iron-rich liquid core. Particularly strong tidal heating effects caused by the planet's high orbital eccentricity would serve to keep part of the core in the liquid state necessary for this dynamo effect.[37][103]
Mercury's magnetic field is strong enough to deflect the solar wind around the planet, creating a magnetosphere.
https://en.wikipedia.org/wiki/Mercury_(planet)#Magnetic_field_and_magnetosphere
If Mercury had an escape velocity strong enough to retain an atmosphere for long, Mercury's magnetic field would probably serve to protect that atmosphere from rapid erosion by the solar wind.
In 1967, Venera 4 found Venus's magnetic field to be much weaker than that of Earth. This magnetic field is induced by an interaction between the ionosphere and the solar wind,[112][113] rather than by an internal dynamo as in the Earth's core. Venus's small induced magnetosphere provides negligible protection to the atmosphere against cosmic radiation.
The lack of an intrinsic magnetic field at Venus was surprising, given that it is similar to Earth in size and was expected also to contain a dynamo at its core. A dynamo requires three things: a conducting liquid, rotation, and convection. The core is thought to be electrically conductive and, although its rotation is often thought to be too slow, simulations show it is adequate to produce a dynamo.[114][115] This implies that the dynamo is missing because of a lack of convection in Venus's core. On Earth, convection occurs in the liquid outer layer of the core because the bottom of the liquid layer is much higher in temperature than the top. On Venus, a global resurfacing event may have shut down plate tectonics and led to a reduced heat flux through the crust. This insulating effect would cause the mantle temperature to increase, thereby reducing the heat flux out of the core. As a result, no internal geodynamo is available to drive a magnetic field. Instead, the heat from the core is reheating the crust.[116]
One possibility is that Venus has no solid inner core,[117] or that its core is not cooling, so that the entire liquid part of the core is at approximately the same temperature. Another possibility is that its core has already completely solidified. The state of the core is highly dependent on the concentration of sulfur, which is unknown at present.[116]
The weak magnetosphere around Venus means that the solar wind is interacting directly with its outer atmosphere. Here, ions of hydrogen and oxygen are being created by the dissociation of water molecules from ultraviolet radiation. The solar wind then supplies energy that gives some of these ions sufficient velocity to escape Venus's gravity field. This erosion process results in a steady loss of low-mass hydrogen, helium, and oxygen ions, whereas higher-mass molecules, such as carbon dioxide, are more likely to be retained. Atmospheric erosion by the solar wind could have led to the loss of most of Venus's water during the first billion years after it formed.[118] However, the planet may have retained a dynamo for its first 2–3 billion years, so the water loss may have occurred more recently.[119] The erosion has increased the ratio of higher-mass deuterium to lower-mass hydrogen in the atmosphere 100 times compared to the rest of the solar system.[120]
The solar wind is more concentrated at the distance of Venus from the Sun, and Venus has been without a magnetosphere for at least a billion years, yet Venus still retains an atmosphere many times as massive as Earth's.
Mars lost its magnetosphere 4 billion years ago,[158] possibly because of numerous asteroid strikes,[159] so the solar wind interacts directly with the Martian ionosphere, lowering the atmospheric density by stripping away atoms from the outer layer.
https://en.wikipedia.org/wiki/Mars#Atmosphere
The Galileo craft made six close flybys of Ganymede from 1995 to 2000 (G1, G2, G7, G8, G28 and G29)[22] and discovered that Ganymede has a permanent (intrinsic) magnetic moment independent of the Jovian magnetic field.[90]The value of the moment is about 1.3 × 1013 T·m3,[22] which is three times larger than the magnetic moment of Mercury.
Given that Ganymede is completely differentiated and has a metallic core,[8][77] its intrinsic magnetic field is probably generated in a similar fashion to the Earth's: as a result of conducting material moving in the interior.[22][77] The magnetic field detected around Ganymede is likely to be caused by compositional convection in the core,[77] if the magnetic field is the product of dynamo action, or magnetoconvection.[22][93]
Despite the presence of an iron core, Ganymede's magnetosphere remains enigmatic, particularly given that similar bodies lack the feature.7 Some research has suggested that, given its relatively small size, the core ought to have sufficiently cooled to the point where fluid motions, hence a magnetic field would not be sustained. One explanation is that the same orbital resonances proposed to have disrupted the surface also allowed the magnetic field to persist: with Ganymede's eccentricity pumped and tidal heating of the mantle increased during such resonances, reducing heat flow from the core, leaving it fluid and convective.[57] Another explanation is a remnant magnetization of silicate rocks in the mantle, which is possible if the satellite had a more significant dynamo-generated field in the past.7
Note that it is uncertain why Ganymede has a magnetic field and the similar objects Callisto and Titan do not.
Note that Titan has a dense atmosphere despite not having a magnetosphere of its own.
Saturn's moon Titan and Jupiter's moon Io have atmospheres and are subject to atmospheric loss processes. They have no magnetic fields of their own, but orbit planets with powerful magnetic fields, which protects these moons from the solar wind when its orbit is within the bow shock. However Titan spends roughly half of its transit time outside of the bow-shock, subjected to unimpeded solar winds.
https://en.wikipedia.org/wiki/Atmospheric_escape#Titan_and_Io
So in our solar system Venus and Titan are examples of two worlds that have kept dense atmospheres for billions of years despite not having magnetospheres of their own. And on th eother hand the lack of a Martian magnetosphere does make it lose atmosphere faster.
Part Four:
How massive can a planet be and have a strong magnetic field?
On page 54 Dole decided that since humans would not want to colonize a world with a surface gravity greater than 1.50 g, such a world would be the largest humans would colonize. If a planet is made of materials similar to Earth, one with a surface gravity of 1.5 g would have 2.35 Earth mass, a radius of 1.25 Earth radii, and an escape velocity of 15.3 kilometers per second.
Of course alien life forms might flourish in surface gravities which were much too high for humans to endure.
Somewhat different minimum and maximum masses of habitable worlds were given by Helr and Barnes in "Exomoon habitability constrained by illumination and tidal heating", 2013, but they were considering habitability of any type of liquid water using life, not merely for life with the same requiements as humans.
https://www.researchgate.net/publication/234103511_Exomoon_Habitability_Constrained_by_Illumination_and_Tidal_Heating
On pages 3 and 4 they write:
A minimum mass of an exomoon is required to drive a magnetic shield on a billion-year timescale (Ms ≳ 0.1M⊕, Tachinami et al. 2011); to sustain a substantial, long-lived atmosphere (Ms ≳ 0.12M⊕, Williams et al. 1997; Kaltenegger 2000); and to drive tectonic activity (Ms ≳ 0.23M⊕, Williams et al. 1997), which is necessary to maintain plate tectonics and to support the carbon-silicate cycle. Weak internal dynamos have been detected in Mercury and Ganymede (Kivelson et al. 1996; Gurnett et al. 1996), suggesting that satellite masses > 0.25M⊕ will be adequate for considerations of exomoon habitability. This lower limit, however, is not a fixed number. Further sources of energy – such as radiogenic and tidal heating, and the effect of a moon’s composition and structure – can alter our limit in either direction. An upper mass limit is given by the fact that increasing mass leads to high pressures in the moon’s interior, which will increase the mantle viscosity and depress heat transfer throughout the mantle as well as in the core. Above a critical mass, the dynamo is strongly suppressed and becomes too weak to generate a magnetic field or sustain plate tectonics. This maximum mass can be placed around 2M⊕ (Gaidos et al. 2010; Noack & Breuer 2011; Stamenković et al. 2011).
So the supression of the planetary dynamo would not only end the magnetic field, it would also end plate techtonics and the carbon-silocon cycle, also considered to be important for life.
That is an even smaller mass than the 2.35 Earth mass that Dole considered to be the upper limit for human habitability,a nd which came with a surface gravity of only 1.5 g, which is not very impressive as science fictional high gravity palnets go.
However, by playing around with the radius and density, it might be able to design a planet with only 2 times the mass of Earth which has a much higher surface gravity.
For example, a planet with 2 times Earth's mass and a radius of 6,500 kilometers, slightly larger than Earth, would have a surface gravity of 1.93 g. Such a planet would have 1.0202 times the radius of Earth so would have twice the mass of Earth in 1.0618 the volume of Earth. Thus it would have 1.883 times the density of Earth, 10.386 grams per cupic centimeter.
A planet with 2 times Earth's mass and a radius of 6,000 kilometers, slightly smaller than Earth, would have a surface gravity of 2.26 g. Such a planet would have 0.8417 times the radius of Earth so would have twice the mass of Earth in 0.835 the volume of Earth. Thus it would have 2.3952 times the density of Earth, 13.207 grams per cupic centimeter.
A planet with 2 times Earth's mass and a radius of 5,000 kilometers, would have a surface gravity of 3.26 g. Such a planet would have 0.7848 times the radius of Earth so would have twice the mass of Earth in 0.4833 the volume of Earth. Thus it would have 4.1382 times the density of Earth, 22.818 grams per cupic centimeter.
Osmium, the densest known naturally occuring (thorugh extremely rare) element, has a density of 22.59 grams per cubic centimeter.
A planet with 2 times Earth's mass and a radius of 4,000 kilometers, would have a surface gravity of 5.02 g. Such a planet would have 0.6278 the radius of Earth and so would have twice the mass of Earth in 0.2474 the volume of Earth. Thus it would have 8.084 times the density of Earth, 44.575 grams per cubic centimeter.
Hassium, is believed to be the synthetic element with the greatest density, 40.7. But that is a calculated density - apparently they haven't been able to make enough of it to measure its density.
And there might possibly be other possible heavy elements which would possibly be more stable and possibly also more dense than Hassium.
https://en.wikipedia.org/wiki/Island_of_stability
And it is possible that there are exoplanets with extremely high densities. Kepler-131 c is supposed to be have a density of 77 plus or minus 55 grams per cupic centimeter, between 22 and 132 grams per cubic centimeter.
https://en.wikipedia.org/wiki/List_of_exoplanets_discovered_using_the_Kepler_space_telescope
Part Five:
Habitable super-Earths?
Super-Earths, rocky terrestrial planets more massive than Earth, might possibly be habitable.
Would they have magnetic fields?
Earth's magnetic field results from its flowing liquid metallic core, but in super-Earths the mass can produce high pressures with large viscosities and high melting temperatures which could prevent the interiors from separating into different layers and so result in undifferentiated coreless mantles. Magnesium oxide, which is rocky on Earth, can be a liquid metal at the pressures and temperatures found in super-Earths and could generate a magnetic field in the mantles of super-Earths.[91] That said, super-Earth magnetic fields are yet to be detected observationally.
https://en.wikipedia.org/wiki/Super-Earth#Magnetic_field
According to one hypothesis,[92] super-Earths of about two Earth masses may be conducive to life. The higher surface gravity would lead to a thicker atmosphere, increased surface erosion and hence a flatter topography. The end result could be an "archipelago planet" of shallow oceans dotted with island chains ideally suited for biodiversity. A more massive planet of two Earth masses would also retain more heat within its interior from its initial formation much longer, sustaining plate tectonics (which is vital for regulating the carbon cycle and hence the climate) for longer. The thicker atmosphere and stronger magnetic field would also shield life on the surface against harmful cosmic rays.[93]
https://en.wikipedia.org/wiki/Super-Earth#Habitability
I do not know how massive a super-Earth could get while still retaining a magnetic field.
A recent study says that large moons are necessary for a life bearing planet - a rather speculative opinion - and that computer similations show that super-Earth could not form large moons.
https://excellentpix.com/can-super-earth-planets-be-habitable-a-new-study-says-no.html
If both claims are correct, science ficiton writers who care about sceintific plausibility should probably restrict the sizes of terrestrial type planets with life to masses of 2 Earth mass or less, with surrface gravity of about 1.6 g or less.
