If you're open to messing with the gravity and atmosphere of the planet, heavy fliers become much more plausible (although at the cost of potentially making the planet less hospitable for visiting humans).
Relevant parameters:
- Lower gravity
- Less weight for the wings to lift
- Less weight for the bones and muscles to need to support, so the bones themselves can be lighter
- A denser atmosphere
- Each stroke of the wings can provide more thrust, which should permit faster flight using less power
- Then again, denser air means proportionally greater pressure drag- but I do not know how much that matters
- Gliding will provide more lift, since, again, there's more air to push off of
- Then again, less gravity probably means a slower glide speed, which may or may not cancel out the lift boost
While I am certainly not qualified to analyze all the possible effects of all those different factors, I can take a stab at some of the simpler-looking ones.
Say the ideal griffin has the body of a lion (250 kg and 3.0 meters long, call it), and the wings of a bald eagle scaled up to match the length and width of the lion. Typical bald eagles can have a length of 1.02 m, a 2.3 m wingspan, and 6.3 kg mass. Scaling that up by a factor of 3 gives a bird 3.06 m long, with a 6.9 m wingspan. If I assume that lift is probably proportional to the area of the bird's wings, which is proportional to the square of the wingspan, this eagle should be able to lift 56.7 kg (9 times its Earth mass) and then some. Which is less than the mass of the bird itself (170.1 kg = 6.3 kg * 3^3), and nowhere near the mass of our lion. That's the Square-Cube Law in action.
However, that's not the end of the story. If we want the griffin to be able to lift its 250-some kilograms as easily as our giant Earthbound eagle can lift 56.7 kg of its mass, we can reduce gravity and thicken the atmosphere to compensate.
To fly on the planet (call it P), we need
$$F_{l_P} \ge F_{g_P}$$
(read that as "Force of lift on P $\ge$ force of gravity on P")
Since a denser atmosphere there will give more lift as compared to Earth
$$F_{l_E} \cdot \frac{\rho_P}{\rho_E} \ge F_{g_P}$$
Substituting in mass * gravitational acceleration for those forces (from Newton's second law of motion, F = ma)
$$m_{equiv} \cdot g_E \cdot \frac{\rho_P}{\rho_E} \ge m \cdot g_P$$
where $m_{equiv}$ is the 56.7 kg figure calculated above.
Rearranging that a bit gives
$$\frac{m_{equiv} \cdot \rho_P}{\rho_E} \ge \frac{m \cdot g_P}{g_E}$$
After substituting in some known values
$$\frac{56.7 \cdot \rho_P}{\rho_E} \ge \frac{250 \cdot g_P}{g_E}$$
And after rearranging again
$$\frac{\rho_P}{\rho_E} \ge 4.41 \frac{g_P}{g_E}$$
So the atmosphere on this planet would need to be about 4.5 times as dense as Earth's, or gravity at its surface would need to be 4.5 times weaker, or somewhere in between. In theory.
I'd recommend reducing gravity by at least a factor of 3, if not 4. This goes back to the square-cube law: Even though the cross-sectional area (and thus the strength) of the eagle's wing bones increased by a factor of 9, the griffin weighs much more than 9 times what an eagle weighs. So the griffin's wings might just break whenever it tries to take off in Earth gravity, denser atmosphere or no.
The question then becomes: Could a planet with much less gravity than Earth actually hold onto such a thick atmosphere long enough for griffins to evolve? Sure. Venus and Titan both have much thicker atmospheres than Earth does; and Titan's gravity is much weaker as well. I do not know why these conditions exist on those worlds, but clearly they do.
Finally, would such a creature actually evolve in such an environment? I doubt it; not as I described. Lions are big and bulky; the classical griffin would surely be outcompeted by something with a smaller, lighter, more aerodynamic body- something more birdlike, in short. As for the more reptilian design in your question? Sure. Maybe. I don't know. I'm no biologist.
As for riding them: No idea, but you'll probably have better luck with birdlike wings that just attach to the griffin's shoulders than with batlike wings extending all the way to the hind legs and tail. If people are going to saddle these things, they'll need somewhere to put their feet, and bat wings would get in the way. Unless the rider was perched more on the griffin's shoulders, in which case the weight of the rider would be seriously unbalancing. Dragonriding has similar issues that have already been discussed here on Worldbuilding... somewhere.
