Assuming nothing is more than 1.8m above the surface; Radius of 606km
- Your human-sized observers have human eyes.
- Every human eye stays at or below 1.8m altitude at all times.
- No-one ever compares shadows in different places on the planet. Or compares the stars.
- No-one ever builds anything taller than 1.8m high.
So - the human eye at 20/20 vision is accurate to about 1/60th of a degree at 20ft:
The "20ft" (6.1m) is where the eye muscles that focus your eye for near vision relax - it's just "anything greater than 6.1m has a 1/60 degree accuracy".
So we're trying to find the radius of the circle made by the intersection of two points anywhere along these two lines (which become tangents), where the angle between them is 0.016 degrees:
Having not done geometry for 2 decades, I messed around with some diagrams for several minutes before giving up and plugging it into FreeCAD's sketching tool. Which promptly failed to solve it in all sorts of crazy ways relating to floating point accuracy that I find very interesting but nobody else will.
Experimentation and spoon-feeding the solver eventually gave me the answer that given a radius of 606km, and a requirement that no eye ever gets 1.8m above sea level, the world will always appear flat within the human eyes vertical resolution.
Even if they invent lasers to precisely mark where the horizon "should" be, they won't be able to detect it.
If they only have people 1.8m high and walk around and look at each other, they still won't be able to tell, as fellow humans will be less than 1/60th of a degree when they disappear over the horizon.
How to calculate something more realistic from this - ie. Lets add buildings!
Because trig of really small numbers can be simplified to pass-through functions, you can scale the radius and the feature height linearly (within reason). So if you're largest ship is 18m tall, to not notice something fishy when it sails off, the planet needs to have a radius of 6060km. Same if you have a building 18m high and start walking away from it.
If the largest structure is 360m above the spheres surface, then the planet needs to be 12120km in radius for a human not notice it shrinking over the horizon.