Basically, it's a continent that wants to be really hard to find (you can find it if you try really, really hard). For example, if you got in a ship and wanted to sail to it you wouldn't be able to find it unless you were extremely lucky which is unlikely. The continent is around the size of South America in a world with 6 continents that is slightly bigger than Earth. It should be hidden not inaccessible as certain people can acess it if they know how. The time period is around 1700s.
Antarctica, while not hidden, was essentially "protected" by ice shelfs until 1895 when the first human set foot on it. Thanks to the dangers posed to ships by ice shelfs, and a very unfriendly climate, most people still steer clear of it in 2018.
Your land mass could simply be surrounded treacherous ice, very dangerous looking rocks, etc.
The inhabitants of this land mass would either have to
1) Wait until they could build "ice-cutter" boats to become a seafaring nation(s)
2) The "ice people" would have to build something like the Japanese in the 1200s for the 2nd Mongol invasion, which was essentially a series of forts at every beach where a ship could land. The Japanese also had the help of Typhoons if they could keep an invading army at sea long enough.
Japan managed to hold off invaders until into the 20th century. Between "ice everywhere" and some sentries to pick off explorers you could likely keep your land mass the stuff of folk lore.
Under your current constraints: it's impossible
By the 1700s we were not only sailing everywhere, we were mapping everything. Let's assume for a moment that your continent was unapproachable as such a distance that sailors couldn't see land (for whatever reason). Now, calculating the distance between two above-sea-level objects is no small thing, but let's assume that at distances greater than 20Km, you can't see the continent.
It would take "no time at all" for a big blank spot to develop on the world maps with a pin in it holding a small flag with itty-bitty text neatly written in a beautiful caligraphy saying, "By the beneficence of Jove and King Horace there be wonder here!" or some such that would translate to, "what the heck is here?"
So, while the details of your continent would be unknown, the fact that a continent-sized swath of ocean is unknown would be the worst-kept secret on the planet.
And every sailor out there would be hungry to figure out what's there.
Conclusion #1 You can't hide the existence of something the size of a continent.
so, let's ask ourselves, what could keep the average sailor from knowing about it? Reefs, ocean shelves, sandbars, fog... the problem with continent-sized problems is that there aren't continent-sized solutions. Unless you throw realism out the window and hold to magic/fantasy, there is nothing that can keep a fog bank around an entire continent. There's too much distance. Too much change of climate from one side to the other.
Consider South America. On one side is the angry, hurricane-laden Atlantic Ocean. On the other, the benign and comfortable Pacific Ocean. So, storms and other climate-related solutions are out, even in part. They're not permanent. They change based on the local ocean, sunlight, landscape, etc. etc....
Things in the water are out, too. Reefs wear down or build up. Storms break them. Shallow water is overcome with oars and shallow drafts. They're inconvenient, but not insurmountable. Worse... around an entire continent? No.
Conclusion #2 Continent-sized problems require continent-sized solutions after all.
So, let's not hide the fact that there's a continent there, let's hide simply hide the interior of the continent. Suddenly that little flag reads, "By the beneficence of Jove and King Horace there be dirt here! Be there gold?"1
So, let's build a ring of fire: volcanoes. Basically, let's create a continent that's one big unstable tectonic plate, and the world is bursting out around it. It's not fog that's curtaining the continent... it's ash. Lots of it. And it's a recent development such that the magma flows and ash haven't had enough time to build up new coastline that's outside the pyroclastic cloud.
This is hard to believe... but it's plausible. It's also on the clock. All that magma and ash will begin to build up new coastline — and fairly quickly (tens of years, IMO).
So, everybody knows it's a continent because volcanos are well known... but it's a formidable ring of volcanos. Even if you know how to approach the continent... you'll need Maui's help to get past the volcanos.
Conclusion #3 You shouldn't try to hide the continent. That's impossible. No one in the 1700s would suddenly slap their foreheads and exclaim, "you mean there's a continent there?" You should make the continent unapproachable. The secret isn't its existence, but how to step foot on its interior safely. And I'm voting for volcanos to make that happen. Or dinosaurs.
1 I'm a Terry Pratchett fan, so this was necessary. Very necessary.
The continent is under 60 feet of water, underground. The continent has a small cone like a volcano that rises above the ocean where rain water and sunlight come in to this underground and underwater cavern. The civilization lives near that single entrance while the rest of the continent is dark and potentially dry.
They could possibly create mirrors to reflect the sunlight to a larger part of the underground/underwater region.
Once they figured out how to create electricity much more of the continent could be used for living, growing crops, etc.
Now to figure out how to keep the island from being discovered...
Make the Cost of Trading with India Lower
It's popularly said that Columbus was attempting to prove some novel scientific concept of the roundness of the Earth when he set sail, but that really is not the case. Navigators used circular Earth equations even before the 1400s to get everywhere. In fact, the calculations of the radius of the Earth were pretty good.
Columbus did the math as best he could and calculated that he could ship goods more cost effectively by boat than the current costs of going over land and dealing with dozens of middle-men, governments, bandits; or the other alternative of sailing around the coast of Africa at great risk due to storms and taking a very long time to take such a long route.
Arrogant as everyone was, no one even considered the possibility that there might be a landmass in that enormous expanse.
If your continent could manipulate the political and economic situation to keep an easy and low-cost trade route to India open, it's extremely likely that no one would bother to sail west. If you have that kind of political clout, you could also maybe discourage people with enough money to send a few boats on what very well might be a suicide mission from doing so.
Dinotopia1 comes to mind as it is a large continent hidden by storm dashed reefs. Along the same lines if you have strong water/air currents that divert away from the continent only a strongly motivated person could make it to the continent.
Also if you allow magic Themyscira, Wonder woman's home, comes to mind which is protected by fog and a magic mirage.
One other idea comes to mind from One Piece, Amazon Lily is surrounded by sea monsters (and calm ocean) and only those with seastone on the bottom of their ships can travel to the island safely.
Your continent should be so low, LOW TIDE will cover it.
Your continent will be a big sand barge where people can rest on it for some time, so... how can this be considered a livable place?
It should have 3 or more plateus, with fresh water for drinking, and when the continent sinks the platues will be covered but just near the brim, to make it appear somewhat like a sandbarge.
Your mountains will look like small Islets.
But how about your people?
Well, they will be living on boat houses which is ancored in the continent. You can also create a big barge for more people to live in. That way, your people is also masking where the island is located, they'll be like wondering sailors from afar, they might be discovered when the tide rises but... well, that continents government can take care of that can't they?
My favorite solution for this type of problem is a giant asteroid impact feature, which creates a central high region plus concentric rings of mountains around the central feature.
Possibly the outermost ring of mountains is glaciated and people think that the entire circular area within the outer ring is also an icy desert. But actually there are sufficient volcanic vents etc., for internal heat to keep temperatures mild in the inner areas of the impact feature. Thus there are concentric rings of land and sea with a total area of ice free land the size one desires.
And maybe the natives of the impact feature have discovered one or more tunnels through the outermost mountain ring under the glaciers so that they can sail on the outer ocean.
One problem is that the impact would have destroyed all life on that planet and it would have taken billions of years for life to reappear and evolve into intelligent beings, and by then weathering and geologic forces should have destroyed the impact feature.
Possibly that world was a lifeless, airless moon or planet without plate tectonics and a super advanced civilization terraformed it to make it habitable. Possibly the world was too small to retain a breathable atmosphere for more than about 100 million years and the aliens who terraformed it inhabited it for 80 million years, and the present occupants are descended from human colonists who settled millions of years later and eventually lost their civilization and are rebuilding it and don't know that their world is doomed to lose its air in a few million years.
You can probably place it underground, because it would be hidden by both sailors and satellites, if your worried about the sailing part, you could have a meteor crash into the planet, and mountains rise to the sides, and there would be a mountain pass explained by erosion. And maybe the crater's depth reaches into the continents ceiling. This could be helpful for unique species in the continent too, so a plant or animal species uses bio luminescence to light up the area. If you want a normal species in there, you could say that they were carried away into the continent.
Alternatively, you could make the area have more clouds, maybe the trees there produce rain, as seen in the amazon rain forest. They are able to release Biogenic volatile organic compounds into the air as well as water. And usually, it causes a storm when it rains, so you get a natural barrier.
Take as model our continents: if South America would have been presented with his longer side horizontal instead of almost vertical an connected with North America, in a slighter large planet with the right ocean current it could be very hard to find with ships such as the ones Columbus had (XVI c not XVIII). Currents at this point in time where decisive.
This example also implies that you keep the historical conditions of our world. Only people from one continent tried hard enough to search for an alternative merchant route and had enough technical abilities to exploit the discovery.