Colony population growth is different from country, or the whole continent population growth. On one hand, colony may have a plenty of arable land and untapped natural resources (which a settled nation doesn't have), which can lead to population boom. On the other hand, it is particularly vulnerable to diseases and crop failures. The famous Mayflower pilgrims could have perished, one and all, and American history books would have to look for another group of forefathers.
Let's assume the following:
- Conditions were very favorable for the colonists. There were no major crop failures or diseases;
- The new land was suitable for agriculture and colonists had access to high-yield crops like corn and potatoes;
- There was no significant immigration since the first group had arrived;
- The colony had access to all technological inventions from the rest of the world;
In this case there will be a population boom, resulting in doubling the population every 20 years or so.
2020 - 1630 = 390 years
390 / 20 = 19.5 doubling cycles
If 2020 population is 36 millions, we had to start in 1630 with just 48.5 people.
So the math works, but sticking to my assumptions for all 390 years may be unrealistic.