your giant can easly fit into a forest. as you said they are only 6 meter tall, so you won't have them picking over three any time soon. pine or oak are at least twenty metter tall when grown
what im more interested in, is how young your world is. if your giant at the time to be driven to the forest after having a homeland they must have lived decade there before being driven off, and probably spend more decade spreading said rumor. i don't see how all those event could happen in less than a century, so it would have given even more time to the forest to grow. and when "engeenered" a forest can be grown quite quickly. what take extremly long is the developement of a full ecosysteme.