I know that gas giants are much larger than terrestrial planets, but wouldn’t several million or billion years allow for it to gain mass from collecting space debris and the like?
First, gas giants are many times heavier than terrestrial (rocky) planets. The lightest gas giant in our solar system, Uranus, weighs in on 14.536 Earth masses (Neptune is smaller, but a bit heavier). Even if all the asteroids in the Asteroid Belt (total mass 3% of the mass of the Moon) and all the objects in the Kuiper Belt (total mass ~ 1/25 to 1/10 the mass of the Earth) were poured onto the Earth, it would not get appreciately heavier. Even the Oort Cloud - the teneous sphere of proto-cometary bodies surrounding the Solar System out to ca. 3 light years - only has an estimated total mass 10–100 times that of Earth.
Second, a gas giant isn't just a massive planet. It has to consist mainly of hydrogen and helium. Even if you could pile enough asteroids and comets onto the Earth to make it as heavy as a gas giant, it would be an extremely different beast. Almost all the hydrogen and helium in the solar system were caught up in the Sun and a few gas giants while the solar system was formed - there simply isn't enough gas floating around to slowly transform a terrestrial planet into a gas giant.
Third, it is believed that a gas giant couldn't form as close to the sun as the Earth is, since the solar wind would slowly strip the hydrogen atmosphere away. Kepler has discovered a few "hot Jupiters"; gas giants close to their suns, but it is believed they were formed further out and moved closer due to gravitational influence from other large bodies - and that they will shrink if they stay close to their suns.
The closest possibility I can think of is that a gas giant could absorb a terrestrial planet - but I doubt that this is what you're looking for.
You'd need a number of unlikely things to happen, chief amongst which is an incoming supply of material that doesn't deliver so much energy that it blasts stuff back out into space but exceeds the rate at which the atmosphere is depleted due to effects like the solar wind or Jeans escape.
If you want your world to become a gas giant, then you need to first increase its escape velocity to the point that it significantly exceeds the average drift velocities of light elements like hydrogen and helium at the temperature of that planet.
As you can see, Earth would need to increase its escape velocity to nearly 20km/s to hold on to hydrogen, and to do that its mass (assuming density stays about constant_ needs to increase by a factor of 5. This means you need four earth masses of earthlike stuff just sort of flying around, in trajectories which will cause it to fall in to the target planet instead of being catapulted out into space, and it all needs to accrete without blowing any of the rest of it away and only then can you start dropping in the multiple earth masses of water (or whatever other volatiles seem nice) onto your super-earth in order for it to form a gas world.
I don't know if there's a minimum acceptable mass to distinguish a gas giant from a mere superterrestrial, but it probably lies somewhere between about 10 and 14.5 earth masses (the latter being the mass of Uranus). So you now need another 5+ earth-masses of volatiles that will fall into your proto-gas-world, gently enough that it doesn't all immediately boil off again.
(note that you could combine these steps, if you could gently land four+ earth masses of volatiles on earth, somehow, you could avoid the need for the rocky and metallic stuff in the mass-building phase)
So that's 9 earth-masses of spare matter. The mass of the Oort cloud is only just enough to push you over the 5-earth-masses threshold, so you'd have to crash a few of the rocky inner-system worlds to get enough stuff in one place. The same is likely to hold true for any other mature system. Obviously a system still developing has its protoplanetary disk, and some of that disk is probably going to form into a gas giant, which will start off as a rocky nucleus upon which more stuff and volatiles will accrete.
It certainly seems plausible that every gas giant looked like a small rocky planet at some stage of its development.
The reason that stars in general have a mixture of giant and small, gas and rocky planets is that at a certain stage in the original formation of the planetary system, pretty much all the gas and dust near the star has been swept up. There's no more space debris to collect because it's all in the planets.
After this point, all the planets are rather stuck at their current size and composition. Any body that is not already a gas giant is out of luck.
Not in a mature system, unless something weird happens.
The density of the cloud leading to a planetary system formation is way higher than the density of the space left after the system has formed.
Thus that additional mass has to come from somewhere else, like:
- the planetary system crossing another cloud and capturing most of its mass
- a crossing with another planetary system and subsequent gravitational havoc somehow resulting in mass gain