Question contains 3 incompatible premises
- it would be intelligent and actively working against humanity
- cloud is extremely thin
- this cloud is moving at relativistic speeds.
they are incompatible in following combination: 1+2+3, 2+3
they are valid in following combinations: 1,2,3, 1+3, 1+2
- with the premise that they do work on principles which are already known to us, if not then resistance is futile, and the only author may save the planet and the system, by any plot he likes.
Defense 2+3
The first thing to do is to think about where it comes from because it can't be from a far away, and to think about why this pointless thing just happens. Except that, do nothing - they will burn on collision as John Dallman pointed
They are launched not from far away, or they are part of a way much much much more a bigger problem, galaxy sizes, something just enormous even for me, who argues about the viability of disassembling of planets in this century, and for which dismantling Jupiter in 100 years is not fast enough. It looks more like an invasion from another galaxy by enormous force, or someone is testing this civilization at close proximity, some thoughts here starting at "Where is a big thing, though."
Smaller the system is, and more complex tasks it has to accomplish - more fragile is the system. Probability of collision of 100nm particle with hydrogen atom of interstellar medium(at lowest 0.2 particles per cm3) is about 2e-09 chances per 1 m traveled. 10% chances not to collide and 90% chances to collide with one hydrogen particle is the 1'151'292'514m distance traveled or it is slightly more than 1/150 a.u.
Ok, let say one collision isn't fatal for it but traveling 1 a.u. mean high chances to get about 150 of such collisions, but let say it has a probability of 0.5 to switch off that unit after 150 collisions, damage it completely. And I'm very generous here because one act of fission is enough to evaporate that thing into a small plasma cloud. Which will form some kind of protective layer for the rest of them, a bit later about this situation.
So in the worst case scenario half-life of the front layer of that cloud will be 1 a.u. or about 500 seconds(as it moves with speed of light).
Calculate half-life of the cloud is a more challenging task, as plasma from front particles will shield the rest of the cloud, but still, this cloud will wear out because of some particles will sneak inside the cloud, and because that plasma shield will ablate too.
But at some extent we may have examples of such cloud, there are clouds after supernovae blasts, as an example https://en.wikipedia.org/wiki/Crab_Nebula and such
It has a diameter of 3.4 parsecs (11 ly), corresponding to an apparent diameter of some 7 arcminutes, and is expanding at a rate of about 1,500 kilometers per second (930 mi/s), or 0.5% of the speed of light.
https://en.wikipedia.org/wiki/Supernova
Such extreme catastrophes may also expel much, if not all, of its stellar material away from the star,[5] at velocities up to 30,000 km/s or 10% of the speed of light.
Gravity pull is the smallest percentage for breaking force, most of the pulling force comes from the interstellar medium.
By other words thin cloud at relativistic speed without forming a more complex structure to protect itself is incompatible with intelligence in this galaxy, or you will notice it by how all stars in the galaxy are dimming, as they(GGC masters) prepare to launch that cloud. It is like shooting pump gun underwater.
Defense 1+3
This will be a problem.
And this problem will be not the: ~100+ nanites of 100nm each heading towards Earth at speeds of .9c
But more likely some more complex structure, which is divided in chunks proportional for masses of all significant bodies, including the star (but really it depends on the size of that GGC system, it can't be too small though, or it will have problems to slowdown in our system)
Traveling as a bigger and more compact(dense) system allows to reduce of the deterioration rate of the system and allows it to repair itself etc.
1% of the sun mass at 0.9c speed is enough to evaporate our system completely, and take all heavy elements (everything except H), use H as a propellant and travel further. 2 waves expected - one evaporates the system, another collects separates extract elements from that cloud. The effect will be similar to supernova. The time between waves, 1-100 years.
The first wave may be anything, not necessary GG, just hydrogen cloud. And this thing will be seen from far far away, in the first place because it consumes the energy of previously dismantled systems.
defense - take all bio-samples from the earth, evacuate libraries and people, and head in the perpendicular direction from that thing, and pray for your gods.
Another way is to try to take control over that thing, but the most likely the star system will be lost, because it may be impossible to stop the attack even after gaining the control over the system.
Defense, more like OP it wishes, earth.
The main part here is that you have to have similar systems. It is pretty hard to do something with a stone against a tank, not impossible, but it is not likely that something good will happen there if there are no proper/matching tools that can be used.
Individual units of GGC are not a problem. They may be a bit tougher than usual microbiological live but hard gamma will efficiently disinfect them, the same way as it used for medical instruments disinfection.
They still need an energy source, and energy source will limit them how fast they can grow etc. Attack them, cut their energy sources.
This way if individual particle lands on earth somewhere you have to detect it, and do it fast enough(days, month, years). And it highly depends on which kind of disguise it may use. But small unit probably can't use any kind of disguise, because it is not native infection which spreads trough living things (they are most abundant sources of energy which is easy to extract from, another alternative is sunlight)
Sunlight will limit their grow speed by 100W per square meter on average. Consuming biological matter may be a faster way for them to grow. But as a small unit is most likely dumb as usual microbiological life, and it can be noticed by observation from satellites. Both cases can be observed from satellites. Also, entry trajectories will be seen because of gamma rays generated during the descent and it will help to determine the place of its landing.
Tool from this answer, starting from "Note about Venus scrap, snake elephant" can help to detect those particles fast(in different ways, let say just cover overall planet), most important it may help to fight those GG systems and it is less advanced (as I think) in creation as a molecular assembly.
The strength of Gray Goo is not in the capabilities of a single nanite, but in a system which it may create from themselves(or other matter as well) en masse. The difference is like to compare a human consisting of a big set of cells and some microorganism consisting of one cell.
And if it is intelligent, it will be intelligent not as single nanite, but as significant size system of those nanites. But rules of physics still apply to that system, essentially it needs energy and its capabilities will be limited by available energy. Most energy comes from sunlight, biological materials on the surface, gas/oil in the earth's crust.
GG is not some kind of an unstoppable force, a doom whatever. The problem begins when someone is too late to apply countermeasures when it is grown bigger than someone's capabilities to counteract it.
Coordination, observation, energy sources control, fast enough reaction - are keys to eliminating the problem at its start, at its source.
- Here is discussion and it drifted away from materials for space habitats to about discussing nanites threat, maybe will be interesting for someone.
Defense, more like OP it wishes, space/orbit
Single nanites can't maneuver, again they are useful as some arrangement of them, as a system of them. And as a big system, they may easily be an invading force even at 0.9999c, but as a cloud of single independent units, it is helpless. Just place a foil 100nm thick on its way and it will just annihilate. A gas cloud on its way will lead to the same effect. Basically take an asteroid and evaporate it with a thermonuclear blast or in a less spectacular way - grind it, spread the dust. Effectively, on average, it has to be like 100nm thin layer of foil(as the average density of that dust cloud on the way of nanites) to be as effective as that foil barrier.
to protect the earth in the way, it needs 1'920'000'000 tonnes of water as an example, or 2-3 times more (by mass) tonnes of rocks. Thus an asteroid of 10 cubic kilometers of rocks will be enough if it is dispersed correctly.
This method will work with single units up to pretty low speed, less than a few km per second (speed difference between a nanite unit and the shield, I'm just eyeballing that speed, maybe 10km/s.)
Defense against the carefully planned attack
If they acting like a dumb swarm, it way less a threat compared to a situation of a planned attack against humans driven by an evil intelligence.
If humans control the system(including Oort cloud), and all energy in the system, all matter sources, and attacking forces are not big enough and have less energy then available for humans - then humans can counteract. They may fail or may succeed.
But if humans do not control the system and if initial invade force is small, the attack can be already happening right now, thousands of years, somewhere in Oort cloud, or on Gas Giants, as an example, Great Red Spot may be an observable mark of such attack, we do not know why it exists, and why it is red. If the attacking system will gain more strength than we have, then there will be no chances to stop the intelligent invasion, or a self-learning system, without to create a more efficient and faster-growing and intelligent system to counteract the threat.
The only chance in this situation is to create a better system that is faster, more efficient, more intelligent, operate it in a better way than other intelligence.
Everything boils down to Defense 2+3 case, it has to be understood what is the source of those attacking particles, the reasons for the situation. Are they just symptoms of an illness in the solar system or what, and how big is the problem really.