Let's make a bunch of assumptions:
- The largest primary is about 3 times bigger than Jupiter.
- To really be a parent, the barycenter of a parent-satellite system must be within the parent.
- Everything has approximately the same density
- Orbital stability will magically work itself out (this will give us an upper bound)
Let's call the twice the distance between the barycenter of a parent satellite system and the farthest extent of that system $D_p$ then the corresponding diameter for the subsystem $D_s$
Now if the mass of the parent is $M_p$ and the mass of all the sub satellites together sum to $M_s$ then the requirement that barycenter be inside the parent yields:
$$\frac{M_s}{M_P}(D_p-\frac12D_s)<\left(\frac{3M_p}{4\pi\rho}\right)^\frac13$$
Now we know that for each parent none of the satellites can pass within the roche limit of the parent (the limit would actually be farther out due to the fact that the satellite system isn't solid but this will get us an upper bound) Lets call the diameter of the satellite system $D_s$ and the diameter of the parent system $D_p$. The Roche Limit gives:
$$\frac12 D_p>2.4\left(\frac{3M_p}{4\pi\rho}\right)^\frac13+D_s$$
If we claim that each subsystem is proportionate to the parent system then we have:
$$\left(\frac{D_s}{D_p}\right)^3=\frac{M_s}{M_P+M_s}$$
Now if we're trying to maximize the ratio of satellite mass to parent mass both of these inequalities should be equalities.
Solving the system yields:
$$D_p \approx 2.6 D_s$$
Which means each successive moon would weigh $17$ times as much as the previous one.
Now to get from a single atom moon to something 3 times the size of Jupiter would take:
$$\frac{\ln\left(3\frac{1.89813 × 10^{27} kg}{1.6726219 × 10^{-27} kg}\right)}{\ln(17)}=42$$
So a system could have a maximum of 42 layers if we stopped at planets as the primary body. Note however, this doesn't consider orbital stability and I have no doubt that even a system with 10 layers would be unstable on the time scale of a century.
Bigger
If we went up larger and larger, we could eventually incorporate black holes and then relativity plays havoc with the equations. However, I think that at the extremely large end, the expansion of the universe would distort and pull apart any orbits with radii on the order of billions of light years. So if we said that was the limit, then you could nest about $85$ layers, which is a lot, but I would hardly call that infinite.