Please note, this was done with "seems right" figures. There is actual maths performed elsewhere, including the base of my answer. I am intentionally leaving it here, without editing my "seems right" numbers, both for archival purposes and to not have 3-4 homogeneous answers.
First, the distance
So according to a random page from google a medieval style crossbow had a total range of 380-390 yards.
So shooting up will take some distance out of it, but the bolt is "relatively" light, so lets round it to a nice 350 yards.
Now, could it take down a balloon?
The answer is "that depends". In short you'll want to have it puncture/tear a hole/several holes that leak more hot air/floaty gas than can be replaced.
I should imagine the hole made by a standard crossbow bolt is rather small and the material should be rather durable, both as defence and necessity to prevent "minor wear and tear" type things from causing a critical failure on the balloon.
Your bolts could have a very sensitive mechanism that will cause it expand into some claw like fashion for maximum tear potential upon hitting the fabric/balloon. This will increase weight, and cost, of the bolt so we'll now round the range further down to 300 yards.
Can 8-10 bolts take down a balloon?
Ultimately, it's your story so if you say it can, it can.
However, bolts are rather slow to reload so you're better to have 8-10 people fire once rather than 1 person firing 8-10 times.
With the assumptions above, it definitely seems plausible, presuming they're in range.
TL;DR
8-10 people shooting a specialist bolt should be able to take down a balloon from within 300 yards with a solid degree of plausibility. Their effectiveness is increased the closer they are as it's likely they can punch through both sides of the balloon.
Addressing comments
Hot air balloons fly between 1000 and 3000 feet
According to google, 1000 feet is 333 yards, so that puts the lower bound of flying into the upper bound of "lucky shot".
If this is going to be a hard constant, it turns "plausible" to "highly unlikely".
You mentioned this is a prototype, so I'd wager they'd be rather hesitant to fly too high for fear of coming down hard. That would mean your bandits stand a chance if they stay just above the treetops or lowish over hills/plains.
Can a medieval person find wind currents?
Predictably, maybe not. But it won't take that experienced a pilot to quickly figure out that different heights tend to have winds that blow different directions and act accordingly. This method is used by birds so I'm sure a human can figure it out.
A skilled pilot may be able to predict these with more accuracy. Even if it's "mountains tend to make the balloon want to go direction" rather than a scientific reasoning.
Please note, no maths was actually performed so actual results may vary significantly. I just applied some common sense "fridge logic".
For actual maths, please see John's answer, the result is quoted below:
As long as they are flying more than about ~ 120yards up they are perfectly safe.
As a side note longbows actually preform a bit better, they have a slightly higher max velocity, crossbows lets you fire heavier projectiles at similar velocities not faster ones. Longbows can manage 177 fps or around ~165 yards straight up. So stay around 200 yards and even the best archer on the planet can't touch them.