In humans, I'm pretty sure it wouldn't, for the reasons listed at the end of this answer. In other creatures, though, perhaps if the ear simply senses the energy of the sound waves reaching it, and reports that as a fluctuating signal. There would likely need to be multiple "ears", that trap sound from various directions, since it would be impossible to have more than one person talking at once without being able to have the brain focus in on one voice; the overlapping volumes would change the meaning of the speech. The more ears they have, the better the brain would be able to process the input and focus in on individual voices. Perhaps a series of several pits around their head, each one of which senses the relative energy of the sound waves reaching it.
Even then, they would be much more likely to develop a language based on timed pulses, ala Morse code. For the creatures to not develop this, they would have to have a limitation in their ability to sense passage of time. The time sense of these creatures would have to fluctuate wildly, to the point that they would be unable to reliably tell the difference between a dot and a dash. Perhaps their brains are sort of like modern computer processors, which change processing speed from moment to moment as needed, slowing down to conserve energy, and speeding up when decisions need to be made; with the sense of the flow of time adjusting accordingly.
With these limitations, it would rule out the language being based on a constant sound with modulated volume; it would need to be repeated bursts of sound with each burst being at a constant volume throughout, since timing can't be considered. It might take the form of some sort of barking sound, with the barks varying in volume. A conversation might be begun by an individual "barking" a few times to set the initial volume of the conversation. If the other individual wishes to speak (or loses track of the conversation) they might break in in the same way, and wait for the other to make a statement relinquishing control of the conversation before saying their piece.
You are correct that relative volume would be necessary. That could be in one of two manners: volume relative to the previous sound burst, or relative to the initial calibrated volume. Making it relative to the previous burst would require recalibration if the speaker reached the top of their vocal range. One way to do this is being limited to three phonemes (quiter, same, and louder) so that after a long series of louder phonemes one could modulate very far down in volume in one step without changing the meaning.
Still, very significant hurdles would need to be overcome:
- Any relative movement between speaker and listener would have to be carefully compensated for, since increased distance would change the volume
- Sound is usually somewhat focused; turning your head (or sound generation organ) slightly to the side would change the volume, and therefore the meaning of your statement
- Any conversation would have to begin with an exchange of volume calibration noises, and if one got distracted in the middle, non-verbal communication would have to be used to reset the calibration, making extended conversations difficult
- Changes in ambient noise can affect the conversants unequally, leading to loss of calibration even in the best situations