Say, that Nation A absolutely destroys Nation B in a battle and annexes them. Can Nation A realistically demand that the conquered people either convert to their religion
Sure. Cuius regio, eius religio was specific to Western Europe in the aftermath of the Protestant Revolution, but the concept is very old.
or be expelled?
If we assume that the conquered people are extremely zealous and would rather be expelled than convert, wouldn't that render the conquest fruitless as the conquered region would become empty and need to be colonized?
This reminds me of the Roman conquest of Palestine.
Things went pretty smoothly because the upper caste of Jews (the Sanhedrin mentioned in the Bible) had been Helenized, and the Romans allowed Judaism to be practiced without molestation.
But when Caesar declared himself a god, and ordered that busts of his likeness be installed in all temples, you can imagine that the monotheistic Jews were a tad irked.
They revolted in 66 A.D., the Romans stomped them down and scattered them across the Empire. (This was the third Jewish Diaspora.)
They revolted again 66 years later, and this time, the Emperor kicked all the Jews out, forbidding them reentry.