TLDR: celibacy through reformation.
One possibility is that the religion, at its core, is unsustainable. For instance, the Shakers sprung up during a time of religious fervor (can't remember what that was called). Wikipedia claims that At its peak in the mid-19th century, there were 6,000 Shaker believers
; last I checked, there were about two of them left. This was due, in part, to people like Mother Ann Lee being hard acts to follow, but also due to the fact that the Shakers were celibate. No new Shaker babies means they had to rely solely on new recruits, who eventually stopped coming as interest waned.
Now, the fact that your religion got to 1 billion followers probably means that its members can breed. But it's possible that over time, that could change; just like Catholicism underwent many changes after the Vatican II, perhaps the leaders of your religion thought it was time to shake things up (pun intended).
If they start calling for wholesale celibacy for all followers, there will be two effects: one, the faithful will stop breeding, and two, the casuals will leave. This will give you a big initial dropoff, followed by a gradual drain over a few generations. Eventually, like the Shakers, there should only be a few left.
As for why anyone would call for celibacy, well, it depends on the tenets of your religion, and the culture that surrounds it. There have been a few cases of celibate Christian sects that thought an end to sex meant an end to man's domination over women (again, going off the Shakers, they were all about sexual equality, also the Acts of Paul and Thecla seem to imply that celibacy empowers women). Other religions tend to discourage sex since it's an indulgence in non-religious pleasure. I'm sure there are more reasons, but honestly you could make up just about anything and in the right circumstances, people would believe it.
die out
do you mean that the followers of the religion die out or the concept of that religion dies out ? $\endgroup$