The main reason to have a calendar is to predict important points in time (when to sow, for example). Those points are often tied to the year. Identifying points in the year is most reliably done with astronomical markers (for example, IIRC the Egyptians used the appearance of Sirius above the horizon as sign to start sowing).
Also very obvious observable periodic changes (the moon phases!) will likely enter into the calendar (if living on the sea, the moon phases will also intimately be linked to the tides; note however that adoption of lunar cycles may also have been driven by the fact that the female fertility cycle is very close to it).
The base unit of a calendar normally is the day (when there is no night-day cycle, I can imagine another base unit being used). Anything below that base unit is time keeping, and not related to the calendar (except for the defintion when a day begins; the usual calendar uses midnight — the middle between sunset and sunrise — while the Jewish calendar uses the sunset as start of the day.
After the base units are set, numeric properties play an important role. For example, the moon cycle is between 28 and 29 days. 28 = 4*7, while 29 is prime. Usually divisible units are preferred; the 7-day week most probably is related to the 28-day moon cycle. On the other hand, there are about 13 28-day months in the year, which again is a prime; however, make it 12 months of 30 days, and you have two quite divisible numbers; and even better, both are obtained by dividing the same number, 60, by a small number (2 or 5).
Of course, that way you inevitably get a mismatch of your cycles (for example, a year is between 365 and 366 days long, with 13 28-day months you get 1 to 2 days too little, with 12 30-day months, you even get 5 to 6 days too little. That can be solved by having a few "special days" to fill up the year (if the priests set the number of special days according to astronomical observations, this also saves you the trouble to make complicated rules like leap years), or it can be solved by making some sub-units a bit longer or shorter (like our calendar, where we have some months with 30 and some with 31 days, with a mostly regular pattern — the irregularities (and the much shorter February) are due to later modifications of the calendar by Roman emperors (for example the emperor Augustus — after renaming the month Sextilis to August — wanted to have his month (which up to then had 30 days) to also have 31 days, so he gave it another day, exchanged the month lengths of the following months (maybe because otherwise there had been three 31-day months in a row), and took the extra day away from the February (which already had a differing length anyway).
The latter also shows a third influence on the calendar: Messing with the calendar for various reasons, from actual time-keeping reasons (adapting the leap year rule to better fit the astronomical year, after the difference becomes problematic, as done with the Gregorian calendar) over cultural/religious reasons (resetting the date number from Roman to Christian calendar, based on the claimed birthday of Christ — the fact that Christian year numbers are divisible by four exactly if Roman year numbers are, which allows for the same leap year rule to result in the same leap years, makes me suspect that the birthday of Christ wasn't the only consideration when fixing the starting point) up to just self-esteem of emperors (the Augustus example above).
So in short, the following things affect the calendar:
- The cycles important to daily life, as well as easily observable cycles in nature (especially astronomical ones).
- Numeric properties, especially divisibility (but also the religious or superstitious significance of numbers may enter here; OTOH that often is also derived from the divisibility).
- Intentional changes to an existing calendar, often with a goal not directly related to timekeeping considerations.