The idea that a wizard wears a long robe is, surprisingly, a pretty old one. It's not a modern fantasy stereotype.
At least since the time of a Roman toga there was an implicit connection between long loose clothing and the lack of necessity to fight or work for a living.
In the 10th century the whole Europe started adopting Byzantine fashions, and we see kings in loose long robes on the miniatures, while fighting men and peasants wear shorter tunics. The Church had also adopted Byzantine fashions, so both Catholic and Orthodox priests and monks even now wear clothes that didn't change much from 10th century.
Free townspeople of high standing and scholars wore long robes in 15th century too.
In 17-19th century all the manners of loose silk robes of Eastern origin were worn as a part of domestic clothing by gentlemen of leisure, philosophers and writers. This banyan is an ancestor of a modern dressing gown and a bathrobe.
So we can conclude that the idea that an older man who doesn't need to fight or engage in other physical activity, but rather spends his time reading, writing, studying, teaching and discussing with other similar gentlemen, will wear long loose clothes, is entrenched enough in European culture that it doesn't need a reality check.
As for the materials, there are several ways to approach it. If we position medieval and not modern approach to sweat and body odor, you can continue wearing wool in a lot of environments. Ancient Greek and Roman clothes were mainly wool - and it gets pretty hot in Mediterranean. As other answers write, Bedouin clothes were traditionally wool.
If you are not afraid of your own sweat, wool is pretty comfortable when hot - maybe because it's a material more or less similar to human hair, being made from animal hair :)
Other materials like silk and cotton are already covered in other answers.
About the only environment you don't won't to wear wool in is warm and hot tropical. Buddhist monks still manage to wear long loose clothes there, even though it's rather a wraparound thingie and not a tailored robe.
UPD: as far as carrying the gear is concerned, in a medieval-ish world a wizard shouldn't carry all his gear with books and potions on his person. Just like a knight wouldn't always wear his armor and weapons. A knight would have a packhorse and a squire. A high-ranking wizard will have a mule and an apprentice to take care of that for him.