Air movement causes no noise until it interacts with other objects. Sound is the vibration of air. Most of the sounds we associate with a sudden change of pressure are the result of the container, not the air movement. The "pop" from a balloon is caused by the balloon rubber moving quickly through the air as it contracts. Mixing vinegar and baking soda will cause a slight "fizzy" sound as the bubbles pop. Even though the baking soda and vinegar are creating gas which is displacing nearby air, there is no real sound from that movement. You can test this with many objects. Removing the powder from a firework and igniting it will not cause the "bang" associated with their typical sound. It is the rupturing container which causes it. The exact same energy is released, but one is much quieter than the other.
The air would fill the void based on two factors. The first is gravity, with an acceleration of 9.8 m/s/s. The second is atmospheric pressure of 14.7 pounds per square inch at sea level. Nearly 15 lbs of pressure is hardly enough to cause massive destruction, and the distance it would have to travel (only several inches at most from the outer layer of skin to the core of the body) would not allow for much acceleration. The mass of air is also not very large, as your example human body compressed into a uniform space would likely fit an area of 2 square feet. The air would rush in and collide with air from the other side, the pressure would equalize and that would be it. Air is not solid enough to make much in the way of sound as it hits itself at such a low speed.
That being said, there would be a slight sound if you were standing directly next to the person. Wave your hand quickly past your ear, or stand in front of a small fan. That slight "whoosh" is what you would likely hear as the nearby air moved to fill or vacate the area. Have a person stand several feet away and wave your hand in the air towards them. Ask them what they heard. The answer will likely be "nothing", unless it is a sound from the movement of your clothing or a creaking joint. There is very little air movement further away from the localized point. The mass volume equivalent of a human is insufficient to cause a loud noise simply by moving air a few inches. To further demonstrate, you can have a person walk forward. When they do so, they are pushing aside their surface equivalent worth of air. This air flows around them like a fluid and creates eddies behind them. Actually, if you walk through water in the shallow end of a pool you can see this visually. A slight bow wave is created in front of the person with a point of lower pressure behind them which pulls in more water to equalize the pressure. Even though they are moving their surface area equivalent worth of air mass, there is no real sound associated with the movement. If you were to wave your hand next to desk covered in papers, the rustle of the papers can provide you with your sounds, but waving your hand from across the room wouldn't do anything to the papers. It is also important to remember that the higher in elevation the person is, the less air is displaced as the air is thinner. At sea level the sound would be the loudest. High up in the mountains, the sound would be quieter.
Instead of sound, you could always use the sensation of a slight wind or breeze to indicate when your character teleports. Nearby objects would move slightly as the air moved past them. Dust, paper, trash, and other debris can easily stand in for sound. If it absolutely must have sound, then you could add an ionizing effect or something similar. Transitioning from a physical to a ghost form could create a localized energy field which is compressed by the vacuum and detonates in a snap of electrical discharge (mini lightning bolt). Ghosts are "known" to be detectable as energy so it would make sense for there to be an electrical effect. Reverting back to physical form could discharge it as more of a static type halo effect as the body would be pushing away charged air.