Safety isn't the problem, privacy is. But the influential people who fear being spied upon by flying-carpet riders don't want to expose their true concern, so they quietly arrange a safety worry campaign. As you note, the fatality numbers don't seem to add up to a real threat. (Or it could be like automobiles in the real world, responsible for actually quite large numbers of deaths for people both in/on the vehicle and outside the vehicle, yet the populace does not seem bothered by it until there's an "awareness-raising" program.)
Since this is a false scare campaign, the creators would add in extra spurious safety concerns, however rare, with viral misleading images and videos:
- items accidentally or maliciously dropped from flying carpets harming innocent people below - falling on them or starting forest fires.
- persons falling overboard in open water (or miles up) and having virtually no hope of rescue before their grisly death.
- riders traveling unprepared to locations they lack environmental protection for, e.g. space.
- accidental, uncommanded, or unauthorized flight of carpets, a hazard for everyone involved, and especially likely to affect people emotionally if a small child takes off on one (or a medium child does so with one or more smaller siblings).
- contraband smuggling; police have no chance of catching drug-runners moving at nearly orbital speeds with the wave of a hand.
- counterfeit-quality carpets which stop working mid-flight.
But underneath it all, the primary concern is certain people don't want someone seeing what they're doing in their palatial back yards (or outdoor business parks), and while those people had the situation under control regarding satellites, helicopters, light aircraft, and model aircraft / drones, they're afraid they won't be able to control flying carpets if they become popular.