Can a civilization collapse to the Stone Age?
A civilization? Sure. All you need to do is wipe out their source of power/electricity, and leave them in a state chaotic enough that they can't rebuild. A lot of economies have collapsed like that, to varying extents.
Why is electricity so vital?
The reason is simple. Without electricity... how will you cook? How will you keep yourself warm? How will you see at night? How will you know what's going on in the world? How will you communicate? How will you get food from the store, if they are having all these same problems,and all the food rots from lack of refrigeration? How will you get clean drinking water?
Few people have methods of dealing with these problems, if the grid breaks down entirely. Even if you personally could manage, the inability of those around you is going to create a lot of chaos and damage.
Won't outsiders help you get back on your feet?
What about outside aid? Even the USA working to help Puerto Rico is failing. That's one of the foremost economic powers assisting a small island nation.
Isn't it hard to cause this?
Not at all. Detroit, for example, is bringing in cattle and using a local community currency in place of money. Electricity is becoming a rarity there, where neighbours are working together to try and get enough money to get electricity.
That goes to show how important the NATION'S organization is. If it's good, then you can kill 90% of the populace, and they'll rebuild themselves within three generations. If it is bad, then they'll personally cause their regression to stone age cattle farmers.
But it's not really the stone age, right?
It basically is. Again, how will you eat? A lot of people will be scavenging for barbecues and fuel, but others will have to try and cook over wood fires like cavemen.
How will you keep yourself warm in the winter? I you're lucky, you still have an intact insulated house, which will remain warm without electric heating. If not, or your house falls into disrepair... you'll have to try and scavenge enough clothes and blankets and fuel for fire. When those run short, you'll have to hunt animals for furs.
Why not just rebuild?
Again, organization. Puerto Rico can't simply, "just rebuild" because it would be too expensive, and their economy is in shambles. That's why the USA isn't paying many billions of dollars to redo their whole power infrastructure, because it's so expensive and difficult.
And that's the government... normal people have it WAY harder. If you can't find a power generator, what are you doing to do... make one? Because that's a really complicated device to try and make from whatever tools you can scavenge with a coal furnace (if you can even obtain a coal furnace). And many of the components, you just won't know how to make. The microchips and delicate electronics we use in so many of this era's devices.
In the case of an EMP, it will be very hard to even find many components, as most will be frazzled.
But this couldn't happen to a major nation, right?
Puerto Rico was once one of the largest trading ports in the world. There's no such thing as, "too big to fail".
There is concern that an EMP from North Korea could drive the USA into the stone age. The cost of repairs would be astronomical. The US already badly neglects its infrastructure, so whether they can foot the bill for repairs when the economy grinds to a halt until the repairs is in question.
Civil unrest would make recovery difficult or impossible. Even under good conditions, it is likely many areas of the USA would go without power for decades (many areas already have problems with bad roads, sewage, and power infrastructure).
What would this world look like?
Worse than the poorest parts of India. In many places, India is cellphones and wood fires. This would be wood fires, copper knives, and a few AK-47s, fighting overs cans of beans.
Though, actually, the knives and tools will probably be made of good steel. There's so much scrap steel around, it's much easier to tear apart a car and use leaf-spring to forge a knife, rather than trying to make your own iron blooms or melt down wires to get copper. If not for scavengeable steel, in fact... it would literally be the stone age, no one would even be able to mine copper, much less bronze. They would have to use stone tools, as stones would be something you could still find.
Luckily, there's plenty of scrap to forage for.