One reason that large arthropods were common in the Carboniferous was that vertebrates had not yet got the hang of eating plants. Vertebrates which could eat plant matter as a primary food source didn't happen until the early reptiles cracked the problem. The food chains we have currently (diverse plants eaten by diverse vertebrate herbivores, eaten in turn by diverse vertebrate carnivores) did not dominate until the end of the Permian period.
Prior to that the vertebrates were all carnivores (think giant amphibians and early reptiles) which ate fish, arthropods and each other. So the food chain would be like these:
Plant ➜ invertebrate ➜ big invertebrate ➜ big amphibian or reptile ➜ huge amphibian or reptile.
Plant ➜ rotting plant (easier to digest) ➜ invertebrate ➜ big invertebrate ➜ big amphibian or reptile ➜ huge amphibian or reptile.
It'll probably be a bit more mixed up in the middle of those food chains, with adult big invertebrates also eating small or juvenile reptiles.
So to reset to the Carboniferous, vertebrates have to 'forget' how to eat plants. In realistic terms, this means that the plants have to become inedible in some way.
Possible reasons:
- You do a Death of Grass scenario, as in the John Christopher novel of the same name, and all plants in the grass family (Poaceae) are killed off by a fungal blight. All the grazing animals die (or have population crashes), as do all carnivores and omnivores who depend on them. This includes human civilisation falling over, as all the staple cereal crops such as rice, maize and wheat are no more.
- The gut microbes which enable vertebrates to digest the tough bits of plants (cellulose, lignin), become extinct. For instance, ungulates (hoofed animals), kangaroos, and elephants. There are LOTS of different microbes and single-celled organisms doing this, so I don't know if killing them all off is plausible. The mammal herbivores such as cows, antelopes and zebras would then be getting a fraction of the nutrition out of plants that they used to. Some species are going to die out. Some habitats, where it was difficult to get nutrition even with all the right microbes (e.g. Australia's spinifex grasslands) become uninhabitable for mammalian herbivores.
- The plants evolve (or have been genetically engineered to have) some toxin or tough material to replace cellulose, which mammals and their microbes can't cope with — but the new 'Carboniferous' arthropod herbivores and their microbes can. Again not sure how plausible this is, because there are hundreds of thousands of plant species in the world!