First off - who says mammals are dominant? There are about 10,000 species of birds, and 5,500 species of mammals. Mammals only get the 'dominant' label because (a) we are mammals and therefore biased, (b) the herds of megafauna which we like to watch in wildlife films - wildebeest, kangaroos, deer - are mammals.
Some of the birds had a bloody good go at being dominant. For instance, the terror birds were successful carnivores in South America for about 60 million years. Their mammalian competitors were marsupial carnivores, not placental carnivores for most of this. (The herbivores they were preying on were placentals). There were similar but unrelated giant killer birds in Europe and Asia - the Gastornithidae.
Where birds are going to lose out to mammals is being big. To be big they have to give up flying, because you can't be the weight of a sheep or a wolf and still get airborne. Basically to become enormous you have to become flightless: ostrich-like or terror-bird like.
Any niche which the birds don't occupy (because they need to stay small to fly) will be exploited by a mammal or reptile.
Mammals have teeth. That makes them really good at processing tough plant foods such as grass, or crunching up bones. A bird can do anything a mammal can in terms of making its intestines longer, or its stomach more efficient, but it can't chew its food. You could give your birds teeth again, but you will have turned them back into feathered dinosaurs.
Eggs can't run away. Mother mammal is lugging her babies around inside her womb, at the point that Mother Bird & Father Bird are sitting on their babies in eggs. So mammals can do things like travel while pregnant, but birds are stuck in one place until the eggs hatch. (Once hatching/birth has happened, things even up a bit). You could have some of your birds evolve live birth, and thus give them a life style more like a mammal's. There are fish, reptiles and amphibians which give birth to live young, so it's not completely outrageous to suggest a bird can too.
Eggs are size limited. The elephant bird's egg seems to be the maximum size which can be achieved. Even giant sauropod dinosaurs had eggs not much bigger than an ostrich egg. Birds therefore can't do the mammal thing of producing one enormous baby per year - they can't have a foal or a calf or a fawn. An adult ostrich and an adult deer might weigh the same amount, but the deer's newborn fawn is far ahead of the ostrich's newly hatched chick in the race to reach adult size. Again, live birth sorts this out.
Mammals have milk. It's a brilliant way of turning mum's fat and protein reserves into baby food. The milk can be specially tailored to baby's needs: high fat for baby polar bears, for instance. Birds (mostly) just have to collect what nature offers for their babies. However, pigeons and flamingoes produce crop milk which has the same function AND can be produced by both mum and dad. Make crop milk a more widespread feature of your birds, and they can feed their babies more like a mammal.