Leave the Culture to the Greeks
While Elves are Superior on measured statistics, this does not lead to total domination in every field. Elf culture is rather limited in scope and thoughts are limited by strict ideological codes that are blinded by superiority. But in the dark places where no one can see, Elves crave the diversity of thinking that only exists in the much more freer human culture. Humans argue with the best of elves and often win by twisting the elf logic in ways they did not consider. If the elves are for cultural purposes vegetarians and hate humans because they will eat meat, the first human to point out that at least humans eat what they kill... elves leave humans to die on the field because they just don't like them... and don't kill them for meat or fuel or anything useful other than they exist wins not just this war, but every war.
Human arts are dedicated works that elves never even attempt to reproduce. Human literature sees its heights with Shakespeare... Elves with a fairy tale that is a mediocre version of "The Boy Who Cried Wolf". The Humans have pantheons of Gods and Goddesses who wield awesome powers and assisted epic heroes. If in a modern setting, Elves are absolutely addicted to human TV shows. All of them... and yes, they keep up with the Kardasians.
Elves may be physically superior, but a book on note worthy elvish cultural contributions to society is a 3x5 index card.
I ain't got time for this
One of the weaknesses that I personally build into long lived races, is that they have no sense of urgency. An elf may live to 500 years is going to perceive long-term time in different ways. Elves will take far longer to take an action than humans because they think they have time to think this through... where as humans had already committed to the action and may even have completed it, The Elf is finished in multiple rounds of evaluating the correct action for any flaws. Elves working with humans will be given a suggestion and consider the courses of the actions very thoroughly... tactically, strategically, the risk vs. reward, the ethics, the metaphysical consequences, the likelihood of success, and is midway through the likely punishments of not taking into account a hither-too unknown third party when the human thrusts the McGuffin to the Elf, and explains the Elf didn't raise any concerns so he went through with his plan. Humans are dead sooner, so they do things quicker... often with little regard for finesse or safety, even by his fellow human's standards, to say nothing of the elves' opinion this is the craziest lucky person he's ever met.
Poor incentive to fight fair
One of my favorite lines in a movie comes from a conversation between Orlando Bloom's character Will Turner and Johnny Depp's Jack Sparrow in the first Pirates of the Caribbean film. Will tells Jack that he is a superior swords man, and Jack counters that in their last fight, he won. When Will points out that Jack only won because he cheated and if the fight was fair, Will would have killed Jack. Which of course prompts Jack to point out that the statement was "Hardly incentive to fight fair."
Like Will, the elves are perfectionists and expect the humans, an inferior enemy, to fight fair and within the bounds of the rules. The humans, like Jack, realize that this means their own death and elect fight dirty. If the humans are in a defender's position, they could really come in handy as they're content with getting the enemy to win. So long as there is no capitualation, a defender in native land has better command Gurilla Tactics are historically very effective against a superior foe. Consider the U.S. Revolutionary War, in which one of the world's superpowers surrendered to a rag tag army that rushing through the woods to take cheap shots while they marched in formation. This was also instramental into the U.S.S.R's retreat from Afganistan (a country whose terrain is so terrible, it's called the "Graveyard of Empires" having been a problem for the British Empire, the U.S.S.R., the United States, and even the Mongol Empire) and (Ironically) Vietnam, which was the only time in it's history the United States lost a war. It's dirty fighting against a superior foe that believes you'll fight like them: Fair. This is the kind of mentality that was the reason so many ships were lost to submarines in World War 1... many naval powers thought that Submarines were dishonorable war machines and didn't field them... and then watch as their enemies that did slaughtered.
One likely way for the empire to form is that humans and elves would be united under a hegemonic empire rather than a traditional one. Hegemonies rely not on forcing people to join your empire, they request to join because of strong cultural ties and respect. The United States is an example of a modern Hegemony where every one of the 50 states requested to join, rather than were conquered. The modern U.S. is often accused of having a cultural empire as there are very few people in the world that don't know Mickey Mouse or Superman or the global phenomena of the Marvel Movies. If humans have better cultural development than elves, it could be that elves would rather partner with humans out of admiration and respect for where they are recognizably week. This isn't the only time two species with seemingly better skills teamed up. In real life, early humans were considered to be "Persistent Predators" and much of our physiology was built to hunt in this manner. The core concept of this method is distance hunting is that you don't have to be stronger or faster than your prey, you just have to out last them for a run. If a human was hunting a rabbit and the rabbit out ran the human until and took a rest, thinking it was safely away from the predator... fifteen minutes later it awakes to the same group of humans getting close... and runs again... and each time the same story happens... The Rabbit runs, the human follows, the rabbit thinks it's safe and wakes up way to early to rest to the human encroachment. The first creature to collapse from exhaustion is the loser... and humans have the best endurance of any animal on the planet. The only speed record among animals held by humans is the fastest Ultra-Marathon... a 100 mile run.
Of course, there is an inferior creature on this world that hunts like humans did, the wolves. As the theory of evolution states, two life forms can not occupy the same niche in the same environment. However this particular competition ended in a rather unique way. Wolves were given the offer to unite or fight. Those that united with humans today are the domesticated dogs, and earned the title of "Man's Best Friend" and in many families are given status as honorary humans and members of families. Those that did not were outmatched and nearly became extinct if not for the mercy of humans out of respect for their allies. It comes to my mind that if two species with very different physiological differences can become allies rather than natural enemies in a war for a niche condition, perhaps it's not that unlikely that two races that are similar in physiological nature in all but ear shapes could do the same. Perhaps your empire isn't one of elves conquering men or men conquering elves, but one of an alliance of elves and man against the elves and men that refused the alliance.