The Enemy Of My Enemy
Traditionally the thing which will unite two foes (even bitter ones) is a common enemy.
If the entity who overthrew King B is a very scary and dangerous foe, such as an evil sorcerer, or the leader of a powerful and fanatical cult, then King A will have a good reason to be afraid of having such a neighbor on his border.
Furthermore, this new enemy must have a reputation. Someone who will 100% pose a threat to King A.
Justify An Alliance
King A tried to conquer Kingdom B and failed. He knows that a strong foe has seized control of Kingdom B, and furthermore, that the people of Kingdom B hate him. They would rather be ruled by the Usurper than King A.
However, King B is still alive, and still has a small force loyal to him. Furthermore, his presence on the battlefield might inspire the Usurper's units to defect back to their King's side.
It makes sense for King A to back King B's power struggle, because then his armies will have a much easier time advancing through Kingdom B (the people won't be fighting them, as they are supporting their rightful King).
Keeping Everyone Honest
But the real problem is keeping both Kings honest.
What's to stop King A from executing King B alongside the Usurper, and seizing control of the battle-weary Kingdom B?
What's to stop King B (once he's back in power) from ambushing Army A in a convenient location, and then invading Kingdom A?
High-Cost of War
If the battle against this common foe was very damaging for both sides, then neither of them may be in a position to take the upper hand.
You might describe both Kings as having lain plans to betray the other, but having campaigned together for months (or years), new-found respect for each other could cause them both to seek future friendship over yet another war.
The Threat Still Looms
This might make more sense if the Usurper is not killed, and remains a possible threat against them both.
Rather than fight each other and give their common enemy the chance to strike when the surviving King is at his weakest, the two kings might rather cooperate and be ready for a future assault by their common enemy.
Furthermore, the marriage of their children can solidify any alliance.