You would need both, separated in key areas. The problem lies in the conflicting areas of motivation. In a military society, there is duty to one's superiors. You obey them, simple as that. As long as the man at the top has good control of his men and intends to do the 'right' thing, the military system won't collapse. But since the apparent goal on this ship is a democratic society, there would need to be checks and balances which are complicated by the non-renewable nature of resources.
Quite simply, in a capitalistic society, people have the tendency to treat one another badly. Financial transactions, at the level of the individual and small enterprise (which is probably all that could exist on a generational ship) are zero-sum games. Whatever one party gets, the other party doesn't. So, it is better for either part to get whatever advantage they can over the other. In today's economic system, this is balanced by the fact that there are other vendors of almost any product. If you want to buy but don't like the price, go somewhere else. If you want to sell, but no one is buying, lower the price to make yourself competitive.
This fails on a generational ship due to lack of resources and lack of population. There isn't an unlimited supply of medicine; if you don't like the price of the ship pharmacy's heart medication, it is pay or die. It gets even worse when we consider if the ship's systems are privatized. Imagine someone with control over the air flow to different parts of the ship. That's the biggest advantage in a financial transaction you can get: pay or die slowly.
This is where the military come in. If the military, assuming that they are committed to the mission succeeding, can safely control the life support and other ship systems, there goes the biggest fault with the democratic capitalist system. The next part is where it gets tricky. The military would need to enforce some kind of welfare system. The most important resource on a generational ship is the population; they're the entire reason it's there. If the military operated as a not-for-profit, and collected taxes in the form of necessary items, these could then be distributed to the poorer factions of the ship, preventing any unneeded deaths that could damage the integrity of the genetic diversity of the sip, and therefore the mission.
To sum up, a capitalist system could work but the military would need to be in control of the ship itself and have required access to other supplies. Luxury items, food (that was better than the most basic fare) and everything else could be controlled by the capitalist system but it is important to remember why they are there. The mission on the generational ship should inform everything. The mission is why they are there and everything on the ship should be directed to making sure it succeeds.