Like earlier answers I would agree that from an evolutionary standpoint, it wouldn't make too much of a difference to the individuals who never knew any different. Imagine if you had a sixth finger, just to flip your perspective. A four fingered hand would likely be slightly different in proportion though. The fingers would almost certainly be larger in order to compensate for the loss of gripping ability. That could take the form of either thicker fingers on an equivalent palm, or longer fingers on a smaller one. Possibly even a combination of the two.
This could have various affects depending on which direction you take, it's really up to you. Maybe longer fingers means you have increased dexterity. This could mean the species has an advantage when manipulating intricate objects, think a spider manipulating prey caught in its web (although 5 fingers also allows us certain advantages in manipulation). Or maybe thicker fingers means they have more muscle and are thus stronger. This could be handy (excuse the pun) if the species does quite a bit of climbing or needs the strength to crush hard food sources like coconuts or shellfish.
Either of those could evolve into any number of differences in technology for a more advanced race. Think of the things that we need tools to help us that wouldn't be necessary for beings with hands like that.
From a mental perspective there really shouldn't be any difference except for the also previously mentioned use of base 8 math. After all, we think in 10s because we have 10 fingers, if we had 8 it only makes sense that we would think in 8s. This could however lead to various technological differences because base 8 math is in many ways more efficient than base 10, and is after all a different type of math. Basic arithmetic should be pretty much the same, but any advanced math would change drastically. And that would even more drastically change the development of almost any advanced technology.
These are just a few of my own thoughts on the subject, but rereading your question I now realize that you were asking for disadvantages, not possible advantages. I suppose if the hand had no change in proportion, and differed only in number of digits, the hand would then have losses in the above alternatives. There would be a decrease in both dexterity and strength. The fifth finger (whichever finger you choose as the fifth) provides both increased points of contact for grip and manipulation, as well as the simple addition of more muscle. A person with 4 fingers on a hand would thus have weaker hands and be less agile when performing complex tasks.