It's been used in science fiction a number of times, though I don't know off-hand of any rpg settings that use them.
Some quick examples:
Constellation Games by Leonard Richardson has an multi-species alien civilization visit Earth via just such a wormhole. They send a probe carrying one end of the wormhole at slower-than-light speeds, then send a group of explorers through. They have both one-way and two-way wormholes, and use one-way wormholes for exploration, because sending a million people to visit a primitive society is less disruptive than letting a trillion people have casual access. Edit: I almost forgot to mention it, but an interesting detail is that wormholes have a limited lifespan, measured in the amount of mass they can transfer. This means that a thriving multi-stellar civilization would need continuous production of new wormholes, and a continuous stream of ships carrying endpoints to where they are needed, in order to maintain consistent connectivity. That's the type of detail that would make for a lot of fun gameplay in an RPG.
Glasshouse by Charles Stross has a human society living on the fringes of the solar system and in nearby stellar systems in immense habitats connected by wormholes. They're cheap enough that most people have apartments spread out over all of human space, with individual rooms connected by bi-directional wormholes. Although now that I think about it, Accelerando is really where the wormholes are mentioned more. Glasshouse is more about a group that splits itself off from the rest of the wormhole network after a war; the domestic uses are more clearly seen in Accelerando.
A much earlier book that I recall, though I cannot recall the name, was about a society that used wormholes on Earth to connect up their homes and businesses. I believe the plot was mostly about a child who goes wandering around outside; the child's mother thinks he has a mental problem. Not a very interesting story, really.
The Light of Other Days by Arthur C. Clarke (perhaps in collaboration with someone?) is quite interesting. Here the wormholes start out simply as a means of transmitting information between distant parts of the Earth without any time lag, making everyone's internet connections faster. Eventually they discover that the endpoints can be positioned outside of the equipment that generates it, and so the real story begins. They start out by placing the microscopic endpoint right in front of the eyes of a famous quarterback, to capture exactly what he was seeing when he made the big play, etc. At the end of the book they start using wormholes to scan people's minds just prior to their deaths, with the goal of bringing literally everyone who every lived back to life in reverse the order that they actually died.
I recall another where the inventor of the wormhole was blackmailed, and sent his blackmailer to live for a time on another planet. I thought it was called Gone Fishing, but a quick search doesn't turn it up.
Diaspora by Greg Egan has the descendants of humanity try to build a wormhole and fail. They spent 800 years setting up the experiment to create a wormhole (think of a particle accelerator like the LHC, but the size of the solar system), only to find out that although wormholes exist, they aren't actually shortcuts. All wormholes are as long on the inside as they are on the outside. This is a great book, superb on several levels, so I don't want to spoil it. They do ultimately discover a use for these long wormholes, and it's even more fantastic than short wormholes would have been.
As others have mentioned, if your wormhole endpoints can dial up any other wormhole endpoint then you have Stargate, which is actually quite a fun show. You could have a lot of fun games in that kind of setting.
As for plausibility, no, they're not plausible at all. The mathematics can "work" if you introduce some strange matter with negative mass, but that's precisely the sort of thing that the real laws of physics won't let you have. On the other hand, it would be hard to argue that wormholes are less plausible than FTL, so don't let that stop you. Your readers (or players) will suspend their disbelief of at least one thing in the setting.
Oh yea, and Schlock Mercenary, mentioned in the comments above, is excellent. Very good world-building there, and good stories well told.
So there is no fancy glowing portal here : simply a permanent connection that is created between two regions of space. This is indeed nearly identical to a wormhole, with the specific characteristic that both endpoints must start in one single point. The initial "pinch/hole" can after that be moved.
