Would it be possible to make a crude scale mail out of oyster or some other mussel shells. If so how effective would it be? Do I need to invent some kind of super tough oyster for it to work?
Real shell armor does exist. How practical it was is unknown however.
The breastplate of tula was made by drilling and sewing hundreds of shell fragments to a leather base. Helmets of cowry shells also exist. The big issue is it would not survive many hits, shells are brittle and the bigger the individual pieces of shell the weaker that will make the armor because the more that will end up broken after each hit. I apologize the museum website may require some scrolling to see the close up image.
Could you do it? Absolutely yes. Would it be WORTH doing? Well, that's trickier.
Oyster shells are very, VERY tough. Tough enough that the DoD is currently trying to replicate its qualities in modern armor. So toughness isn't an issue at all, and if you could actually MAKE some, it would make very effective protection.
That's where this gets tricky though: How you do you attach the shells to an underlayer or to each other in a way where they won't just constantly be coming off when struck. When you're crafting the scales yourself out of metal or whatnot, you can control the shape to have proper attachment points. Oysters don't NORMALLY have any convenient way to firmly attach them to anything else. If we were trying to do this in our world with our oysters, the way you'd probably have to do it would be by drilling holes in the oyster shells so you could use laces to attach them. This is how traditional scale armor was constructed back in the day.
Of course, the very qualities that make you want to use oyster shells make them VERY difficult to drill, so the time and effort you'd spent making a set of armor like this would be all out of proportion to any reasonable benefit.
I see two possible solutions: You either introduce a tool or technique that's particularly effective at easily putting holes in oyster shells, but impractical as a weapon (some kind of acid might do nicely), OR you find or create a species of oyster that has particularly pronounced protrusions that can easily be used to secure it to armor.
A couple layers of shells should provide a good set of armor if you have nothing else, though they will certainly start cracking with every good hit so don't expect it to last long.
For every question, there's an oddly specific research paper that gives the answer: Shell Hardness and Compressive Strength of the Eastern Oyster, Crassostrea virginica, and the Asian Oyster, Crassostrea ariakensis.
According to the above paper, the compression strength(ability to not crack) of oyster shells varies based on thickness. About 1000 Newtons are needed to crack a 1mm shell, 2000N to crack a 3mm shell, and 4000N to crack a 5.5mm shell. According to this site, a good swing of a baseball bat hits a baseball with 3400N of force, so if the oyster shells in the armor are 5mm or more it would provide decent protection.
However, this Physics StackExchange answer goes into how much force a hammer hits a nail with: more than 9000N, which is likely closer to what armor would need to withstand in a fight. Thankfully, the original paper shows that compression strength increases non-linearly as shell thickness increases, meaning increasing thickness leads to greater and greater resistance to cracking.
Although it wasn't measured, a 7mm thickness would resist up to 7000N of force, and 10mm should resist at least 10000N if the increase continues at the same rate. Though there's probably diminishing returns, a couple layers of shells should give the strength needed to resist glancing blows, and maybe even a few direct hits before shattering.