Assuming you are looking for materials for just swords, you will want to avoid anything very brittle, and anything too soft. A blade must be strong but must not shatter. This will pretty much eliminate wood, granite, and iron from your lists, though iron is a fickle beast and deserves some more consideration, which I'll get to in a bit.
Let's look at Earth history for a minute and see how blades developed in real life. Early blades were mostly Flint or Bone or Glass, but relied primarily on razor edges or blunt chopping power to achieve anything, and would more easily be classified as axes or knives than as a sword.
The first real swords were made from Bronze. The history of bronze itself could fill a book, but suffice to say that Bronze is hard and strong enough to be used in a sword, but is not really comparable to modern weapons in any way. Bronze is relatively soft, but it does have the potential to hold a wicked sharp edge, and it is easy to sharpen due to its softness.
Bronze quickly fell out of style once Iron weapons became feasible. Pure Iron is difficult to acquire, and in the process of purifying it yields a wide variety of end products suitable for use in tools and weapons. Most iron is quite brittle, but working the metal in a forge or using other techniques can soften it and harden it into a useful blade.
This is where Steel comes into the picture. Really, Steel is just Iron that has been worked to the point of becoming softer and stronger. The exact chemistry involved is super complex and there are a variety of steels available, but the technology involved developed as a result of iron working. Something to note about steel is that the iron used in the process can create varying results with the same process. Japanese Iron, for example, is basically the worst. This is ironically why Japanese swords tended to be so well made: in order to produce a weapon worth anything, the swordsmiths had to work the iron over and over and over to get a consistency they could use.
Beyond Steel, there are very few metals (or alloys) that seem to be suited to sword making. I have read that Iridium, when worked similarly to iron, might produce a similar, yet superior alloy to Steel, but I'm unfamiliar with the science involved and Iridium is super rare. A theoretical alloy of Iron and Aluminum may produce a blade-worthy alloy, but no such metal has yet been created. My only other suggestion might be some ceramic derivative. Ceramics are usually quite brittle, but there are some used in knives designed for non-detection that hold an edge well and resist shattering. There may be ceramics suited to full swords, but none of which I am aware. One exciting possibility is a material called Metal Glass, which is a metal structured amorphously like glass; this material has properties somewhere between typical metal and ceramic.