Start with changing the sound of your language.
To understand how sound in language changes, we need to define sound.
The linguistic term "phoneme" refers to any part of speech recognized by a native speaker as a single sound. Phonemes can be quite wide, actually. A language lacking a difference between the sounds P and B (which are articulated in the same place in the mouth) may use both in different circumstances, they are simply heard as the same sounds to a native listener.
Sound changes happen when the definition of a phoneme drifts, perhaps merging it with another one or splitting it into two.
People will speak in the absolutely easier way they can still be comprehended. A good example of this is the word "hue", at least in some forms of English.
The "hy-" sound (/ç/ in the International Phonetic Alphabet) clearly distinguishes it from the words "who" and "you" and it is, acoustically, one sound, yet it is not a phoneme as it is read as H followed by a Y (/j/ in IPA) sound. A phoneme change happens when two contextual rules like this overlap often enough and get learned by new speakers as a new distinct phoneme.
Some sounds are also just easier to produce, which is why you see them more often in languages. If your language distinguishes between P, B and an aspirated P (pronounced as with a puff of air), the aspirated P may drift into the acoustically similar, and in certain contexts easier to pronounce F, as is what happened to Germanic languages.
Try for example to go "etetetete" and then go "edededede". The first one is much choppier, no? This is because D is voiced while T is unvoiced. When alternating between an unvoiced consonant and a vowel, you're forced to turn your vocal chords on and off, which is less efficient than having them on throughout. This is also why many languages devoice consonants at the end of words - the word is over, you turn off your voice as early as possible and the D becomes a T. A language which does this does it systematically. Having D turn into T at the end of words and then not have G turn into K in the same context would need to intersect with some other change to be believable as a phonological change. This is also why most languages form a near perfect matrix when listing phonemes by type of articulation and place of articulation.
A good thing is also just to look at how real languages evolve phonologically. Look up any language, then its ancestor language and then other languages with the same ancestor. You will find interesting systematic shifts this way. Looking at them is also good to shake off your bias for what would seem easier to pronounce to native speakers, like how a language would evolve "difficult" consonant clusters from simple ones:
moróz -> mraz
And perhaps the most obvious way of making things easier to pronounce is to just leave them out all together. Again, it's quite amazing how many sounds you can leave out and still be comprehensible.
English is a perfect example of this as it's reflected in the spelling. The vast majority (some exceptions are words that have been re-spelled to fit the modern way of reading them) silent E's at the end of words were originally pronounced. Same with French and most languages with silent letters.
What's important to understand about sound deletion however is that it's never the stressed sounds, as that's what people listen for when you speak.
Your example with "whom" works perfectly here. M is at the end of the word, with typically comes right at the end of a sentence or right before a more interesting word.
Also, what gives if you leave off the M? Is it that much harder to understand its role in the sentence?
Grammatical changes as informed by sound changes
Leaving off the M would remove the case distinction, if it wasn't already clear from syntax. It is fairly exposed to the elements of sound reduction, so with those two factors out of the way, you have lost a grammatical case from your language.
Grammatical shifts are popularly theorized to work a particular cycle. Languages drift from being agglutinative to fusional to isolated and back to agglutinative again. (Definitions here). All of these stages have been observed on their own, but no language has been seen going back to square one as of yet.
The way this works is that if you slur the endings on an agglutinative language like Finnish, they will eventually merge (like in Estonian) into words having only a couple of affixes and some modifier words as in Russian, which if you slur the endings, they will eventually go away (like in Bulgarian) and turn into a language which primarily functions through syntax and use of modifier words such as prepositions, etc. As more common words, such as pronouns, get slurred, they will eventually merge around the more specific and important verbs, nouns and adjectives into long, affixed words.
One thing you may wonder about the previous paragraph is how you would get new grammar words from to replace the affixes previously filling those roles.
These are taken from regular words which get used in certain contexts often enough that they lose their original meaning (known as semantic bleaching) and are later analyzed by speakers as having some other function (grammaticalization).
A good example of this is help verbs. The verb becoming the grammatical form for the future tense usually has a meaning like "go", "come", "push", etc.
Imagine "I am going out (in order) to run" becoming "I am going to run".
Prepositions often get double roles as marking possession, telling time, even grammatical functions like marking the dative case (like "to" in English).
Even contextual sound changes become grammaticalized, which is how you get umlauts such as:
foot -> feet
Which is also why other Germanic languages use umlaut letters (Ü, Ö, Ä) to connect the umlaut to the original form:
German: Fuß -> Füße
Swedish: fot -> fötter
In terms of writing
All sound changes and grammatical changes happen faster than spelling rules can adapt. This means that most spoken languages are written in a form that is more or less a 100 year prior form of the language (very much depending on the language). Now, what makes spelling not just an outdated form of the same language is spelling reform and the addition of new words.
Systematic spelling reform is hard. The more you put it off, the harder it gets and quite often, people get used to thinking in the terms of these mutated spellings. So what happens quite often is that the words which are changed are the few which don't adhere to the new norm.
This is how we get letters like C having two sounds. Everyone knows that C is S before E and I and K before everything else. It doesn't matter that it used to be pronounced only as K, if you spell a word "cing", people will not know you mean "king", even if that word may have been one of the few remnants of words sounding the same as 500 years ago.
This is how you get Spanish and French having both "Qu" and C to represent this sound.
A good video on vowel deletion.
Another good video on sound changes.
A good video on the evolution of the Latin script. Some factual incorrectness but interesting nonetheless.
Some Wikipedia articles on specific types of sound change:
https://en.wikipedia.org/wiki/Assimilation_%28phonology%29
https://en.wikipedia.org/wiki/Dissimilation
https://en.wikipedia.org/wiki/Lenition
https://en.wikipedia.org/wiki/Fortition
https://en.wikipedia.org/wiki/Consonant_mutation
https://en.wikipedia.org/wiki/Syncope_%28phonology%29
https://en.wikipedia.org/wiki/Apocope
https://en.wikipedia.org/wiki/Epenthesis