Unless something is done to make memories reliable, this will result into a complete dystopia
Let's imagine that for a moment, anybody could alter your memories simply by touching you. Shake your friend's hand as a greeting and you suddenly remember you still owe him 100 dollar for that time he bailed you out of jail because you launched a bunny into space and got fined for it.
You're still not certain why you ever thought launching a bunny was a good idea, but hey, you clearly remember doing it.
Now this might sound silly, but you can see where I am going with this. Allowing anybody to touch you would be opening yourself to basically anything they could possibly want.
That leaves two possible solutions for dealing with the problem:
Prevent people from touching you
One solution would be to simply prevent anybody from ever touching you. People would go around in hazmat suits to prevent their memories from being tampered with, only coming out with their closest friends and family who they absolutely trust. Society as a whole might have a difficult time advancing quickly when everybody is in bulky outfits that restrict your movement a lot, but the alternative is that anybody could be turned into a terrorist with nothing more than a touch.
Don't use the human memory
If the human memory is that easily manipulated, people are not going to be able to trust their own memory, ever. You have a world with magic, you might also have advanced technology, the obvious solution would be to simply stop storing your memories in your brain and instead use something else. Upload them into the Cloud, store them in a computer chip, a magical repository, whatever.
Simply stop using your easily fooled human brain and instead store all memories in something else.