You just want one? Do the store clerks tracking a seedy customer use the same way of communication as the lovers talking in a coffee shop?
There are two major parts to making such a secret language. One is to prevent observation of the language being used, and the other is to make the language opaque to anyone who doesn't know the secret. This is true for spoken and non spoken languages.
Preventing observation is easy. In fact, its easier than it is with spoken words, because signs don't leave line of sight. All you have to do is make it hard for anyone besides the intended recipient to observe. Your lovers in the coffee shop should be more than capable of this. In fact, if they're having an affair, there's a good chance they already have their own shorthand, regardless of whether they can speak or not!
Preventing understanding is the other approach. Codewords, or in your case code-signs, would be easy to introduce for topics that we don't want others to pick up on. In our world, there's actually a remarkable amount of non-verbal communication in these forms, especially in situations where you are being hounded by someone and want others to get help.
A hybrid between them is to be subtle. Perhaps the symbols are all shown out in the open, but unless you know how to tie them together, they are meaningless. This approach is great for covert ops where the act of finding a "quiet place to talk," or in this case a hard to observe place to sign, is a hint that something is going down. Consider gang behaviors. When a police officer comes down the street, there is a sudden flurry of activity to ensure nobody is caught doing anything illegal. You might not even notice it because you aren't attuned to what they are doing.
Of course, hidden sign language like this isn't just limited to worlds that don't speak. Frank Herbert's Dune includes such a sign language for the Bene Gesserit to talk when others are watching, like here where a Reverend Mother is under the watchful guard of 8 imperial guards.