Which two numbers between 1 and 10 inclusive are the most difficult to distinguish when lip-reading? [closed]

Question

The heroine in my story is a princess who has fallen in love with a commoner, but because of the differences in their ranks, they may not marry under normal circumstances. However, there is a loophole: The princess may set a public challenge for her suitors. The challenge may be in one part or many, and each part must either be an objective test, or if subjective, must be administered anonymously if judged personally by the princess or judged by a third-party panel not involved with any party.

Traditionally, the Princess may set any challenges she likes, and may aim it at a particular man, but if another can better meet her challenge, she must marry that other man.

Now, in my story, the Princess and the man she wishes to marry are attempting to cheat. In the subjective part of her challenge that she will be judging personally, each of ten suitors pick a number at random from a bucket, and the cheating suitor silently mouths his number to the Princess so that she can choose his entry that will be paraphrased and written down by an unknown scribe with only his number identifying his response.

Given that the princess is not deaf, but does have some experience in lip-reading, which two numbers between 1 and 10 inclusive are easiest to mistake for one-another when lip-reading when spoken or mouthed in English?

Since this part of the story is intended to emphasise the penalty imposed by the gods for cheating, I want the princess to misinterpret the number mouthed by her favoured suitor as a different number... hence the question: which two numbers look most alike when lip-read?

Answer 1

9 & 10

Spoken speech can be broken down into what are known as 'phonemes' or different distinct sounds. The equivalent for a lip-reader is 'viseme' or visually distinct sounds. Given that the human mouth can make different sounds with the same shape, visemes can correspond to several different phonemes. It's complicated to go through the whole list, but here's a link for the comprehensive list, originally used by Disney animators, of all things. I'll go through the numbers, one by one.

• One: The 'n' can be mistaken for 'd' or 't', giving 'what' or 'wad'
• Two: The 't' can be mistaken for 'd' or 'n', giving us 'new' or 'dew'
• Three: The 'th' and 'r' are pretty distinct, though the long e can be substituted.
• Four: 'f' can only be confused with 'v'. No luck with confusing it with any other number except five, and the rest of the word disqualifies it.
• Five: This can be amusingly read as 'vife', but the 'i' is going to sound like an 'i' cluster and the 'f' can only be swapped with 'v'. No dice.
• Six: 's' goes with 'z', and 'ix' can be swapped for most 'i' vowels, giving us words like 'sigh'.
• Seven: 's' can be 's' or 'z', 'v' can be 'v' or 'f', 'n' can be 'd', 't', 'n'. The 'e' vowel sound it uses can be 'eh', 'ey', or 'uh', which gives access to a truly staggering amount of possibilities. Albeit a lot of them aren't real words and none of them are numbers.
• Eight: The 'eigh' in 'eight' can be substituted for 'ax' or 'ah', and the 't' for 'd' and 'n'. This means that 'eight' can be lip-read as 'aught', and 'aught' is slang for zero. (Grossly incorrect slang for zero, I'll point out. But slang nonetheless.) But zero isn't on this list.
• Nine: 'n' is 'd', 't', 'n', and 'i' is the i-range of vowels, so we have words like 'tide', 'night', 'dine'. We even have 'tine'.
• Ten: 't' and 'n' both belong to the same group alongside 'd', and the 'e' is part of the 'ey', 'eh', 'uh' group. So we have words like 'den' or 'net'.

Final summary: Eight can be easily misread as 'aught', slang for zero, which would be good if zero was on the list. It's still possible, perhaps if the reader read 'aught' and assumed he might have missed a word before it, so he'd mistaken 8 for 10. That being the case, 9 and 10 are very similar as well, possessing only one major difference in their vowel, and, depending on the accent of the speaker, possibly not even that. (Though it'd have to be a thick accent.) It's also worth noting that the word 'aught' means 'all', literally, so you may be able to work off that somehow. I hope this helps.

Answer 2

What you are looking for is a confusion matrix for visual speech recognition, of digits, ideally from a real-life experiment. I tried to find some references on the matter. Most address the issue from an automation perspective, attempting to create models that can perform "lip-reading" in the AI sense. There doesn't seem to be a consensus on the matter, however, the following information might have the potential to pinpoint what you could/should take into account, if you would like to make a choice on your own.

1. In this paper, some 30 people spoke the digits from 0-9 and a complicated "artificially-intelligent" system was set up to recognize the spoken digits visually. Their result is visualized as follows (original source):

Cell (row, col) in this image represents how often the actual digit row was identified by the model as digit col. For some reason, the horizontal axis is termed "Predicted class" although I think it should be termed "Target class", as it typically represents the actual value. Lighter colors represent higher percentages, and diagonal cells, obviously, represent correct identifications. One thing that probably stands out is the slightly higher tendency to confuse zero with seven (both have two syllables, huh?) and four with two and three. Also, nine was only correctly detected 60% of the time and one and five seem to be the most distinguishable (by a small margin, of course) digits.

2. In this paper, 10 subjects spoke digits from 1 to 10, and a different model was set up for visual speech recognition. The final confusion matrix is (snapshot of Table 4 from the linked paper):

Nine was also correctly identified fewer times than the other numbers, with ten also showing quite a low detection success rate. This paper uses a smaller dataset and the final result is not extremely telling, plus it uses a rather different underlying recognition model. I don't think we can draw more conclusive results from this. Nevertheless, the results technically contains what you are asking for, i.e. the confusion probability between visually recognized lip-spoken digits, albeit not by an actual human, rather by a model.

3. In this paper, we can see another confusion matrix, visualized in the same spirit, for digits 0-9 (Fig. 3 from the original):

Similar to the result of paper 1 above, five seems to be less confused in general, with nine being practically the hardest to recognize correctly. One apparent confusion arises between zero and two (I would spuriously attribute that to the two ending in the "uw" viseme, which makes it important to not miss any "frames" during lip-reading, in order to correctly distinguish between those two). In fact, to quote the original paper:

In the CUAVE dataset, number pairs zero and two, six and nine were most frequently confused. Zero and two share similar viseme sequences near the end of the utterance while six and nine share similar viseme sequences at the start of the utterance which explains the more frequently occurring confusions for these number pairs.

Maybe we are getting somewhere... Let's travel back in time, a slight bit....

4. In this paper of 1994, publication-related link, an actual human confusion matrix is given, together with that of a contemporary best artificial system, for the visual recognition of the first four English digits (1,2,3,4):

The results are from 9 subjects, 3 of which are hearing-impaired and having been taught to lip-read at 2-8 years of age. The paper argues that the two matrices have a correlation of 0.99, which renders the artificial system a very good approximation of an actual human, as far as "confusability" is concerned. Three seems to be the most confused number among those (I would say somewhat counter-intuitively, no?).

Almost there, just one more piece in line:

5. In this paper, using another interesting lip-reading recognition model based on visual space transformations and neural-network classifiers, the authors arrive at the following two confusion matrices (two alternative though similar models), tested, again, on the CUAVE database, from which, apparently, the model of the authors attempts to recognize spoken digits from 7 individuals:

R.R. refers to Recognition Rate (%). Note that each row sums to 35, meaning that rows represent input numbers, and columns represent what was the actual identification (elements of the diagonal represent correct identifications). In short, seven seems easier to confuse, but zero even more so! Seven was confused with six slightly more often in the 2-d model, while three was confused with six slightly more often in the 3-d model. The 2-d model also confused zero with six a lot. Both models confused almost equally often nine with eight. see END NOTE

Now, before I wrap this up, one final, but important, addition:

6. In this paper, a virtual head was considered, in the context of improving acoustic intelligibility by adding a visual component. While the entire paper is interesting, I just want to highlight an important point:

For the natural head, the confusion matrix shows that the visemes (h/n/ng) and (g/k) were less well identified than other visemes (Fig 9). This may be because the tongue movements that distinguish these visemes from others were less visible from the external view.

So, do not forget that the tongue is not clearly visible from a nontrivial distance, which is an important consideration that adds to visual lip-reading confusion. Digits that utilize the tongue more will, technically, be confused more!

Bottom line

While there is no concise take-home message here (i.e. less subjective than what you can probably pick on your own), I hope this at least gives you an idea of how this is looked at from a visual speech recognition perspective. Also, while those are theoretical mathematical models trying to visually identify the digits there, you could probably get an idea of what would be easier to miss from a machine-learning perspective. Humans are definitely much better at training to understand digit visemes on lips. Also, you can find much more similar results if you search for other kinds of visemes, such as words or diphthongs etc.

END NOTE: (By "confused x with y" in this context, I mean "x was misidentified as y").

Answer 3

I don't have sources to back it up, but six and eight both have almost no lip movement, because they're mainly formed by the tongue, and are both in the [i]-range, making the lip-positions mostly indistinguishable.