What you are looking for is a confusion matrix for visual speech recognition, of digits, ideally from a real-life experiment. I tried to find some references on the matter. Most address the issue from an automation perspective, attempting to create models that can perform "lip-reading" in the AI sense. There doesn't seem to be a consensus on the matter, however, the following information might have the potential to pinpoint what you could/should take into account, if you would like to make a choice on your own.
- In this paper, some 30 people spoke the digits from 0-9 and a complicated "artificially-intelligent" system was set up to recognize the spoken digits visually. Their result is visualized as follows (original source):
Cell (row, col) in this image represents how often the actual digit row was identified by the model as digit col. For some reason, the horizontal axis is termed "Predicted class" although I think it should be termed "Target class", as it typically represents the actual value. Lighter colors represent higher percentages, and diagonal cells, obviously, represent correct identifications. One thing that probably stands out is the slightly higher tendency to confuse zero with seven (both have two syllables, huh?) and four with two and three. Also, nine was only correctly detected 60% of the time and one and five seem to be the most distinguishable (by a small margin, of course) digits.
- In this paper, 10 subjects spoke digits from 1 to 10, and a different model was set up for visual speech recognition. The final confusion matrix is (snapshot of Table 4 from the linked paper):
Nine was also correctly identified fewer times than the other numbers, with ten also showing quite a low detection success rate. This paper uses a smaller dataset and the final result is not extremely telling, plus it uses a rather different underlying recognition model. I don't think we can draw more conclusive results from this. Nevertheless, the results technically contains what you are asking for, i.e. the confusion probability between visually recognized lip-spoken digits, albeit not by an actual human, rather by a model.
- In this paper, we can see another confusion matrix, visualized in the same spirit, for digits 0-9 (Fig. 3 from the original):
Similar to the result of paper 1 above, five seems to be less confused in general, with nine being practically the hardest to recognize correctly. One apparent confusion arises between zero and two (I would spuriously attribute that to the two ending in the "uw" viseme, which makes it important to not miss any "frames" during lip-reading, in order to correctly distinguish between those two). In fact, to quote the original paper:
In the CUAVE dataset, number pairs zero and two, six and nine were
most frequently confused. Zero and two share similar viseme sequences
near the end of the utterance while six and nine share similar viseme
sequences at the start of the utterance which explains the more
frequently occurring confusions for these number pairs.
Maybe we are getting somewhere... Let's travel back in time, a slight bit....
- In this paper of 1994, publication-related link, an actual human confusion matrix is given, together with that of a contemporary best artificial system, for the visual recognition of the first four English digits (1,2,3,4):
The results are from 9 subjects, 3 of which are hearing-impaired and having been taught to lip-read at 2-8 years of age. The paper argues that the two matrices have a correlation of 0.99, which renders the artificial system a very good approximation of an actual human, as far as "confusability" is concerned. Three seems to be the most confused number among those (I would say somewhat counter-intuitively, no?).
Almost there, just one more piece in line:
- In this paper, using another interesting lip-reading recognition model based on visual space transformations and neural-network classifiers, the authors arrive at the following two confusion matrices (two alternative though similar models), tested, again, on the CUAVE database, from which, apparently, the model of the authors attempts to recognize spoken digits from 7 individuals:
R.R. refers to Recognition Rate (%). Note that each row sums to 35, meaning that rows represent input numbers, and columns represent what was the actual identification (elements of the diagonal represent correct identifications). In short, seven seems easier to confuse, but zero even more so! Seven was confused with six slightly more often in the 2-d model, while three was confused with six slightly more often in the 3-d model. The 2-d model also confused zero with six a lot. Both models confused almost equally often nine with eight. see END NOTE
Now, before I wrap this up, one final, but important, addition:
- In this paper, a virtual head was considered, in the context of improving acoustic intelligibility by adding a visual component. While the entire paper is interesting, I just want to highlight an important point:
For the natural head, the confusion matrix shows that the visemes
(h/n/ng) and (g/k) were less well identiﬁed than other visemes (Fig
9). This may be because the tongue movements that distinguish these
visemes from others were less visible from the external view.
So, do not forget that the tongue is not clearly visible from a nontrivial distance, which is an important consideration that adds to visual lip-reading confusion. Digits that utilize the tongue more will, technically, be confused more!
While there is no concise take-home message here (i.e. less subjective than what you can probably pick on your own), I hope this at least gives you an idea of how this is looked at from a visual speech recognition perspective. Also, while those are theoretical mathematical models trying to visually identify the digits there, you could probably get an idea of what would be easier to miss from a machine-learning perspective. Humans are definitely much better at training to understand digit visemes on lips. Also, you can find much more similar results if you search for other kinds of visemes, such as words or diphthongs etc.
END NOTE: (By "confused x with y" in this context, I mean "x was misidentified as y").