Ok, since all other answers give some general advice, I instead provide a full-fledged solution, that would probably work for real use case with minor tweaking. Start from a usefull dictionary of stuff you need to bit-encode:
- List of all english words (58.000 or so, now expecting that list to include technical terms though)
- List of all technical terms, including eventual keywords for programming languages (foreach i.e). I expect it to be below 10.000 for most fields (personal estimate)
- Usefull symbols (punctuation, formatting symbols, short names for math functions, single digits and characters). would probably be less than 255 things.
- Code names (example: astronaut 1: Bill, communication received: Roger etc.
You get with something around 70.000 things to be encoded in the most efficient bit space.
Then analyze all communications in the past decades, find sequences of words or single words that are used more often.
Add to the dictionary the sequence of words. I would not be surprised if we are still below 70.000 usefull terms.
Now prune the dictionary, remove synonims without technical relevance, outdated or never used terms. I expect it to shrink to at least 30.000 terms, If done aggressively probably we can stay safe with 15.000 terms. For that part I would elect a number of people (say 1000 people), be it experts in english language and technicians/scientists/programmers/engineer. Each person is awarded with 1000 random words and it is given the task to sort that words by likely-ness to be usefulll given the context of the mission. Each person should carefully check the meaning of each word, search for synonyms etc. It would probably be a year of work because it has to be done carefully. It is not and error that I gave to much words too much people, each word will be examined by more than one person and in more than one field.
Ok, now it gets the interesting part you sort the words/setenctes/symbols using the following priorities:
- Occurrencies of words and word sequences in last decades of communications.
- Occurrencies of words and word sequences in english language.
- Evaluate if there are emergency words (like "CAME IMMEDIATLY BACK").
With less than 16.000 words, you know every single word can be encoded in 14 bits. However since you know some words are more often used than others (in example the most occurring text sequence would probably be "SPACE" or ".SPACE") you can prioritize certain words to be encoded in less bits. You can take inspiration from UTF-8 or other bit encoding schemes. UTF use 8 bits' bytes, here since you need 14 bits encoding space you could use bytes made of 5+5+7 bits:
if first bit is 0, the next 4 bits forms a word ( the 16 more often used words, guarantees a transmission time of 1 hour for those words)
if first bit is 1 and second is 0, the next 8 bits forms a word (the 255 more often used words after the first 8, so we actually reached 263 words in 10 bits)
Note that the 263 most common words may contain something really not reasonable, so those must be reviewed by a team of people. Just that thing could save thousand bytes in future communications.
if first two bits are 1 and third is 0, the next 14 bits forms a word
You could also reserve some emergency modes to encode information that is not included in the dictionary:
if the first 3 bits are 1, you enter text mode. that means you send single characters and digits there are 26 characters and 10 digits, "T,E,A,O" are not sent since those are most common letters: we have 36 - 4 letters, so 32, enough to require just 5 bits, so at this point for each letter you send 0, plus the bits to encode the character
if during text mode a 1 is received insted of the 0-starting-character you rollback to regular mode.
You could use as much special modes as you wish (coordinates, small software updates to ship etc.) Each special mode can be activated by setting first N bits to 1, or by using a special word in the dictionary (so first two bits are 1, third bit is 0, and then fourteen 1s). You can even use a custom mode to de-prioritize words on the fly. In Example:
Received sequence 1111111111100, then received XXXX (4 bits word), then received XXXYYYYY ( 8 bits word), those 2 words are swapped in dictionary. So that future communications can benefit of 5 bits less for a word that is going to be used more often. This sequence requires 14+4+8 bits (26 bits, so if you need to use a particular word more than 5 times you already saved some bits:
You had "Banana" in dictionary, it was a 8 bits word, so required 10 bits for transmission, and you had "Yes" that was a 4 bits word and required just 5 bits for transmission. If you know you are going to use Banana 6 times, you send the swap sequence for "Yes" and "Banana". The sequence is 26 bits, and Banana-4 written 6 times is 30 bits, so in total 56 bits, If you don't swap Banana, writing Banana 6 times requires 60 bits. You saved 4 bits using the swapping sequence.
In reality the swap sequence should double the limit, so it become usefull if you write Banana more than 11 times, because you have also to keep in mind that at some point you want to switch "Banana" and "Yes" again.
The good side of this encoding, is that since information is transmitted at very low pace, you can have whole teams working on saving bits with the help of computer algorithms (that can automate insertion of certain special sequences). While you are sending your first 5 bits, you have time to examine the next horus/days of communications and continuosly improve them.
Since the dictionary include sequences of words and sentences I can expect it to have in average 1 word per hour in regular communication. So basically 1 word/5 bits, which is exceptional!
ALL ABOVE THAT:
Probably all the above can be done by 1 people alone (in example I could write some programs to help and do that in one month or so), but since that task requires a lot of safety, various experties fields etc. It is highly suggested that many many people work on that to assure the minimal number of bits is used in the end.
When writing text to and from the ship, people will be helped with text editors that shows words that can't be encoded, and suggest alternative sequences or words that use less bits. The text editor will highlight in blue sequences of 4 bits, in green sequences of 8 bits, yellow the longer sequences, and suggest special modes when it thinks it is appropriate. It will add also a line break after each day of transmission (you will likely to see a lot of line breaks), and text that was already sent cannot longer be erased from text editor.
A simple example: If we run the above algorithm on the text above "ALL ABOVE THAT": with the following oline tool:
Compute words frequencies
you see that some words are really good candidate for 4 bits encoding:
And there just 150 words that are not used uniquely, those words could become good 8 bits-candidates, and also some sentences like:
- to be
- 4 bits
- more often
- of words
- you can
- the dictionary
- is 0
- 8 bits
And some senteces are good to keep in the 14 bit dictionary, in example:
- would probably be
- you could use
The following sentence:
You could use the dictionary for that
will be encoded in
You could use| the dictionary| for| that
Which means in bits:
17 | 10 | 5 | 5
Yellow | green | blue | blue
just 37 bits for a sentence!