How much can you compress information?

Question

My main character has access to the entire knowledge of humanity (note: This is year 4000s so they have much more than "our humanity"). He created another consciousness for himself; this new consciousness has limited capacity which is way way smaller than the entire humanity's knowledge he has access to, so in the story he compresses this library of knowledge and puts it inside this new consciousness.

For more context, he can't expand his capacity with servers or other data storage methods, the only capacity available is that of his new consciousness.

He also doesn't have access to magic.

The main question is: is it is possible to compress knowledge infinitely? If not, what would the roughly smallest ratio you could compress information be?

He can use all science related methods be it digital or material.

Even though I said he is from year 4000s the answer doesn't have to be accessible in 4000s, time period is irrelevant to the question. I just want to know the possibilities.

EDIT - I have looked on my own, they say it's not possible to compress information infinitely, however the limit wasn't found, so I want a rather high estimate which I couldn't find.

Answer 1

Can data be compressed infinitely? No. Thank goodness. Part of the problem is the effort required to compress data represents the effort required to get at it. Thus, the more highly compressed it is the more inaccessible (aka, usless) it becomes. From our sister site, SuperUser we find the following example:

Top performers (based on compression) in this test are PAQ8 and WinRK (PWCM). They are able to compress the 300+ Mb testset to under 62 Mb (80% reduction in size) but take a minimum of 8,5 hour to complete the test. The number one program (PAQ8P) takes almost 12 hours and number four (PAQAR) even 17 hours to complete the test. WinRK, the program with the 2nd best compression (79.7%) takes about 8,5 hours

Decompression is often faster than compression, but it's still a factor you might want to consider. There comes a point where the effort to compress and decompress isn't worth the value, leaving you with the narrative necessity of increasing the limited storage capacity.

Having said that, a comment on that same post on SuperUser points out that a billion-letter notepad file containing a single letter, "A," can be easily compressed to next to nothing while a file containing random characters compresses much less efficiently. That's why you're having trouble finding hard numbers. There isn't a single number guaranteed to represent all types of data and all kinds of files. A datafile, on average, is lucky to see 60% compression. I've seen original TIFF graphic files that couldn't be compressed 2%.

let's talk instead about physical storage. I once had as mementos of my college days, a couple of 1Mb hard drives. They were five inches thick and eighteen inches in diameter. Today that same megabyte of storage fits on a space smaller than the tip of a sharpened pencil.¹

Let's combine that with the burgeoning technology of atomic manipulation. That's the ability to manipulate individual atoms. This is something we can't do yet on a practical level, but ignore that. Instead imagine constructing an atomic-level lattice that can be read optically or electrically. The compression of data physically becomes breathtaking. It has been suggested the sum of all human knowledge is approximately 295 Exabytes (an exabyte is 8x10¹⁸ bits). However, it shold be noted that humanity's knowledge is expanding at an equally breathtaking rate.

Big Data is growing exponentially. In 2006, the world was generating 161 exabytes of data each year. Just 14 years later, we churn out an additional 2.5 exabytes of data every single day. And while we’re producing loads of data now, it’s nothing compared to what’s coming—for example, storage for the media and entertainment industry alone is projected to grow about 13.3x between 2017 and 2023, and the storage capacity for human genome data is estimated to reach 40 exabytes by 2025. (Ibid.)

To be honest, a lot of that information can, as we say, be compressed. First systemically by removing duplication and making judgements in value. Knowing the dimensions of a bathroom in your home to twenty decimals of precision is, honestly, useless information. Second, through mathematical compression. Let's throw caution to the wind and suggest that in the future we figure out ways to compress better than 2:1. Let's suggest a believable 10:1. Let's then suggests (yup, a lot of assumptions are being made) that the total amount of information that your creation needs is no more than 1,000 exabytes (one zettabyte) of compressed data. Only half jokingly, I'm assuming "the entire knowledge of humanity" needn't include the pictures my little sister drew when she was five years old. Having established that there are limits in usefulness, we're really not dealing with the entirety of human knowledge. Once again... thank goodness.

So, a zettabyte of compressed data going onto an atomic lattice. Ignoring the science (because we're inventing it), let's say a single atomic peak with a ten atom valley in each direction. So we need a space of 10x10 "atoms" to store the data. Taking something somewhat (OK, almost completely) at random, let's asume a mid-sized atom like Antimony, atomic size 150 picometers. A zettabyte lattice would be 1x10²¹x100*150x10^-12 = 1,500,000,000 square meters (if I did the math correctly...). That sounds outrageous until you consider that in terms of today's technology that's one-billion terrabyte hard drives. So, 1,500 square kilometers. Assume we need a nanometer spacing betwee layers of antimony to accomodate our Clarkean data reading system and we get a cube 1.5 meters square.

OK! What did we end up with

Thank goodness for the science-fiction tag.

• Assuming a successful data compression rate of 10:1, which is 5X better than we can do consistently today.

• Assuming "the entirety of human knowledge" can be believably represented by 1 zettabyte of compressed data, meaning 10 zettabytes of uncompressed data.

• Assuming we can hardwire the decompression process into the data retreival system such that the time it takes to obtain information can be ignored.

• Assuming atomic manipulation to make the storage medium.

• Assuming a single bit of data can be read using a volume of 15 cubic attometers.

Your data storage device is 1.5 cubic meters.

However! The important part of this mental exercise is that you can define that volume and the storage it represents to be anything you want. What I've given you is the scientific references you can use to rationalize your choice.

---

_{¹ I might be wrong about this, but if I am, we're darn close to achieving that goal.}

Answer 2

On average? Not at all. It's a simple application of the pigeonhole principle: $x$ bits can convey $2^x$ different messages. $x-1$ bits can only convey half as many messages. You cannot fit all the messages that for in an $x$-bit space into fewer than $x$ bits. So for any message you want to make shorter, there must be some other message that gets longer.

The only reason compression works at all is because we don't care equally much about all possible messages. We only care about a small number of the total number of messages that can fit in $x$ bits where $x$ is really big, so we can take just those messages (the ones that look like meaningful text, or valid image files, or whatever) and re-index them to fit into a smaller bit space. Meanwhile, random-looking messages either don't change at all or get bigger to accomodate compressing meaningful messages. And the better your compression is, the more the output will look like random data, as it should efficiently fill the space of shorter messages entirely with the messages you care about, which means messages you care about should map to a uniform distribution over the space of bit strings. Thus trying to compress something that has already been compressed, and thus looks random, will either do nothing, or make it bigger, on average. And when you have data that you care about but already looks random (like encrypted data), it also cannot be compressed. That's why you should always compress before encrypting, rather than the other way around.

Answer 3

Data compression works by removing redundancies, so it depends on what is considered to be "redundant" in the sum of human knowledge.

Lossless compression handles redundancies in a way that allows them to be replicated exactly on decompression. Huffman coding for example works well with repeated strings of data by reducing them to a single figure (a "word" repeated n times and in so many places). Lossy compression on the other hand rerepresents data in a way that removes the stuff that isn't very important. In a jpeg image, high frequency information in the RGB color channels is truncated because humans suck at perceiving color compared to changes in brightness.

Data can't be losslessly compressed infinitely. If you have some algorithm that "compresses" an image down to say two bits, 01, and can faithfully reproduce the original image, then you haven't actually compressed anything. You just moved the information about the image from being in the image container to being in the algorithm. (In this extreme case, something like a lookup table.)

If the sum of human knowledge is one great big text file, then with the best tools you can expect a 4:1 compression ratio.

Compressed files have little redundancies left to enable further compression. They look a lot like binary noise, and any further compression needs to be lossy. Not really possible with plain text unless you reduce the source content (maybe lots of abbreviation, using simplified wording, etc.).

Answer 4

It's sufficient to send a nudge "research this, the result should look like this" and when tuned right, the receiving consciousness should be able to recreate all the knowledge itself.

Consider the kind of tech we already have. For example NVIDIA has already developed extremely low-bitrate video compression for human faces, based on AI. The receiving neural network knows how to animate facial expressions based on its training plus still picture of a face plus little information received.

Answer 5

Information isn't just about what is. Information is also about what could have been. Information isn't just the fact that a particular number, in binary, is 10101110, it's also about the fact that it could have been any one of 255 other possibilities. We measure information with entropy, which is the ratio of what is versus what could have been. The above number is 1 value out of 256. We typically phrase this in logarithmic terms. 8 bits implies 2^8 possibilities (256 possibilities).

When we talk of entropy, we talk about the irreducible aspects of this ratio. If some information has 8 bits of entropy, it means that there is no way to reduce it to anything less than one of 2^8=256 possibilities.

This measure of entropy is the smallest one can compress the information. If you could compress it any more, that would imply that there were fewer possible possibilities in the first place. That being said, entropy for arbitrary information is incredibly hard to calculate. Its entirely plausible that all of the knowledge of humanity might be derived from a single sentence. We've not found one so far. We don't think it comes down to that. But it is theoretically possible.

If some information can be described as the result of a random variable, we can indeed put some numbers on it. If some information describes a dice roll, we know that it has 2.58 bits of entropy (which can be computed from the fact that there are 6 equal-possible outcomes of a dice roll). If one were ever to determine a way to perfectly predict the result of such a dice roll, then we would no longer be able to claim it has 2.58 bits of entropy. It would have less. To date, we know of no way to compress the result of a dice roll.

Answer 6

Currently, the most compound storage in existence is the brain. All of 'the internet' combined could be concentrated into a space the size of an oil tanker. (This may be outdated.) This storage, despite it's immense size, could not store the information of a single human brain, let alone all of them.

From an engineering perspective, your most 'realistic' solution, is to construct a matrioshka brain, a device that uses the whole energy output of the sun for computing, and then you want to find a way to create computing smaller than a brain, which might be possible if there are use able particles smaller than qubits. Or you could just keep a copy of everyone's brain, powered directly by the star. Then using that same star, you can constantly shoot out a tight beam of energy towards your 'Avatar' who can then receive it. Depending on the reliability of this, and the distance, this could become a new plot device, as your main character may need to wait for vital information, like how we have to search sometimes to remember things.

Answer 7

How much you can compress information depends a lot on the type of information you're concerned with. You can contrive absurdly high compression ratios in a plain English representation by, for example, describing "a string of 30 quintillion A's", which has a compression ratio of exactly 1 quintillion to one. I can't even write that string because it's far too big for any consumer-grade computer to store.

But that's probably not what you're thinking of because that's not particularly useful to have a ludicrously long string of A's.

The Complete Works of William Shakespeare is 2.5MB zipped and 6.9MB in plain HTML, for a compression ratio of about 2.76:1. Thing here is that HTML is an especially redundant data format, so it compresses better than plain text would, but at the same time, there probably aren't that many tags, so for plain text Shakespeare might fall somewhere around 2.5:1

I think it's safe to assume that most literary written works would compress somewhere in this neighborhood. You could maybe do a little better with fancy algorithms available in the year 4000, so optimistically, we'll say 3:1 on average for literary works.

Technical writing, which the sciences would be based on, tend to be much more information-dense, and would thus may be limited to lower compression ratios. But it's also not that big of a difference, so we'll go with 2.8:1 for technical writing and scientific papers.

Audio and video compress pretty well because we can afford to lose some information because we won't notice the small differences anyway. Audio can be compressed to somewhere just shy of 6:1 (with MP3, though Ogg is similar) before it becomes noticeable, and you can probably go up to about 8:1 before it could be, in a sense, comparable to the quality of a human memory of an audio clip. (not that I have any data or science to back that assertion) A 720p 24fps video compresses pretty nicely (unless there is confetti) to 12.5Mbps, for a compression ratio of 56.6:1. Assuming future algorithmic improvements, you could maybe get that up to 60:1 or perhaps even 100:1 if some algorithm can get really good at throwing out detail where you typically wouldn't pay much attention.

Realistically, you're not going to be able to do much better than this. It's just not mathematically possible unless you can settle for some extremely lossy compression schemes.

It would not be possible to encode the entire body of scientific and literary knowledge into a single person's brain. I don't think you, or anyone for that matter, fully understand how many books and scientific papers have been published in just the last 100 years. You're going to have to be very selective about what information gets crammed into that brain.

Answer 8

You can PI compress. As in a number to a digit start in PI and a length. https://github.com/1184893257/PiCompress

Its horrifically slow.https://en.wikipedia.org/wiki/Chudnovsky_algorithm gets you O (n(log n)^3) as time complexity.

You can also compress a file into a game of life start pattern + cycle counter and overlay several of those in boolean operations.

https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

If you use e/Pi/ Constant Snippets as game of life start patterns, you can get pretty low, pretty fast.

http://www.shodor.org/media/content/petascale/materials/UPModules/GameOfLife/Life_Module_Document_pdf.pdf

You can also parallelize the game of life- due to it being a "physical simulation" meaning- it has a c constant at which information can affect other information within the Game of Life. So if you merge your file together from small clusters its pretty parallizable.