20
$\begingroup$

There's a pretty common trope that what you post on the internet is there for ever which can make for some good story lines, but how realistic is that really?

After all, all content is held on physical servers maintained by (for the most part) by companies with a smattering of other organisations & private individuals & few companies persist for longer than a couple of hundred years on average, even if we ignore that, servers die or are replaced, accidents & disasters such as fires that can wipe out a server & it's backups happen.

So common sense seems to suggest that like the human body replaces every cell over a number of years so to with the internet, over time every extant server will end up replaced piecemeal over the decades & though some server content may be loaded onto new servers several times eventually through the vagaries of chance every old post will be lost.

Question: So how long can we realistically expect posts & content to last on average?

The most ubiquitous literature copied into multiple formats in multiple languages both on & off the web it can be copied back from aside of course, it's the average persons posts we're looking at here.

I am not asking how long any literature (& especially not those with any hard copy existence beyond the web) that may be posted entire or referenced many times in many posts may survive but how long on average most ordinary individual 'everyday' posts can be expected to persist.

$\endgroup$
13
  • 3
    $\begingroup$ You can only trust your data to stick around long-term if 1. you saved it on your own computer, locally, and have backups. 2. the file format can be opened by software that you have direct control over (not an "app" that can be removed or crippled by an update, no "subscription model" software, no software that requires internet connectivity or has DRM). 3. You are conscientious about transferring all your files to new computers as you acquire them, otherwise the physical storage will become obsolete and will no longer interoperate with new machines (and will physically degrade). $\endgroup$
    – causative
    Aug 14 at 10:48
  • $\begingroup$ Even then, operating systems will change and become incompatible with the program you used to open your file. If you truly want to future-proof your data, do all of the above and convert as much of your data as possible to a probably-immortal format with a simple, open standard, like plain text or png. $\endgroup$
    – causative
    Aug 14 at 10:55
  • 8
    $\begingroup$ It depends very very much on the specific work. For example, Herodotus's book of Inquiries (or "Historiai" in Greek) is still with us, more than 2,400 years after it was written (around 430 BCE). In the mean time it was copied, recopied, translated, converted from papyrus to vellum, from vellum to paper, from manuscript to print, and from print to electronic form; I feel pretty confident that it will survive 2,400 years more. $\endgroup$
    – AlexP
    Aug 14 at 11:51
  • 1
    $\begingroup$ ... The point being that we do not have all the inane texts which were written in the same 5th century before the common era. Only those which during two millennia and a half have always been considered worthy to be preserved, and those which were extremely lucky. Fortune and Worthiness have been great filters, and I see no reason why they would ever cease to be great filters. (And if you want an average you should begin by defining the domain over which the average is to be taken, that is, to define the set of "posts and content" of which you want to estimate the lifetime.) $\endgroup$
    – AlexP
    Aug 14 at 11:57
  • 3
    $\begingroup$ @causative: Barring serious advances in anti-aging medicine, you copying your data to new machines is likely to be a far cry from long-term. $\endgroup$
    – jamesqf
    Aug 14 at 16:55
30
$\begingroup$

Yes, Internet Archaeology is already a thing, because people already use the Internet Archive’s Wayback Machine to investigate web content that is no longer hosted at its original URL.

Much like real-life archaeology, not everything will survive and what does survive may not be a representative sample of the culture of any particular time. There’s a lot of transient data on the current internet for which there is no archiving tooling - things like voice calls, private messages, chats on game servers, …

Here are some things you can do to increase the chances that your content will survive long-term:

  1. Make compelling content that people will want to store lots of copies of, like bestselling books or blockbuster movies.

  2. Make lightweight content that lives for near-free in the margins of wider corpuses. An example might be Neocities: the value of preserving any one Neocities site is low, but the full set of all Neocities sites might form a valuable collection. “Lightweight” is key here, to make it economically feasible to store the collection. It’s fully feasible with a set of small, basic HTML websites like Neocities, but very much infeasible if the content is trillions of hours of video like YouTube.

  3. Family historians might preserve the non-notable content of their ancestors indefinitely, subject to similar constraints on size/economics. It’s easy to store a bunch of your grandparents’ emails as interesting heirlooms, but not so much a hypothetical 3-tonne newspaper hoard that they left.

Ultimately, storing data costs money, so you need to think of ways to make that cost worth it, or small enough not to matter.

$\endgroup$
7
  • $\begingroup$ Storing the newspaper hoard yourself is probably pointless in any case, since most (large) newspapers already have archives of everything they have ever published - though those archives may not (yet) be accessible by the general public for free. $\endgroup$
    – alephzero
    Aug 14 at 20:42
  • $\begingroup$ The question should be: given that the IA is a non-profit, how long will their business model last and how scalable is it while accounting for Kryder's Law? E-book publishers have successfully sued them for “willful digital piracy on an industrial scale.” - it doesn't necessarily have to be cheap, it just can't cost companies with lots of lawyers optional profits, aka: copyright infringement, which is not to be confused with thievery. $\endgroup$
    – Mazura
    Aug 15 at 1:18
  • 2
    $\begingroup$ 'Make compelling content that people will want to store lots of copies of" - So, make a meme template out of it? $\endgroup$ Aug 15 at 6:57
  • 4
    $\begingroup$ This person tracked down the real-world location of an Internet meme with basically archaeology $\endgroup$
    – user253751
    Aug 15 at 15:10
  • $\begingroup$ "very much infeasible if the content is trillions of hours of video like YouTube." <-- I did some back of the envelope calculations and determined that it would be very possible for individual users to archive everything they ever viewed on YouTube with an extension that auto-reencoded it to a minimal viable quality, as they viewed it, with storage requirements much lower than the rate of growth of typical storage capacity. Such archives could then be fed into a larger archaeology project. $\endgroup$ Aug 15 at 17:05
10
$\begingroup$

Data survives for as long as someone is willing to pay the costs (equipment, power, cooling and labor) of preserving it.

Accidental loss of data is pretty much not a thing anymore, at least for professionally managed data. Everything will be replicated in multiple physical locations, and if a disk or server dies, you just insert a new one into the cluster and it automagically repopulates all its data from the remaining copies. As long as there is someone paying for those replacements, data will never get lost. And the newer equipment will have more capacity too, so you get an upgrade (lower costs per bit) as a side effect.

A more pernicious problem is having data but not the software to read it. This is very common with closed data formats, but as an industry, we’ve (mostly) learned to either preserve software along with its data or just switch to open formats.

$\endgroup$
9
  • 10
    $\begingroup$ "Having data but not the software to read it" is obsolescence, not bit rot— The BBC's Domesday Project being a famous example. "Bit rot" is when the physical storage medium degrades enough (E.G. HDD/SSD loses its charge) to corrupt the data on it. $\endgroup$
    – Will Chen
    Aug 14 at 20:55
  • 1
    $\begingroup$ First sentence is missing the other side of the coin: ... provided that no one is actively spending money to thwart the distribution of the data. I presume that in a thousand years you'd have better luck if you spoke Chinese, as from what I understand their view of IP is quite different from a Westerner's. $\endgroup$
    – Mazura
    Aug 15 at 1:34
  • 1
    $\begingroup$ @WillChen Fixed. $\endgroup$
    – StephenS
    Aug 15 at 2:24
  • $\begingroup$ Am even more pernicious problem is/will be, having the data, the software, but not the hardware or other hardware capabilities, needed. Wrong CPU, old OS, software needs some port that doesnt exist, or doesnt understand any existent video card/display...... $\endgroup$
    – Stilez
    Aug 15 at 8:51
  • 1
    $\begingroup$ @WillChen In the usages I have encountered (mostly software development), "bit rot" has nothing to do physical degradation of media, but rather refers to the way software metaphorically rots if you don't update it - not because the software itself changes (as happens in physical rotting), but because everything else changes and eventually the software no longer works in the new environment. Wikipedia claims "bit rot" also used to refer to physical storage degradation, but it's certainly not the case that that is the only meaning of the term. $\endgroup$
    – Ben
    Aug 16 at 1:49
6
$\begingroup$

According to @Andy's research here, a whopping 10% of all links in Stack Overflow posts were gone, and this was only seven years after the inception of the network. That can probably be extrapolated if you assume it's some kind of Poisson distribution.

This fact was one of my inspirations for creating the Broken Image Repairer and various offshoots which repair broken links across the Stack Exchange network. Sometimes, the script is able to find the original source in the Wayback Machine; on other occasions, often after contributions by other users in the network, we were able to locate the content in other places.

So, yes, Internet archaeology is happening, in a certain sense, already right here right now. There is even a badge named after it.

$\endgroup$
5
$\begingroup$

Content can way outlast its original form. Many classic works of 'literature' came from spoken tradition. Eg: Grimms fairy tales, many Roman/Greek plays. The same with paintings. Have you seen the Mona Lisa - no, but you have almost certainly seen images of it.

If my memory serves, the average website life expectancy is ~3 years. That includes long lived behemoths like yahoo, google, facebook. It also includes the myriad of blogs with a single 'look, I started a blog' post. The thing is: you can never tell if a site will be up for 5 days or 5 decades, and you can never tell if someone has downloaded the content and will make it available without you knowing. My local hard drive has a couple websites mirrored on it that simply don't exist anywhere anymore.....

Unless you're an odd person like me who likes to fiddle with hosting websites on a microcontroller, the servers themselves are immaterial. These days most people host on AWS. The data is stored redundantly across multiple locations, and the serving of content is probably handled by a distributed CDN like cloudflare. I have no clue how long a single 'server machine' lasts because the service is never tied to a single machine.

But think realistically. 200 years ago was the telegram, today we have the internet, tomorrow we'll have something else. I am fairly sure that whatever replaces the internet will mantain some of the content from it - just as the internet contains screenshots of photos of sketches of cave-paintings of cats.

$\endgroup$
11
  • 1
    $\begingroup$ "I am fairly sure that whatever replaces the internet will mantain some of the content from it - just as the internet contains screenshots of photos of sketches of cave-paintings of cats" // sure, but those are in the greater scheme of things the extreme rare exceptions,, how much of all the Sumerian missives documents accounts & business records have survived for instance, not exactly a lot, other than some truly ubiquitous literature from the time none practically speaking ;) $\endgroup$
    – Pelinore
    Aug 14 at 11:47
  • $\begingroup$ @Pelinore, how much of all the Sumerian missives...? Now that we understand the value of preserving the past and have the technology to actually combat extreme age through either photography, digitization or both, efforts will be made to avoid unnecessary loss. Yes, a server farm or two will burn down or have their contents eaten by viruses, but with every passing year our ability to preserve the past improves. When I write fiction about deep space colonies, I always give them a copy of the internet. Not because they will need it to survive, but so it can survive the eventual end of earth $\endgroup$ Aug 14 at 13:46
  • 3
    $\begingroup$ Re: "I have no clue how long a single 'server machine' lasts", the usual replacement cycle is 3-5 years at large organizations like AWS. Re: "efforts will be made to avoid unnecessary loss", other than archive.org and the Long Now Foundation, I can't think of anyone bothering to do that and both are operating on rather minimal budgets. Most internet content will be lost rather quickly. $\endgroup$ Aug 14 at 15:43
  • 2
    $\begingroup$ Anybody who has seriously tried to use the output of the digitization projects run by Google, Microsoft, and others. soon discovers that the work was done in such a disorganized way as to be useless. Try finding all the volumes of a multi-volume book. It's often futile. The "missing" volumes are probably somewhere, but indexed so badly that they are impossible to find. $\endgroup$
    – alephzero
    Aug 14 at 20:48
  • 1
    $\begingroup$ Whatever replaces the internet will probably be called "the internet". $\endgroup$
    – hobbs
    Aug 16 at 5:09
4
$\begingroup$

It'll depend on which society you're performing your internet archeology from, and what information that society intends to preserve, especially if data types have to be converted to be forward compatible, or have tools that are backwards compatible as necessary.

A lot of the other answers have discussed the hosting aspect of a site, but I'd like to approach this from the worldbuilding consideration of which society is the one you're viewing the internet from.

Some sites might still be up, technically, but if your internet archaeology is from other countries than the one where the content is hosted, you may run into outright blocks on the content you're attempting to gather archaeology of. This video from the Patriot Act with Hasan Minhaj about Saudia Arabia + Censorship in China is an easy example of some of the issues with regards to censorship of internet properties or content making it hard to view them from an archaeologist approach, but also skims a little bit over the case of a workaround, known as a VPN (Or Virtual Private Network).

VPNs here would allow a user to view a website as if they're in another region (i.e. I personally can't watch Last Week Tonight videos uploaded to YouTube on their channel immediately, as being in Canada, they're blocked for ~2 weeks, but if I VPN'd into the U.S., I would then be able to view them when they were initially uploaded), provided that region still exists, and has a VPN that you can connect to in their region.

In a society post our current one, if, say, a localized Mad Max situation localized to the U.K., no longer has computers on its land due to an extinction event (That probably removed people too, or people tied to the culture of the U.K. in this example.), if Netflix still has the licensing setup for the U.K. to have U.K. specific content, someone in the U.S. would need to travel to the U.K., set up the VPN connection and subsequent internet cabling to allow for a remote connection to Netflix, and then attempt to view it that way, if you wanted to see how the U.K. had been able to view Netflix, and perhaps see what was Trending before the Mad Max situation was involved.

But, okay, those are extreme situations - that's a localized societal collapse view of archeology of the internet, and ideally, we're thinking these computers are still up and running, and their information can be accessed, perhaps miracle-hosted by a few Internet Archive style groups. Surely the technology will be kept up to date and accessible?

That requires Unicode to continue being the standard for transcribing texts online (Specifically, that it'll work in UTF-8, or at least that UTF-8 will continue to be supported in the future.), and that old documentation won't be replaced as they continue to add new codepoints and specific characters, if by chance, they run out filling spaces for Wingdings to be supported by the Unicode Consortium. That's unlikely to change anytime too soon, but if we're looking at attempting to preserve internet content across every website regardless of if someone is willing to update their libraries, then the concern of forward compatibility in those documents could be of concern. Unicode itself is relatively new, and there were quite a lot of other codepoints that needed to be put together at that point that were subsets of other character sets and their ordering of characters, that to expect it to stay absolutely locked in place is unlikely. At which point, you run into issues like the one described by the Wingdings addition (Specifically, if some characters did not render correctly, they would instead jump to an unrelated character - like a "smiley face" Wingdings character into a "J" ASCII character.) In a Mad Max world with technology still around but without international internet cables, there may be a drive to develop devices that specifically jump back to older, more limited character sets, thus ruining the ability to really be able to read the documents.

Okay, that's not great, but at least the presentation will stand up for the document, right?

This is more of a minor detail than the others, but if you're looking for them to stay the same look and feel, on occasion, HTML document elements have been known to be deprecated.

Depending on how a specific site is developed with CSS components expected in question, the deprecation of a feature there could cause documents to fall out of the regular presentation as well. Admittedly, this is cherry picking at the moment, but could be more widespread after a few decades, or centuries, of internet developments being made.

Conversions of all of these aspects could be done to keep them preserved, especially at an automated scale, but if society evolves to not update especially older documents, or comment systems, to update their compatibility with newer devices, that could lead to some specific subsections of posts on the internet to become difficult to preserve from an archaeology standpoint, and may lead to deteriorating conditions of the documents in question.

$\endgroup$
2
$\begingroup$

Here's my best guesses:

  • Files uploaded to cloud storage, sure they'll be around as long as the company hosting it.
  • Free plans of web hosting like Github Pages, Wordpress, or Neocities could be up for quite a while, again, until the company shuts down, or in some cases, shuts down particular pages if not in use.
  • Shared hosting - as long as the owner keeps paying. I would expect this to have a much shorter life than, say, cloud storage.
  • VPS (my hosting plan) or Dedicated Server - Even shorter after the owner stops updating. These plans are much more expensive, and probably won't be used at all if the owner doesn't see them as worth the cost.

However, there is another option. I would like to point you to the WaybackMachine. Their project is to take static snapshots of as many webpages as possible and preserve them for as long a time as possible. Depending on how long they stay up, there could be a lot preserved for decades, or even (possibly) centuries. (do not a lot of content just can't be captured by this project, but most static information can)

Internetarchaeologists of the future, look to WaybackMachine as your savior.

$\endgroup$
2
  • 1
    $\begingroup$ as long as the company hosting it... and someone paying for it. $\endgroup$
    – user253751
    Aug 15 at 15:21
  • $\begingroup$ @user253751 True, I was thinking of free systems like Google Drive. $\endgroup$ Aug 15 at 23:50
1
$\begingroup$

Data tends to accumulate rapidly over time and it becomes very difficult after a point to decide what to keep and what to delete. Chances are all individuals and institutions will keep on backing up and archiving all the time without thinking too much about how much of it is useful.

Data storage costs will keep on falling along with advancements in searching technology(AI/ML/NLP etc) and these trends will continue far out into the distant future.

That said, Organizations can have a policy like non-critical data that has not been "touched" for 3 decades maybe safely deleted. For example if your tweets(assuming you are not a celebrity or famous) remain untouched for 30 years, delete it. So some of your data will remain forever and the rest will outlive you by a few decades.

$\endgroup$
1
  • 1
    $\begingroup$ welcome to worldbuilding, take our tour and refer to the help center as and when for guidance as to our ways. Enjoy the sights. $\endgroup$ Aug 14 at 21:20
1
$\begingroup$

Historians made a lot of work relying just on few fragments of the already scarce material written in the past. Given the amount of data we are storing now even if a very small percentage of that data will survive, there will be more than enough to conduct long and detailed studies. So rather than thinking about the average lifespan you can take into account extreme cases, those cases might involve:

  • Replication in new formats when the old ones become outdated. This requires that our society goes on for millennia without a collapse.
  • New technologies able to recover data from CDs whose plastic re-crystallised making them unreadable with our technologies. Or the ability to recover any other degraded support. This could imply sci-fi tech, but it would extend the lifespan for millennia even in case of a social collapse and the growth of a new civilization.

BTW Don't rule out the classic old paper. Some good airtight vaults with the proper choice of sources might render internet archaeology useless

$\endgroup$
0
$\begingroup$

Forever. Or at least, the lifetime of the Universe and infinity.

The same as ANY electronic spectrum information.

It will propagate forever, just as we are now receiving light from the cosmos that originated billions of years ago.

The trick is, reading it.

$\endgroup$
3
  • 3
    $\begingroup$ 1) it's the internet not a broadcast signal 2) if it was a broadcast signal you'd have to travel faster than light to get back out in front of it so you could 'read' it again which is impossible 3) a broadcast signal will eventually attenuate to the point it can no longer be 'read' so even ignoring the rest "Forever" is overly optimistic .. so I assume yours was meant as a joke answer. $\endgroup$
    – Pelinore
    Aug 14 at 15:36
  • 1
    $\begingroup$ Like I said, the problem is 'reading it'. However, the information itself survives forever. All internet signals are some form pf EM radiation, and there is always leakage. Spy satellites pick up the signals all the time. $\endgroup$ Aug 15 at 1:42
  • 2
    $\begingroup$ In information theory, it's no longer considered information if you cannot read it. $\endgroup$
    – Bergi
    Aug 15 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.