How would we compile our code if all our binaries disappeared?

Question

How would we compile our code if all the binaries in the world disappeared and we had only the source code? At first you might think “It’s all okay: I have my Roslyn code here”, but wait! It’s in C#! So you look at an older C# compiler which is in turn written in, say, C++, but wait... and so forth. Would you end up soldering yourself an ASM compiler?

How would we rebuild current software, if we had all the current hardware and all software source code, but no actual software? Would we follow the same path as before, or would we take some shortcuts and end up having, for example, managed code earlier? Would we skip unmanaged code entirely and end up having a Singularity-like OS? Would we build our new bootstrappers and compilers so that they can use our existing (and probably, by the time we get things running, ancient, code) or would we write everything from scratch?

Answer 1

The process will happen in stages. We will not see a bunch of geeks crowded in a room for 4 years followed by a working i7 booting Linux. Those big complicated CPUs and architectures are only a small part of the modern computing world. There are also thousands upon thousands of smaller chips which are much easier to stand up.

We would start with a small chip, like an ATtiny. It can be programmed over a SPI serial bus, which is clocked 100% by the master. This means you can have a set of 4 switches controlled by human hands, and if you twiddle them in the right order, you can push instructions into the ATtiny's memory.

Now the first program doesn't need to be much. This is a good thing, because twiddling switches like that is hard. The first application would probably be a text editor like application which lets you write an application with easy-to-use switches (i.e. set 8 switches for a byte, then hit the "confirm" key to store it into the program), and some basic read/write capabilities (seek to byte 200, replace it with an e8). Before turning it off, this would be extended to write to the flash memory on the ATtiny. Now we have a working computer (albiet a small one). When you turn it off and turn it back on, it can reload the text editor application.

Now the next piece of the puzzle is to use this to program other chips. This is easy: the same SPI interface you used to develop the first app is the one you'd use to build up apps in others. You could write a second program into flash memory which, on command, is burned into this second chip. Now we can easily duplicate our text editor program, letting multiple developers work on the challenges in paralell.

Now we can use this to program a bigger chip, like an ATmega. In reality, we could have started with the ATmega, because it has similar programability, but I felt the story had a more clear feeling if we started by programming something that's too small to hold onto any modern tool like a C++ compiler. Regardless, once we get to a large enough chip, we can develop a real assembler. Perhaps dust off one of those old dot matrix printers and print text to the parallel port. Dust off an old PS/2 keyboard and wire it directly into the ATmega. Now we have a full blown assembler, with some modicum of human decency to it.

Now the bootstrapping phase commences. We can use this ATmega to bootstrap some of the larger chips, like the i7, because we can use it to program FLASH chips, like the ones the BIOS is connected to. It's not easy, but we have a lot of engineers who are still having to work down in those layers, so someone will remember enough to get the bios working. With this, it is not unreasonable to expect working keyboard and screens to work (no more printing code to printers to read it). We should also be able to access hard disks again, so we can write code more permanently. I would expect, at this point, someone to write a bare-bones version of vi (yes, it can get more bare-bones than vi), and an assembler, both of which work on a rudimentary file system. All of this is well within today's developer's capabilities. In fact, a few of them would think it's actually kinda fun (until the Mtn. Dew runs out).

Now for the real bootstrapping phase. Someone will write an assembly driver to access a modern file system, like EXT3. It won't be pretty, but that's enough to find a copy of some ancient version of C from the early bootstrapping days. C was bootstrapped by writing a very simple C compiler which handled some (but far from all) of the commands in the C language. This simple compiler was then used to develop a more advanced version of C, and so forth, adding features each time as they become easier and easier to write.

Eventually we'd write enough of a C compiler to compile GCC itself. Once we compile one GCC compiler, the brakes get released. Within a matter of weeks we would have every single Linux application operational, including Linux itself. From there, the other operating systems could boostrap themselves.

Answer 2

We'd get on the phone to some of the people who did it the first time.

No, seriously. We have a lot of experts in this field who are still alive, and they may well hold enough knowledge between them to build a working BIOS. From there, we can carry on the process a lot faster than it was done the first time: we simply consult the next set of experts for the next bit of knowledge, which we use to construct the next level of software up.

Eventually, this process gets up to the experts at Microsoft and all the big computer companies who make and maintain the languages we use today, and they simply press F5, compile the language's source, and provide us with a fresh new set of binaries.

The bigger problem here is redistribution. If everyone's binaries have disappeared, you need to either reconstruct every computer, or have everyone buy a new one. And then for some of the new languages that aren't pre-installed, you need to beef up your distribution servers so they don't crash when everyone tries to get a copy at once.

Answer 3

Today, we won't be able to use any computer -if all binary programs disappeared-, since all of them has some firmware (e.g. the BIOS) to boot, and that firmware is inside some ROM (often, some Flash ROM).

The time where the primordial bootstrapping program could be easily entered (without the help of some other computer) is long gone. In the 1960s you could boot IBM7094 with switches!

If you change your hypothesis to "all the disks I have are erased" and if your firmware gives you some write access (and if you had all the necessary documentations on paper!) you could bootstrap your computer.

BTW, I guess that if we lost all the free software binaries on Earth we would have much more trouble than if we lost all the proprietary software binaries (I assume in both cases that source code remains available).

Read much more about bootstrapping, in particular about bootstrapping compilers, e.g. how was the first compiler written. Read also J.Pitrat's blog on bootstrapping artificial intelligence, it is dedicated to bootstrapping issues from a strong AI point of view.

See also this answer and that one to a very related question. You might also be interested by Linux From Scratch, or by Isaac Operating System (written in its own language, running on the bare metal) or even by MirageOS.

A concrete issue with bootstrapping an entire computer today is that it is quite complex (in particular, but not only, because of technological complexities: the Intel x86-64 reference manuals are many thousands of pages, and x86 & PC are very complex notably because of historical & backward compatibilities reasons), so it will take a long time (many years of work) and it is (sadly) hard to get paid for that. So there is a social issue. It might be slightly simpler with other hardware (e.g. RISC-V, which does not really exist or is much less efficient).

A related question is how to restart the Internet if or when it would crash.

Answer 4

Apocalypse Now

Early one morning, all the binaries in the world vanish. Every piece of compiled code is gone; data is still there, the code files are still intact, but every piece of compiled code is gone. Poof.

The first thing on everyone's mind will not be how to get it back, but how to survive; fly-by-wire airplanes will drop out of the sky, phones won't work, the Internet will be gone, cars won't start, power plants would shut down. All those devices and systems, large or small, have little chips that were once full of firmware. Granted, anything built in the 1980s or earlier will probably be fine; cars didn't start getting onboard computers until the 1990s, for the most part.

Still, this isn't just an engineering or programming disaster. It's an apocalypse-scale event. A good chunk of world will be without reliable food, water, or electricity; communication will be down completely.

Where to start?

The best place would be a college or university, since they would be most likely to have a working punch card reader (and a generator). Using punch cards as the initial interface, our intrepid programmers could start writing code immediately.

Of course, those old machines are terribly slow, and huge power hogs. So, our programmers use the punch-card-driven systems to program microcontrollers, or their faster cousins, microcomputers.

These tiny computers will need some sort of interface; hopefully, there is an old transistor-driven keyboard and a few old CRTs laying around. Modern keyboards and LCD screens rely on firmware. With a little code wrangling, our programmers have a computer, a keyboard, a screen, and a way to save their code.

Additionally, any devices with firmware that can be quickly rewritten will be re-flashed; some old hardware, like old CD drives, had very little firmware, but would still be able to read modern discs.

Security Fuses

Getting back what was lost will be much, much harder. Almost all modern hardware will be nothing but trash; almost all chips with a burned security fuse will be useless, turning cell phones and laptops alike into bricks. Hard drives, though the data still exists, won't have firmware to extract that data.

The best bet is open source computers. Someone, somewhere, will have a non-security-locked flash chip with some stored code. From there, the open source computer can be rebuilt, and the programmers can slowly begin to restore software. Otherwise, someone has a back up of all their code on a cheap flash drive; with a bit of hardware hacking, it's possible to extract that data, though it will be a difficult process.

Back to the Future

Once computers (however tiny) are operational, life will get a little easier, but most modern hardware will be useless. Even if the BIOS on a modern motherboard could be rebuilt, the video, ethernet, even sound cards would be missing their firmware. Software would reset to where it was in the late 1980s. It would return, gradually, but I expect it would take longer than the first time around. What with the apocalypse, and all.

Answer 5

One of the biger problems is not the compiling of source code because you would loose all of it, all drives store in binary so everything that we know as current technology would be wiped. And reprogramming it all would be a big task considering that every prototype or version would be gone. I would expect even a food shortage or other events like it considering how many small computers are in the things we use so often in our lives, like cars, power grid management, and appliances.

Answer 6

Technically, firmware is "hardware", so I would expect it to remain. I assume that "disappearing binaries" is more about the information being removed than a physical calamity which struck all information storage devices but narrowly missed anything which didn't look like running machine code. If indeed firmware remained, then I think we would have a good head start. We should be able to boot any system which can run off firmware, and at least have some primitive OS and text editing capability. The interesting problem is that if we have source code, but most of it is electronic, then it is mostly useless to us until we can restore the systems that are able to read it.

Since there is so much potentially usable software lying around, it would make the most sense to reproduce the compilers which could rebuild the software. This means society would most likely not try to reinvent programming languages from scratch until we had recovered the ones we just lost. Since we have lost the use of the internet and digital information storage, our best bet is to target the best-documented languages for which we have books. Without a doubt, a C compiler will be the first and most important high-level tool in the rebuilding. Once you have that, progress can be made very quickly. You can then rebuild entire OSes, many software tools, and compilers for a lot of languages. There is a reason that this 40 year old language still tops the TIOBE list. It is the "English" of the programming world: awkward, annoying, ubiquitous and powerful.

Since there are so many C/C++ experts in the world, once you have a system which can enter text and store bytes on disk, building a compiler should actually not be that hard. Most likely, a bunch of folks would be improving the "IDE" through raw assembly/machine code, and probably re-inventing it from scratch just to improve productivity. Many parts of a minimal OS would be brute-forced just to get this first C compiler up and running. But I'm pretty sure that getting the first self-hosting build of the compiler would be the moral equivalent of the starter on a giant engine finally firing up the flywheel so it can be self-sustaining.

In fact, this process would most likely happen in many places all over the world. It's entirely possible that Russia or Eastern Europe produces the first working C compiler "post-catastrophe" due to the number of hackers/virus writers who have to understand low-level code. Although China has a lot of hackers, they tend to take higher-level pathways into systems. I would be surprised if they created an early C compiler from scratch (although, a big group of enterprising university students may accomplish this through sheer force of will). The US and Western European hackers would have the advantage of the most C books and reference manuals available to them, and in a language they easily understand.

Now, if firmware is also zapped, things get much, much harder, along the lines of switch toggling as described by others. That is so depressing I can't even contemplate it. But I assume that the threads merge once you get to a basic console (keyboard, monitor, persistent store...whether disk, tape, flash, etc.).

Although many languages have self-hosting compilers, most of the compilers could be rebuilt from scratch in C, and most of the original language designers could aid in this effort. I think overall, the rebuild would proceed much faster than people might imagine (from basic console to self-hosting C compiler in 6 months or less). In almost all cases, folks would probably decide that it's better to simply replace what was lost and regain the functionality than to run off the rails and redesign things.

A redesign would occur if you also lost the source code. Perhaps information is retained in books, but if all electronic executables and source were lost, then I think we would see a significant redesign and shortcut to more advanced techniques. I think C would still be rebuilt from scratch, because of its status as a kind of lingua franca. And possibly Java and a few other major languages would be revived (though obviously from clean-room implementations). On the other hand, it would be much harder to restore Linux or Windows or OS X without any source code, from just books.

Interestingly, we could take this opportunity to eliminate a lot of nagging flaws from the languages, tools, and operating systems. Perhaps we wouldn't get C exactly, but a kind of enhanced C99 with a lot of legacy cruft removed. On the one hand, it would be to everyone's benefit to simply implement an exact C99 compiler, so that people from around the world could exchange C sources as the digital world was being rebuilt. This would discourage innovation. And for this reason, Linux would most likely become the de facto OS of the new era, simply because many portions of it could be restored from books and knowledge locked away in certain high-level wizards.

Probably proprietary software would simply fail to compete until the majority of functionality would be replaced. So the rebuild would most likely occur under a very open model, unless some countries noticed that they were progressing much faster than others, and could gain competitive advantage by closing off their progress from the rest of the world. At the end of the day, global commerce would force countries to re-establish international standards, so it is hard to say how long such walls could survive.

Although many failed languages would simply not be reproduced (unless by their loving creators), the most popular languages would surely be revived because of the stored value of programmers with proficiency in those languages. The same is true of tools. However, it would take a long time to rebuild something like Microsoft Office or Adobe Photoshop, let alone Windows Server 2012. These tools may never exist again, and perhaps there would be a new arms race to reinvent each software category from scratch.

Every technology with a published standard would out-compete proprietary alternatives with no public standard, simply because the standards would represent intellectual effort preserved in text that does not need to be redone. But the weakest standard technologies may be displaced by better alternatives simply because the weight of legacy has been lifted and is no longer such a great advantage to bad old solutions.

Answer 7

When we talk about computer languages and computer programming in general, we talk about languages and their level of abstraction. We once had patch panels that where used to program computers. Most computers where not trully programable but hardwired to do a certain kind of operation. Later we got programable computers.

At first, we used binary opcodes to produce code. You would need to know each opcode the processor used, and enter your program byte by byte into the computer memory. Later a shorthand form was invented, wich used words to describe operations. It was called mneomonic form. This is almost like modern assembly. A second step came with macro assemblers, wich allowed macros to be used. Those are the low level languages.

A medium level language is something like C and PL/I, they are not very abstract languages. People usually consider this as being the uttmost hacker language - "hey i am using a medium abstraction language, this should mean i am an uber hacker" - But, usually, lower abstraction languages are more prone to defects. C provides basic function calling based around function arguments, and other less primitive abstractions like arrays and structs. This is a lot easier to program than pure assembly, and if done carefully can even result in portable code. But C lacks abstractions like objects, strings (formal strings, not hackish pointer based strings).

A language with a higher level of abstraction is Java or C#. Unfortunately both languages are interpreted into a virtual machine. This means that code execution must be translated on the fly from an abstract virtual machine into something that the lower level processor understands. Wich is slow. Another language that offers very high level of abstraction is Python, Ruby etc. So people think that high level languages must be interpreted. Object Pascal (the rejected ugly son) is as abstract as Java yet produces trully compiled code (down to binary format).

Managed code is not the uttmost invention. It implies a huge price in speed. Its a kind of abstraction but not one that is without its faults.

The process of creating the ecosystem of a new computer system is called bootstraping. If you create a new computer you will need to do two steps : Create a compiler that can output code into this machine expected form and a new operating system (that will usually be written in that language).

Usually you will start with a cross compiler that will use a certain already existing system to generate code for this new system. Then this cross compiler will be used to create a basic operating system and to compile the compiler into a binary that runs on that new computer.

As real world machines are based on real components, those machines operate at a low abstraction level. You cannot have virtual machines without real machines, because a virtual machine is actualy a computer program just like any other. This means that between your program code and the hardware there exists a hollow that needs to be filled.

tl;dr

First you will need create a mneomonic assembler program from scratch (using your knowledge about the machines), then a macro assembler, and then a true compiler (usually for C language). From then on you will be able to compile everything else.

Answer 8

How would we compile our code if all the binaries in the world disappeared [...]

Have someone from the International Space Station use a Soyuz-TMA to come down with a still-working laptop with all compilers they have available. (Hey, they aren't technically in the world...)

Answer 9

Recovery would be much faster than you think

Find a pre-1986 car, drive down to the computer museum. Start up the Altair, Imsai or Digital Group box, and start it normally: toggle in a bootstrap loader.

Toggle in means setting a memory address on the 16 address switches, then setting a data value on the 8 data switches, and hitting "WRITE", then autostepping to the next memory location, rinse wash repeat.

A bootstrap loader is an extremely short, ~100 byte program that can input data from some other I/O device, like a paper tape reader.

Use the paper tape reader to input your program, or hey - here's an idea, how about an operating system. You know, like Apple Monitor (2048 bytes) which included a lot of code to emulate a TTY on Apple's internal display and keyboard, not necessary, just use a physical TTY or VT100.

And now we break for lunch.

A few hundred bytes in the OS teaches it to read/write a cassette tape. Another couple thousand teach it to read/write disk drives. I'm not speculating, this is how they worked.

Meanwhile, parallel efforts are running on

• storing the bootstrap loader in ROM somehow, to cut down on finger callouses

• a primordial disk filesystem

• a simple text editor (hmm, do VT100s still work?)

• an assembler (assembly language compiler so we can say LDA #INPUT_MODE instead of A9 03.).

And we better get dinner before all the restaurants close.

At this point we're doing command-line just as we do now, except using an actual VT100 instead of a Terminal/PuTTY window. We won't have an ls -R because we don't have a recursive filesystem yet, but small steps Ellie.

Speaking of small steps, we start writing high level languages and cross-developing; i.e. use the Imsai to cross-compile 80386 to get the Compaq 386 booted up, (mind you, your very first x86 code would be native 32-bit and have no concept of the old 64k/1MB RAM barriers), then use that to cross-compile for the PowerMac G3, etc.

The hard part is coordination

The problem is, this effort is happening independently in every single location where a functioning Altair/Imsai/Digital Group machine exists. Now the challenge is coordinating the efforts when the phones are down. If the phones worked, any museum-piece 300 baud modem would get the crews talking and sharing code.

And the phone company is not idle; they're trying to figure out how to get something up.

About a day after telecomm is up, someone will have (re)written BBS software, assuming they didn't find an old copy of BBS source code in BYTE magazine. Now we're doing the same thing we do right here. This increases the speed of the rebuild ballistically - especially since every nation in the world has identified it as a national strategic priority.

Firmware is an annoyance

Unfortunately many components nowadays, like USB fobs, have "firmware" running on a lower level of the system, and don't really have a "hard" backdoor like the Imsai front-panel to force-load a bootstrap loader. If that firmware is gone, each machine would have to be hacked or scrapped. However that won't be a problem until we get into 2000's hardware, so that Pentium and Mac Quadra 800 will probably still run fine. And with those we can redesign/replace the other.

It's true that a Quadra 800 doesn't have a front panel of switches. It does, however, have traces on its PCB, and you would have to do some hardware hacking to get a bootstrap loader into it, or just learn to burn EPROMs.

We know the destination

Since the design problems are extremely well-defined and well-understood -- all that remains is code-slinging. We don't need to reinvent the wheel, we all know what wheels look like, we only need to carve one.

Remember, almost all existing code is understood by somebody who is alive, and who if pressed could rewrite it. Better.

It would be the best thing ever.

It would let us (heh) "reboot" our computing systems design from scratch, rather than constantly drag along old legacy infrastructure that makes everything complicated. Code would be streamlined and unified partly because we have to get the world back up, and there isn't time to support a bunch of legacy ways to do that same thing. DOSbox gone. Windows gone. Flash gone. HTML/CSS/AJAX morass streamlined. Etc. etc. The government would stomp all the patent issues owing to the national emergency.

Haven't you ever wished you could just take a month and scratch-rewrite all your legacy spaghetti code and obsolete standards? Now you can.

Answer 10

Once we have a working C89 compiler, the problem is far from being over: many software can not compile without the glib that requires gcc, which is also a C++ compiler requiring C++ libraries.

So in order to compile most software, you need a C++ compiler, and to do so, you need a C++ compiler.

It will take a lot of time to achieve. And this may be enough time to make C89 software popular, to have new software to be written from scratch -- or converted -- in C89.

Lightweight programming languages like lua that are C89-only would take over the others.