This is the second part of the series “Impressions of Our Current AI Usage”, as outlined by the introduction article.
In the first years of software development, the word “source code” didn’t exist, because code was just that: encoded machine instructions. How they were encoded changed rapidely, from flipping bits in the RAM directly by mechanical switches over feeding paper tapes with punched holes to magnetic storage. But for a long time, we worked with none or little abstraction over the actual machine code. I remember assembler code listings that had two columns of text: the first column for the computer, the second one just translating the first column into human-readable text.
And even with this little bit of clarification what the code actually does, we already needed additional software that took our source code and translated it for the machine.
With the adoption of higher-level programming languages, the additional software stack grew in depth until the distance between the source code and the actual machine instructions was big enough to warrant an intermediate layer of representation. Programming languages like Java or C# put a “byte-code layer” between our textual source code and the binary machine code. The machine we program against is no longer a real computer, but a “virtual machine” or in better words, a model of a machine. As long as we write source code that works correct with the model, we can assume that all the translation layers will find a way to run it correctly on the real computing substrate.
We are used to this kind of programming. We describe our goals using the machine model and a sophisticated machinery of software and hardware parts make it happen.
Forward to today and we use artificial intelligence (or inference using another kind of “model”) to produce source code in our favorite programming languages by describing our goals in even broader terms than before. We might mistake our prompts for natural language and think that we are able to produce source code by just saying what we want.
The question that poses itself nearly instantly is: If we invented a device that transforms natural language into machine behaviour from scratch today, would we include all the intermediate layers into its inner works? Is it really a good idea to transform natural language into higher-level source code, compile the source code into byte code and do all the weird magic to come up with a sequence of machine instructions that resemble the byte code? Isn’t it more efficient to teach the inference how the actual machine works and let it program directly?
Or, asking from the other side, who is the target audience of the generated source code if nobody reads it and the compiler only parses it once? Why does the inference invent all the variable and method names when the compiler throws them away again in the first step of its processing? Of course, right now the inference only imitates our way of working. But we work like we do because we write source code for other humans. If the human at the helm can’t read any layer of code anyway, why not jump directly to the most obscure representation of code and skip all the readability requirements?
As soon as the inference doesn’t imitate but actually learns about the target computing substrate, it will produce working code that is undecipherable for human readers, but optimal for the machine. (If you want to experience this effect in a tiny dose, I encourage you to play the “TIS-100” programming game). And because most inference users don’t need the readable code anyway, they won’t miss anything and get faster solutions with less hassle.
So my guess is that today’s source code will be a dying art, invented for humans and ignored by the machines because it doesn’t provide anything useful for them. The source code of the future will be less readable, more enigmatic and probably more efficient for the machine. Which means that human intervention or even just participation in the software production process will become more cumbersome and therefore even more limited.
My sorrow is that this distancing of the programming process from actual human oversight might provide a hard depedency on inference work alone. It would mean that humans aren’t just scales slower than the inference, but actually incapable of doing its work by hand anymore.
Schneide Blog 
Interesting thoughts. My take on this is however, that the current LLM technology *needs* a high-level language to work effectively.
In my limited understanding they will probably never be able to skip the intermediate layers because they are made for text-based languages, be it “natural” or “programming” ones.
If LLM-based AI actually was intelligent in the human sense, I could well imagine the scenario depicted by your post.
I’m not sure if I agree with the conclusion. This might be applicable for prompt once, write code once. However, usually we want to adapt software after a while. I do think that it is necessary for LLM that there is some higher level information in the code that can be correlated with new prompts. It will be really hard to infer original intent from low level code. The only way out would be fully rewriting the software on every prompt. But that would introduce new bugs for every minor change. And we couldn’t iteratively develop software like we do now. LLMs would need to be perfect in writing bug-free code. Even LLMs need high level code to “reason” about what the user is actually prompting.
I also don’t see yet how LLMs would learn to write code like that. No human can teach it. There is no learning material. So, there is nothing for the LLM to imitate. We would need to prompt AI to create appropriate teaching material first. Then, new LLMs can be trained on that. If nobody is actively persuing this, it will most likely not happen.
Hi Simon,
thank you for your comment, I appreciate your insights. And I agree with your premise: In order to have human-changeable software, we need higher-level presentation. I’m not sure that we will require or want human-changeable software in the future. Let me make four points why I find this a possibility:
1. This thought is not limited to current LLMs. Just as AlphaGo learned playing Go by training with itself, it might be possible to train a machine to produce working software. The output of current LLMs for the same prompt might be a fitness function for the training.
2. The idea to “blackbox” products (to make them unrepairable and even not maintainable) is common for physical goods. The “soft” in software was always the promise of malleability, but not an inherent feature. If there is any economic gain in “one-off” software, it will be pursued.
3. If we look at our current tech stack, only the top layers of software are actually preserved. Everything downwards from compiled byte code is “fully rewritten” on every software start, eventually with slightly modified third-party parts that alter the final machine code. We already have the problem of fluid execution layers while we maintain our stable readable source code layer.
4. Any machine that can directly write low-level code can probably also read it without effort. It will probably “invent” its own structures to infer the required amount of meaning from the bytes, just as we humans have invented our structures.
I hope this explains my thought a bit deeper. It is not a scenario based on current LLM-based AI and not something I expect in the near future. And it’s not something I would like to work with (as stated in my next post of the series).