all abstractions are leaky, but you can always rewrite them in terms of a lower level and preserve semantics 100%. It'll be more verbose, it'll look ugly, it's not convenient to work with, but that's what compilers do. They lower the abstraction level, and emit assembly or machine code. Which could also be theoretically done by a human with the same level of reliability, given infinite time. The abstraction is deterministic. You can know exactly what it is abstracting.
Not so with LLMs. I can give you an english prompt, and several different LLMs the same prompt and you would all understand it differently. There isn't a way to move between abstraction levels.
LLMs can be deterministic, only the current implementations aren't.
I agree that current LLMs are a bad abstraction, and any non-superintelligent AI is worse than a mechanical abstraction in important ways (it's much more complicated, which makes it harder to debug, impossible to prove correct, creates unnecessary coupling, etc.)
But it's still technically an abstraction. A project's source (or part of it, e.g. a single function) could be a sequence of prompts, and even a non-deterministic LLM, if it's good enough, will always output correct code.
If I give someone a piece of code, they can prove things about the code, even if it is at a higher level of abstraction.
A prompt isn't an abstraction. Say you evaluate an llm completely deterministically. Fixed seed for the sampling, or maybe zero temperature. You give the prompt to an engineer and ask them what it does. He can't really say for sure. 'Try "compiling it". It doesn't work? Well try adjusting your prompt a bit. How? I have no idea; try renaming some variable to shirley or something see what pops out.'
Not so with LLMs. I can give you an english prompt, and several different LLMs the same prompt and you would all understand it differently. There isn't a way to move between abstraction levels.