What Naur meant by "theory" was the mental model of the original programmers who understood why they wrote it that way. He argued the real program was is theory, not the code. The translation of the theory into code is lossy: you can't reconstruct the former from the latter. Naur said that this explains why software teams don't do as well when they lose access to the original programmers, because they were the only ones with the theory.
If we take "a great description" to mean a writeup of the thinking behind the program, i.e. the theory, then your comment is in keeping with Naur: you can go one way (theory to code) but not the other (code to theory).
The big question is whether/how LLMs might change this equation.
Even bringing down the "theory" to paper in prosa will be lossy.
And natural languages are open to interpretation and a lot of context will remain unmentioned. While programming languages, together with their tested environment, contain the whole context.
Instrumenting LLMs will also mean, doing a lot of prompt engineering, which on one hand might make the instructions clearer (for the human reader as well), but on the other will likely not transfer as much theory behind why each decision was made. Instead, it will likely focus on copy&pasta guides, that don't require much understanding on why something is done.
But if the person writing the prompt is expressing their mental model at a higher level, and the code can be generated from that, the resulting artifact is, by Naur's theory, a more accurate representation of the actual program. That would be a big deal.
That theory, or mental model, is a lot like a program, but of a higher kind. A mental model answers the question: what if I do this or that? It can answer this question with a different level of detail, unlike the program that must be executed completely. The language of a mental model is also different: it talks in terms of constraints and invariants, while the program is a step-by-step guide.
Anthropic has been doing this from the start and they are justified in it (the plan has different pricing rates than API). People have been making workarounds and they are justified in that as well - those people understand their workarounds are fragile when they made them.
This seems a meaningless project as the system prompt of these models are changing often. I suppose you could then track it over time to view bias... Even then, what would your takeaways be?
Even then, this isn't even a good use case for an LLM... though admittedly many people use them in this way unknowingly.
edit: I suppose it's useful in that it's a similar to an "data inference attack" which tries to identify some characteristic present in the training data.
I think you mentioned it, when a large number of people outsource their thinking, relationship or personal issues and beliefs to chatgpt, it important that we are aware and don't because of how easy it is to get the LLMs to change their answers based on how leading your questions are due to their sycophancy. HN crowd mostly knows this but general public maybe not
We want to gamify prompt hacking and give people an UI to add/remove chunks of the system prompt. It'll be unlocked by collecting widgets around the place.
reply