There are a wide range of “patterns” that derive from compilers that are the higher teachings behind functional programming and OO patterns. People don’t usually call them patterns. (Scheme isn’t special because it has closures, it is special because you can write functions that write functions.)
Reminds me of a thread a few days back when someone was talking about state machines as if they were obscure. At a systems programming level they aren’t obscure at all (how do you write non-blocking network code without async/await? how does regex and lexers work?). Application programmers are usually “parsing” with regex.match() or index() or split() and frequently run into brick walls when it gets complex (say parsing email headers that can span several lines) but state machines make it easy.
> Scheme isn’t special because it has closures, it is special because you can write functions that write functions.
When Scheme was developed, it was special, because it used lexical binding and provided lexical closures. That was new. Writing functions that write functions would not have been new, since LISP had that already -> including Maclisp, which was used for the first SCHEME implementation. Later Scheme got used for exploring different macro systems.
> Scheme isn’t special because it has closures, it is special because you can write functions that write functions.
I swear this informational parasite will never die, any language with eval can write functions that write functions. What makes Scheme and
other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure and it works.
> What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure and it works.
That would be true for source level interpreters, but not for compiled implementations.
It is an error in Scheme to modify a constant list. The implementation is not required to diagnose it (an exception is not required), which basically means that it's undefined behavior.
In Common Lisp, it is likewise undefined behavior.
Running Lisp code is not necessarily still made of nested lists. That may be the case if it is interpreted, but Lisp code is often compiled. Modifying the original nested lists that comprise the source code will do nothing to the object code, except if those parts are modified that serve as literals, which the compiled code may be sharing with the source code.
In short, your understanding is wrong. When Lisp people talking about modifying the running program, that just refers to redefining functions and other entities. Common Lisp has strong support for redefining classes of which there are existing instances. Existing objects can be migrated to new representations in the redefined classes.
None of that is related to homoiconicity or related topics.
>What makes Scheme and other homoiconic languages special is that the code that is currently executing can be manipulated live like any other data structure
I think I know Scheme well, and I don't catch your meaning. Most Scheme implementations are compilers that output a machine-language executable. How can this executable be "manipulated live" in a way that differs from an executable produced by any other language? What am I not seeing?
> the code that is currently executing can be manipulated live
Typically when people say this about Lisp, they mean that they can redefine functions and objects live, having all the code adapt to use the new definitions, or when their code hits an error, they can not only redefine functions and objects live, but also examine the state of the call stack and change the values of variables live, and then let the code run again. This[0] is a clear explanation that I found. There's also a video[1] that shows the same thing.
However, I don't know if those apply to Scheme. Also, I am pretty sure it is unrelated to Lisp being homoiconic, and that it's more about it being dynamically typed and interpretable.
I suppose that, if I'm asked "What is something that makes homoiconic languages special?" and the answer can't be that it makes macros easy, then I would say it's that it's easy to write an interpreter for that language in the language itself, because the data structures that form the syntactic structure of the language are themselves first-class objects. That's why the interpreter from the appendix of the LISP 1.5 manual can be so short. That's also why the EVAL function takes as a parameter a list of atoms, and not a string, as in non-Lisp dynamically typed languages; since a Lisp program is also a list of atoms, that's the form it takes in the interpreter. The first Lisp compiler [0] isn't so long, either. The equivalent eval function in non-homoiconic languages requires a lexer, parser, and special definitions of data structures to represent the syntax, as well as functions to manipulate those data structures.
Scheme was the first language with lexical scoping of variables, and after Scheme was published in 1975, most new languages have had lexical scope and most languages used now (Java, C#, Javascript, Rust) have it. I think the qualities you describe make it easier to experiment with (Lisp-like) languages and language implementations, which is why lexical scope appeared in a Lisp-like first. Also, I think the concept of a "closure" appeared first in discussions about Lisp (maybe as part of the same discussions that led to lexical scope). Ditto I think continuations and continution-passing style. Also the interpreter you refer to in Appendix A of the Lisp 1.5 manual had a subtle problem or area for improvement around functions appearing as arguments to functions, and the Lisp's community's discovery of that and solution to that seems to have been incorporated into other languages, probably first the early functional languages like Miranda, then on to Javascript, etc.
At the level where you're executing assembly every language is the same. Which is a way of saying that assembly is homoiconic. And even in C you're more than capable of W&Xing your binary to change behavior at runtime but you're doing it at a level below the language. It's the (not completely unrestricted) self-modifying bit that's cool
and unique. You can go into an existing function and change what it does by manipulating or outright replacing its code.
Other languages you end up having to evaluate and re-point/redefine the reference and it gets you where you need to go but not nearly as cool. For folks that liked Dijkstra's whole structured programming thing but didn't want to give up the power entirely.
>You can go into an existing function and change what it does by manipulating or outright replacing its code.
I still don't see what this might mean. It doesn't sound like anything I've ever done in Scheme or I've ever heard of anyone else doing. You probably don't mean editing Scheme source code and recompiling.
How about monkey patching in languages like Javascript and Python?
I'd also say "homoiconic" is a less important concept than people would think. A while back I was writing a lodash-like library for Java because I don't like the Streams API (though in the end I just used the collectors from the Stream API because they were easy to fit into my system) One thing I found frustrating about that was that I didn't want to write 8 different functions for the 8 primitive types (never mind 64 functions for the direct product of the primitive types) and conceived of a code generator that would let you write Java-in-Java with a somewhat Lispy looking API that builds up something like S-expressions with typed static imports, here is some sample code
The plan was to write a ferocity0 sufficient to write a code generator that could generate stubs for the standard library plus a persistent collection library, then write most of ferocity1 in ferocity0 and maybe write a ferocity2 in ferocity1 and ferocity2 would be the master tool for code generation in Java. Look how nice the stub generator is
Now many people would think this combines all the drawbacks of Common Lisp and Java and they might be right. On the other hand, I think if you're aggressive in organizing your code and using code generation you might cut your code volume enough to make up for the bulking that happens when you write Java-in-Java. The stubs to write Java-in-Java are a huge amount of code but they're 99% generated and once you have Java-in-Java you can keep recursing and messing around with quote and eval and do all the stuff you'd do in a homoiconic language except you are having to write code in a DSL.
In principle. Most people find somebody else's metaprogramming-heavy Common Lisp difficult to work with. When you look at classic programs from the golden age of AI you find they either rewrote their Lisp prototype in C++ for speed or the Lisp program (like an expert system shell) is much smaller than you imagine possible but it looks like the engine of a UFO.
Though I've only seen other people's Lisp projects online and in books, I haven't seen metaprogramming that made it less readable - usually, it makes the program more readable. At most, it defines a DSL, e.g. COMFY 6502, which is a thin wrapper over 6502 assembly implemented as a DSL in Emacs Lisp. But most of the time, macros provide a readable interface over a bunch of hard-to-understand boilerplate generated code.
> Most people find somebody else's metaprogramming-heavy Common Lisp difficult to work with.
Citation needed. Org names, projects, numbers.
Also, how easy do people find someone's Java metaprogramming? Where the whole metaprogramming system is locally invented, not just its application to the problem?
That code looks readable to me. It is well formatted and the functions are small, with simple control paths. No variable assignments or loops to unravel.
It defines no macros, but refers to some defined elsewhere; the instructions say you are supposed to use a certain other source file related to Chapter 14 of a book.
I could probably ramp up on this in a day, if I had reason to.
Back in the 1980s I found a book at the public library where somebody had a BASIC program that would read a BASIC program without line numbers and with structured programming loops and rewrite it an ordinary BASIC. It was oriented towards CP/M so I typed the program in and modified it to run on my TRS-80 Coco. It was kinda like
Similarly I've written awk programs that write shell scripts (guilty pleasure), Python programs that write AVR-8 assembly, toolkits to write Java ASTs with static imports like
@Test
public void buildAList() {
var stream = callBuild(callAdd(callAdd(callAdd(callBuilder(), Literal.of(7)),
Literal.of(11)),Literal.of(15)));
var expr = callToList(stream);
var l = expr.evaluateRT();
assertEquals(3,l.size());
assertEquals(7, l.get(0));
assertEquals(11, l.get(1));
assertEquals(15, l.get(2));
}
and evaluate with a tree-walking interpreter or use as a code generator in maven, do transformations on the ASTs, etc.
There are many paths to metaprogramming but the Lisp family leads in putting it on your fingertips. Once you learn those methods you can apply them in the trashiest programming languages, but I think often people miss the point.
Reminds me of a thread a few days back when someone was talking about state machines as if they were obscure. At a systems programming level they aren’t obscure at all (how do you write non-blocking network code without async/await? how does regex and lexers work?). Application programmers are usually “parsing” with regex.match() or index() or split() and frequently run into brick walls when it gets complex (say parsing email headers that can span several lines) but state machines make it easy.