https://github.com/terryyin/lizard has been useful to track when functions get too convoluted or long, or when there's too much duplication -- in code generated by agents. Still have to see how well it works long term but it's caught things here and there, I have it in the build steps in my scripts so the agent sees its output.
here’s a basic impl that i trained on tinystories to decent effect: https://gist.github.com/nikki93/f7eae83095f30374d7a3006fd5af... (i used claude code a lot to help with the above bc a new field for me) (i did this with C and mlx before but ultimately gave into the python lol)
but overall it boils down to:
- tokenize the text
- embed tokens (map each to a vector) with a simple NN
- apply positional info so each token also encodes where it is
- do the attention. this bit is key and also very interesting to me. there are three neural networks: Q, K, V – that are applied to each token. you then generate a new sequence of embeddings where each position has the Vs of all tokens added up weighted by the Q of that position dot’d with the K of the other position. the new embeddings are /added/ to the previous layer (adding like this is called ‘residual’)
- also do another NN pass without attention, again adding the output (residual)
there’s actually multiple ‘heads’ each with a different Q, K, V – their outputs are added together before that second NN pass
there’s some normalization at each stage to keep the numbers reasonable and from blowing up
you repeat the attention + forward blocks many times, then the last embedding in the final layer output is what you can sample based on
i was surprised by how quickly this just starts to generate coherent grammar etc. having the training loop also do a generation step to show example output at each stage of training was helpful to see how the output qualitatively improves over time, and it’s kind of cool to “watch” it learn.
this doesn’t cover MoE, sparse vs dense attention and also the whole thing about RL on top of these (whether for human feedback or for doing “search with backtracking and sparse reward”) – i haven’t coded those up yet just kinda read about them…
now the thing is – this is a setup for it to learn some processes spread among the weights that do what it does – but what those processes are seems still very unknown. “mechanistic interpretability” is the space that’s meant to work on that, been looking into it lately.
I think a 'linear types' ish model where the compiler flags an error if you didn't write the explicit unlock call, and only compiles if some acceptable unlock call (or set of calls) is added, would be a good design. It can / would also prevent use-after-consume. I do want the static checks, but I don't think that means that implicit function calls need to be generated on a } or a = (assigning to a variable causes a drop) etc. This is what Rust already does with `.clone()` -- needing the explicit call -- and I think it makes sense for a lot of cases of implicit drop. I have seen discussions about implementing this for Rust and I have it implemented in a C borrow checker experiment I'm trying. ATS is also an example of this, and going further is something like Frama-C or seL4 and so on.
The main point being: the implicitness is not necessary for "compiler makes sure you don't forget". So the original comment about how usage of the explicitly named and paired APIs can clarify intent both for the writer and reader can still stand while not implying that forgetting is involved. I see this dichotomy often being drawn and I think it's important to consider the language design space more granularly. I think a reminder is a better fix for forgetting, rather than assuming what you wanted and doing it for you.
(the explicit calls also let you / work better with "go to defintion" on the editor to see their code, learn what to look up in a manual, step in with a debugger, see reasonable function names in stack traces and profiles, pass / require more arguments to the drop call, let the drop call have return / error values you can do something about (consider `fclose`), let the drop call be async, ...)
This would be nice also to have an API that's usable from C/C++ code running in Wasm. I see libraries often do this where they expose a C/C++ library like Postgres to Wasm and the main / documented API is JS and you have to dig a bit to find the C/C++ API if it's possible to access it that way.
You do have to tag struct fields with a macro, but you can attach contexpr-visitable attributes. There's also a static limit to how many reflectable fields you can have, all reflectable fields need to be at the front of the struct, and the struct needs to be an aggregate.
that forEachProp function... it brings back nightmares of when, before variadics, we used to macro generate up-to N-arity functions (with all the const/non-const permutations(.
Now I use the same trick in our code base to generically hash aggregates, but I limit it to 4 fields for sanity.
Nice! Ebiten is a super nice API for Go. Lots there to be inspired by in API design. Another API I like a lot is Love for Lua (which also actually can be used from C++).
Re: the comments on here about the GC etc. -- I've posted about this a couple times before but I've been using a custom Go (subset / some changes) -> C++ compiler for hobby gamedev, which helps with perf, gives access to C/C++ APIs (I've been using Raylib and physics engines etc.) and also especially has good perf in WebAssembly. Another nice thing is you can add in some reflection / metaprogramming stuff for eg. serializing structs or inspector UI for game entity properties. I was briefly experimenting with generating GLSL from Go code too so you can write shaders in Go and pass data to them with shared structs etc.
The compiler: https://github.com/nikki93/gx (it uses the Go parser and semantic analysis phases from the standard library so all it has to do is output C++) (it's a subset / different from regular Go -- no coroutines, no GC; just supporting what I've needed in practice -- typechecks as regular Go so editor autocomplete works etc.)
Very old video of a longer term game project I'm using it on which also shows state-preserving reload on rebuilds and also the editor inspector UI: https://youtu.be/8He97Sl9iy0?si=IJaO0wegyu-nzDRm (you can see it reload preserving state across null pointer crashes too...)
You do need some JS code that asks the browser to run the wasm blob. You can't eg. just have a script tag that refers to a wasm blob yet.
libc does help with things like having an allocator or string operations etc., or for using C libraries that use libc. And that's where emscripten becomes helpful.
Browser functionality like the console or making html elements is exposed through JS interfaces, and the wasm needs
to call out to those. But they may be directly exposed to wasm later (or they may already be at this point in new / experimental browser features).
The hello world in this guide doesn't actually use console.log at all. It adds 2 numbers and sets the page content to the result. All it does is expose an add function from rust and call it from the javascript side.
