I have been experimenting with WebAssembly/Wasi over the last couple of years and decided it would be time to have another look and see what the state of things is. I it would be interesting to try and embed python in a wasm component for its nice sandbox properties.

There's a super cool project https://github.com/RustPython/RustPython which is a python interpreter in rust which can compile to wasip2 so I hooked that up and it seems to work well.

Its easy to build wasi components in rust now with the wasip2 target, excited to see where things head over the next few years.

This makes me think about kolmogorov complexity. The program here looks like gibberish but produces the desired output, would there be even shorter programs that don't look like they make sense but produce the same output? How would you search for these programs?

The current world record for the shortest C program that prints 12 Days of Christmas lyrics is 431 bytes: https://code.golf/12-days-of-christmas#c

Well that post just cost me a couple of hours.

And still 136 bytes off 431:

  #define p printf 
  int main(){const char *g[]={"A Partridge in a Pear 
  Tree.\n","Two Turtle Doves, and","Three French Hens,","Four 
  Calling Birds,","Five Gold Rings,"," Geese-a-Lay"," Swans- 
  a-Swimm","t Maids-a-Milk","e Ladies Danc"," Lords-a-Leap"," 
  Pipers Pip","Twelve Drummers Drumm"};
  const char *d[{"First","Second","Third","Four","Fif","Six","Seven","Eigh","Nin","Ten","Eleven","Twelf"};for(int i=0,j;i<12;i++){p("On 
  the %s%s day of Christmas\nMy true love sent to 
  me\n",d[i],i>2?("th"):"");
  for(j=i;j>=0;j--){p("%s%s%s\n",j>4&&j<11? 
  d[j]:"",g[j],j>4?"ing,":"");}}}

You can shave 25 more by using printf as is, removing "const", removing spaces between char and *, and removing parenthesis around "th".

Are those consts really necessary? There are also a few character missing after char *d.

> Are those consts really necessary?

Technically yes, initializing a plain char * from a const char * (from array decay) is a constraint violation (an error the standard requires the compiler to detect and complain about) even in C89. Many compilers will let you off with a warning though.

which characters are missing?

"]=" after "*d["

It's not necessary in C.

   const char *d[{"First","Second",...,"Twelf"};

is not a valid C unconditionally.

You should tell my compiler.

To be honest, I understand code.golf's premise, but still, code being public (maybe optionally?) would be nice.

I agree. Stack Exchange's Code Golf has public source, but the best there is 644 bytes: https://codegolf.stackexchange.com/a/4198

Is it broken for anyone else the code prints hello world and then some numbers

The leaderboard is public but the actual code is private, that hello world is just an example c program.

> would there be even shorter programs that don't look like they make sense but produce the same output

Yes, mostly likely

> How would you search for these programs?

Brute force search is very inefficient so the real answer generally is "be clever" in the mathematical sense.

In general, KC is not computable so there is no program that can take a string and return the shortest program that computes that string.

However it's always in principle possible for someone to prove that the KC of some particular string is X.

> However it's always in principle possible for someone to prove that the KC of some particular string is X.

Only for a bounded number of strings. I.e. there is a finite set of string X such that for strings outside of X, you can not prove their KC, even in principle.

This result by Chaitin [1] can be paraphrased as: you cannot prove a 2 kilo theorem with a 1 kilo theory.

[1] https://www.jucs.org/jucs_2_5/the_limits_of_mathematics/Chai...

I think it's honestly quite hard to know, as it's really (generally speaking, AFAIPK) impossible to directly compute the KC in most cases, only really from the feasibility standpoint we can check that it's slower than some other version/value.

Which is quite exciting, it sets us up nicely for long-running competitions and the like due to the logarithmic-like growth curve (with sometimes some very fun discoveries on the ultra-tail-end of things! <3 :D)

Am currently running a mini-competition with a current prize bounty of $100 (distributed proportionally by % contribution in logspace) for an LLM that can memorize the most digits of Pi by March of next year. Pi is nice as it actually is quite compressible in theory and seeing if a model is able to learn a sufficiently-close-to-the-MDL set of weights that recovers a highly-compressed algorithm from the data would be extremely nice, indeedy!

However, whether this is feasible or not with off-the-shelf models and such is not entirely easy to know, so for now, it's just a digits-memorizing competition, and we shall see where it goes from there!!! <3 :'))))

Has anyone figured out a way to fine tune this with 24gb of vram? I have tried with deepspeed etc but no luck. Seems to be just out of reach for fine tuning requiring 26gb.

Have you tried quantization? It's often a cheap and simple way to reduce the VRAM requirements.

What hardware are you using? (CPU,RAM,GPU,VRAM)

Have you considered using llama.cpp for a mixed CPU+GPU use (if you have enough RAM)

Yeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb

I'm not sure about this model specifically, but training with 4-bit quantization has been a thing with LLaMA for a while now, although the setup involves manual hacks of various libraries.

Is it possible to offload some layers to CPU and still train in a reasonable amount of time?

There’s also that pruning tool that was on hn in the last couple weeks. It seemed to work really well on the larger models, and could reduce size by 30-50%

You probably don't want to fine-tune a quantized model. They are fine for inference but not great for training.

People should be training model sizes that fit-and-fill consumer GPUs, ie:

2x 24G - for dual GPU ~ 28B model

1x 24G ~ 14B model

etc.

That seems incredibly complicated, you probably don't need anything that complex unless you're doing research.

It’s not complicated, it’s a DIY automated system for people who don’t want to spend the money at the store for similar components. It teaches you how to build every component needed to run hydro with very little maintenance and is completely self-contained. It’s perfect.

I've been working on a uni paper review site. Only available in New Zealand for now. https://papergrade.co.nz

I was annoyed at how all the key data about games was spread out over multiple sites and not easy to see quickly. Ballreport shows the days upcoming nba games and the key data about the teams, odds and players. Its very basic at the moment but updates daily and i will be adding more functionality soon.

I think this is a cool idea and something I was interested in also, if you need any help developing it I'd pitch in!

Thanks for the feedback. I think i just need to imporove the UI and add some more stats.

It doesn't directly interact with the blockchain but it uses its own implementation which is synced between peers.