What if I told you that one can model bidirectional attention just by recurring over causal attention, and it’s still fast enough? Hint: It’s called chain of thought.
I strongly believe it’s time to discontinue diffusion models, solely on the fact that iterated auto-regression is faster, more parallelizable, and just as potent with proper prompting techniques (of course, unless you consider CoT as a form of diffusion, which it essentially is).
I'd respectfully suggest that it's perhaps not time to "discontinue diffusion models". Minsky and Papert set AI back by decades by suggesting neural networks were a dead end which couldn't learn XOR. There's not a chance of an HN comment having the same effect of course but my point is that it's easy to dismiss things prematurely.
Chain of thought is not a form of diffusion. Diffusion models clearly have characteristics that are useful and worthy of further research and should not be “discontinued“
If you follow their paper, their "inductive" argument can also prove that the result is optimal (the sqrt(2) constant is taken from thin air and may also be replaced by 1 without changing the proof).
My guess is that they tested some graphs, and "empirically" saw a bound of sqrt(2), and then came up with a proof to self-fulfill the profecy. They even mention that "for very small graphs, the approximation ratio may briefly exceed sqrt(2)", which is just nuts, given that they have a formal proof that it should never happen.
In case anyone read the paper and can't figure out the mistake in the proof, they assume that OPT_T + OPT_R = OPT, which is obviously false.
They built a dumber clone of Stockfish, and they call it ‘zero’ for some reason. What is the meaning behind ‘zero’ anyways? It used to refer to zero-shot, but now it seems like it’s just a marketing term.
I assume you’re referring to AlphaZero and Leela Chess Zero.
AlphaZero was the successor to AlphaGo. AZ was notable because unlike AG, it used zero human games to learn to play: it just played games against itself. Typically in “supervised” machine learning you take human data and train a model to imitate it. AZ used zero human data to learn.
Leela Chess Zero started out as an open source copy of AZ but it’s probably better than AZ now.
Zero doesn't mean zero shot learning. It was coined by Deepmind for AlphaGo Zero where they used zero human input into the training data. It was trained entirely by playing against itself.
From phrases like ‘deep learning is what makes AI possible’ to a ‘practice guide’ that references Python and Pytorch and Transformers, amongst other things, and apparently a 4-hour contest format, this can only be taken as a joke.
I think that a more fitting name for the contest should is IOAPI, as it’s probably all that a student could do in 4 hours.
Or the ‘International Olympiad of asking your parents to give you money for compute’, depending on the format, whichever you like best.
One of my favourite data structures is sorting the strings lexicographically. It allows prefix search via binary search, it is memory efficient like nothing else, and 100% of the times faster than a Trie in practice. Also, who wants to limit their APIs to searching through a dictionary only by prefix, anyways?
I find it a bit odd how this tree representation as a parents array (which is, by the way, I think the most basic representation in any CS course), got so much traction on HN. I think this goes to show how far a good presentation can drive a trivial idea. On top of that, it just casually presents suboptimal procedures for a lot of essential operations, without diving into too much details about the impact of the suboptimality. Good PR I guess…
The story is much worse than what is presented in the article, especially when talking about floating point errors that add (or rather multiply) up.
More often than not, the error is relative wrt the greatest magnitudes in the intermediary calculations. In essence, if you subtract two floating point numbers, you’re kinda screwed, because you cannot ever handle with good precision cases like A - B, where A and B are big enough numbers. Not to mention more complicated operations like trig functions.
In my opinion, one should avoid floating point as much as possible. And not only when testing for equality (all comparisons suffer from this).
Or, of course, ignore FPEs and proceed at your own risk.
This is not the issue.
Floating point numbers have problems when A and B differ greatly in magnitude, then A-B might easily be equal to A and so ((A-B)-A)LARGENUMBER can still be equal to zero, even if B was e.g. 1.
This is not a problem with floats. This is an inherent* result of the design constraints. You can not fix it and there are no alternatives without this flaw which offer any of the same benefits as floats.
Any operation between two floats is the floating point number which is closest to the result calculated as real numbers, that is the magic of floats.
Floating point numbers are a near magical type, which for many applications are not only a good choice, but the only one which makes any sense. It is near impossible to imagine modern engineering without them.
Indeed. The only way to accommodate numbers where the difference in order of magnitude exceeds the type's significand length is to not combine them, keep them separate and operate on them separately, combining them only for a sample.
One good common example is numerical integration. In any sufficient fine grained posteriori simulation, even with modest limits on position - delta velocity can be too small to preserve when adding to position, i.e when delta velocity is more than 2^52 smaller than position. Keeping a separate accumulator is the only way to handle this without arbitrarily increasing precision with software FP.
Isn’t this just a trie with only two levels? It seems to me that this technique isn’t particularly unknown, as it’s taught in most elementary CS courses.
I think it would be fair to say that it's a kind of funny trie-hash-table hybrid. What's neat though is that it manages to achieve better space bounds than either a trie or a hash table would on their own.
I didn't invent the algorithmic idea --- it's basically a simplification of a data structure that was published in ICALP 2003 [1]. As far as I know, the idea of combining hash tables and tries in this kind of way (despite being incredibly simple!) doesn't appear in either the theoretical or experimental literature prior to that.
I'd be interested in any other earlier sources that people might have.
not quite, as the 2nd level is a hash table which is very un-trie-like. it shares the idea that you don't store the implicit bits known by the address of the first level like a trie.
I strongly believe it’s time to discontinue diffusion models, solely on the fact that iterated auto-regression is faster, more parallelizable, and just as potent with proper prompting techniques (of course, unless you consider CoT as a form of diffusion, which it essentially is).