Right now training is insanely expensive because Nvidia, but I don't think that's sustainable given the demand. Eventually, training hardware may be priced like commodity CPU instances, and won't require a bajillion infiniband-linked nodes for respectable throughput.
But for now... I think you have a point. We would have seen more than Falcon, MPT, Llama, and the open Llama reproductions by now if open source foundational model training was viable.
The training will always be extremely expensive and beyond the grasp of commodity computing at a low'ish cost.
The reason for that, is that the push will always be one direction in terms of size and improvement. That isn't going to stop. It will continue to push the edge of resources.
It's not Nvidia that's holding back the premise. AMD and Intel also can't do anything for you beyond what Nvidia can.
One might proclaim: yeah but in five years you'll be able to train CodeLlama 34B on a modest commodity desktop. Nobody will want to do that at that point, they'll want access to CodeLlama 204B.
The money will be in hosting these as a service business, and building further ecosystems around that. Not much different than the way a lot of major open source oriented companies have made their money, despite their core product being largely free to use.
I think you're correct in the "right now", but I think things fall off dramatically in the next 5-10 years. NVIDIA's progress from 2018 to 2023 was insanely impressive. AMD's was too in fact, at least technologically. Intel is third, but at least still in the game.
Where your premise here falls apart, is the age-old adage, "Don't let perfect be the enemy of the good."
We probably don't need CodeLlama 204B to be highly effective in our jobs as developers. Just like physicians probably don't need DocLlama 566B to be more effective in their jobs. We only need good enough. Good enough to be useful and insightful; not the All-Seeing Oracle that has insight beyond mortal man.
I think that revolution is probably coming sooner than we think. I'd argue that in 5-10 years, we'll have hardware powerful enough that a motivated and moderately well-heeled consumer can train their own LLM with domain-specific knowledge and have a product as useful as Llama8 1.3T or ChatGPT-14. There'll always be a market - I think anyway - for incredibly powerful general-purpose LLMs that are subscription-based (whatever that looks like in terms of cost / month), but locally trained, locally run LLMs designed to do one thing and do it well? I think that's the real money and the real future.
* Pricing stability -- your business costs cannot be at the whim/survival of some particular company. Competition (including local options) is the solution.
* Model stability –– model output must be reproducible. If it's shifting unreliably, that's a showstopper.
* Alignment –– The model I use must be aligned to my business, not to Some AI Business. I don't want a model that was kneecapped in some secret, arbitrary way that a group of largely white, male, West Coast techbros decided was crucial for my business.
* Privacy –– Sensitive company data NEVER leaves the building. It sure as hell doesn't get expedited to a company with strong incentives to hoover up and use that data.
There will soon be 3 big competitors racing each other in the GPU space. Specialized AI hardware will grow in volume and power. Algorithms will be optimized. Perhaps new algorithms will appear for distributed training of open source models by millions of consumers cards (reminiscent of the Folding@home project). Hardware 5 years from now will blow away current tech.
Intelligence is a widely applicable asset with strong incentives to pop up everywhere. It won't sit gated behind a handful of companies. That's true regardless of substrate -- proteins or silicon.
I agree with your assessment on GPU sustainability.
Simply put, AMD is going to have to get their shit together. NVIDIA was embraced by the AI community so long ago because CUDA isn't that tough to use. I haven't looked at AMD's solutions at all - in fact I don't even know what they call their version of CUDA - but I know one thing for certain is that the world cannot depend on a single company to supply the underlying technology for this technological revolution.
That's an easy way to a dystopian hellscape.
And the only reason we haven't seen open source foundational model training is, because as you said, the only significant player in the arena right now is NVIDIA.
Here's hoping Lisa Su is shifting priorities - and quickly.
Absolutely. I forgot to mention - at around the same time I bought open earbuds (Samsung buds Live, to be concrete) which I found comfortable - they don't block external sound and I practically don't feel them in the ears.
My arguments against crypto are never about 'not useful for me'.
You either never heard arguments or ignore them but there are not just theoretical but also real life shows us that it doesn't work.
Your poor person who needs to save their hard cash in a dictatorship can either buy things, get dollars or euros (like what they do in Iran) or try to find someone to take their risky local currency to exchange it for a better digital one.
One were this poor person needs to understand backup strategy, fraud, computer and smartphones.
This is so naive it's ridiculous.
And continue to ignore EVERYTHING bad which happens to crypto the last 3 years.
Of course Sam and others those are NEVER crypto issues right?
Btw the poor people would benefit more from less CO2 instead of some crypto shit.
Those models are all trained by companies and open source primarily fine-tune them.
It's impressive that we can do that but the base is still a lot of money and experts from the best companies/ai experts we have.
There is a minimal chance that we will be able to somehow keep up if Google and co stop publishing or delaying publishing their papers and models.
There are communities forming etc. And it's impressive but the financial 'viability' is key to the current ai progress