>
For Gemini Ultra, we’re currently completing extensive trust and safety checks, including red-teaming by trusted external parties, and further refining the model using fine-tuning and reinforcement learning from human feedback (RLHF) before making it broadly available.
> As part of this process, we’ll make Gemini Ultra available to select customers, developers, partners and safety and responsibility experts for early experimentation and feedback before rolling it out to developers and enterprise customers early next year.
Finally, some competition for GPT4 API!!! This is such good news.
It won't be available to regular devs until Q2 next year probably (January for selected partners). So they are roughly a year behind OpenAI - and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4
> and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4
You are assuming GPT4 didn't do the exact same!
Seriously, it's been like this for a while, with LLMs any benchmark other than human feedback is useless. I guess we'll see how Gemini performs when it's released next year and we get independent groups comparing them.
Everything they published thus far in the generative AI space has been abysmal in quality compared to the competition. I'd be hella surprised if this reaches GPT-4 levels of quality...
I'm a GPT4 subscriber and a Google GSuite work subscriber. I've been using the latest Bard this morning to write and refine python code, and it's just as good if not slightly better than GPT4. I asked it to refine some obtuse code with lots of chaining, and it did an admirable job writing accurate comments and explaining the chained logic. It's ridiculously anecdotal of course, but I used Bard for all of 5 minutes last time they announced. This time seems different.
It's very active today: 50+ trades in the last hour. When I checked it was 69%, but it's gone up and down since then. Click on the "trades" tab to see.
I definitely think GPT is better than Bard, but Bard definitely did live up to the hype in a few ways. The two that blew my mind (and still do to some extent) are the blazing speed and the ability to pull information real time (no more pesky knowledge cutoff date). Bard also felt pretty comparable to 3.5 to me, better in some things and worse in others. Coding was definitely a bust with Bard.
Bard isn't a model, it's a product. Saying comparisons against "Bard" without specifying a particular point in time are like analyses of "ChatGPT" without specifying a model. There have been a number of releases adding more features, tool use, making it smarter, and crucially adding more languages. ChatGPT is not fine-tuned in different languages – it manages them but lacks cultural context. That's one place Bard is quite far ahead from what I've seen.
Most people don't use LLMs. Of those that do most people just know they're using "ChatGPT". A slim minority care about the model.
In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning.
> In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning.
Does google agree? doesn't the fact that they're so deliberately creating user-focused branding for different models (ultra, pro, nano) show they also see the value in the differentiation?
I can't speak for Google, and must emphasise that these are personal opinions. However I'd say that this entire marketing push is mostly for the super-engaged early adopters, not targeted at the general public. Looking at the YouTube videos, the more they seem to be targeted towards a general audience the less they mention these specifics. So, I suspect that the Ultra/Pro/Nano branding will mostly be used on the advanced Bard product that they speak about in the launch blog post, and on the APIs available to developers.
When I was reading the benchmarks and seeing how Gemini Ultra was outperforming GPT-4 I thought, "Finally, some competition for GPT4"!
But when I got to that part, that's when I realized that it could potentially be caught in release hell and not actually see the light of day or significant use. Google, for better or worse, has more of a brand reputation to maintain and is more risk averse, so even if Gemini Ultra can, in theory, outperform GPT4, users might not get a chance to access it for a while.
Absolutely I do. Internally they have some incredible stuff, but the leadership is terrified of letting normies try it out because of the (real or perceived I don't know) damage to the brand that would happen if it said something racist or misogynist, etc.
No way, that's what they want you to think. The idea that Google would be behind technologically would be an embarrassment they can't handle. The 3.5 level gemini pro is probably just as capable of saying racist or misogynist stuff so there's no reason why they're allowing that to be public while the "GPT-4 beating" Ultra is hidden if it's just because of that. More likely Ultra is just not as good as these benchmarks indicate and they still need some time to improve it.
Google can hardly put a picture of a white male on their website. They're so deep in the swamp of virtue signalling it's a miracle they haven't yet drowned.
Well not exactly. Not coming out until later when presumably GPT4 will have grown as much as well. So far each time, Google has failed to catch up to OpenAI. Hopefully they do however eventually.
I’m afraid it won’t be nearly as good as GPT4, because of how lax Open AI can be with intellectual property. Google will not be able to train their model on Libgen or Opensubtitles, because they can’t afford the risk.
won't be available for regular devs until probably Q2 next year, OpenAI will have probably released GPT5 or whatever new model by then. And GPT4 was done training in 2022, the fact Google is thumping their chest about being 2 years behind a much smaller company is kind of pathetic
> As part of this process, we’ll make Gemini Ultra available to select customers, developers, partners and safety and responsibility experts for early experimentation and feedback before rolling it out to developers and enterprise customers early next year.
Finally, some competition for GPT4 API!!! This is such good news.