No single neural network currently has the capability to destroy the world, and it will not occur until the architecture for NNs is altered to become self-driving rather than reactive. Even then, it will not be sentient. The deception occurs when the attention shifts from the people operating new powerful technology (tools) to this technology itself.
Meanwhile, every AI company ever: "We are very excited about agents. We are working hard on agents. We want to roll out agents as soon as possible. Also, persistent memory and online learning would be nice to get."
Like, are you following the things that OpenAI and Deepmind are saying at all? The things that make current LLMs not a threat, they aim to tear down as soon as they can arrange.
OpenAI just released a video network, and one of their core touted benefits was that you could use it as an action controller!
And, um. Do you really think, when the AIs can take a simple prompt and turn it into a ten thousand step plan that requires dynamic skill acquisition, resourcing and persistence, that generating the prompt will be the one single task that stumps them? When we are at that point - and to be clear, every leading AI organisation is sprinting to reach that point earliest - then the difference between doom and safety will be one sentence: "When you are done with that, generate a new prompt." This is not how a world with a long expected lifespan looks.
edit: To be clear, I'm still not accusing OpenAI of making up the doom stuff. Even though when I phrase it like that it sounds like they're directly working on things that obviously end the world, which seems contradictory, I don't think they see it like that. To be honest, I can't explain why any doomer works at OpenAI, except in the way that people sometimes move towards explosions and gunfire. I think it's just a bug in the human brain. We want to have the danger in sight.
"Self-driving" is just "reactive" with an open recursive structure. In principle, a network that processes a prompt, generates a plan, recurses a finite number of times, judges how well it did, generates a training plan to improve, outputs a corresponding follow-up prompt, and then waits for you to press a button before it repeats the whole thing with the follow-up prompt, ad infinitum, is still "reactive" - but nobody would argue that whoever presses the button is performing irreplaceable cognitive labor.
So I just don't think this captures an important distinction at the limit. If a system can generate a good action plan, turning it into an agent is just plumbing.
Actually, we can't be certain that humans themselves are not reactive. It is just that their reactions are either built in (self-preservation, reproduction), or based on much broader input (sensory, biochemistry, etc.). The current level of reactivity of the LLMs is very limited by their architectures, though, and as long as these architectures stay in place, you can't expect them to be "self-driven".
I just don't think this is the case. I suspect reactivity in LLMs is mostly limited by training. Human text data is just not suited to the way an AI needs to output data to plan long action chains - justification, not reasoning.
