Picture this: You wake up, grab your coffee and ask your AI assistant to handle a few things. Not “fix the typos in this email,” but “solve this problem nobody has cracked yet.” By the time your toast pops, it’s done.
Sounds like science fiction? Maybe not for much longer.
A world where machines take the lead
Imagine intelligent machines that easily outperform humans across a wide range of tasks. They break down complex concepts and math equations in seconds and produce creative work and ideas that once seemed impossible to separate from the human mind.
In this world, your model isn’t just a chatbot polishing your emails or checking off tasks like a digital assistant. It’s mapping out problems, making decisions and wrestling with deep intellectual paradoxes. And it’s doing all of it on its own, with no human input required.
That’s one way superintelligence, also described as artificial general intelligence, could take shape if researchers succeed in scaling large language models to that threshold. Large language models are the technology behind many of today’s popular chatbots, so the leap from helpful sidekick to independent thinker is exactly what has people talking.
The big debate among researchers
Not everyone is convinced we’ll get there. Researchers are still debating whether large language models can ever reach superintelligence at all.
For some experts, though, whether it can happen isn’t the most pressing question. Their bigger concern is what happens if it does.
Who holds the keys?
Daniel Kokotajlo is one of those voices. A former Open researcher, he now serves as executive director of the Futures Project. In an interview with Epoch TV that aired Sept. 24, he raised a concern that sounds like it came straight out of a political thriller.
Kokotajlo said that whoever controls an army of superintelligent systems could effectively end up controlling the country, one way or another. In his framing, the danger isn’t just about machines. It’s about power and who gets to hold it.
Think about that for a second. People already worry about who controls their data, their social media feeds and, yes, the neighbor who keeps stealing their Wi-Fi. Now imagine the stakes when the thing being controlled can outthink everyone in the room.
The AI race toward self-improvement
Here’s where it gets even more intense. Kokotajlo warns that AI companies are racing to build models that can design, code and improve themselves on their own. The technical term for this is recursive self-improvement.
In plain English, it’s software that upgrades itself, then uses the upgraded version to upgrade itself again, and again, and again. Picture a student who not only aces the exam but rewrites the textbook overnight, then rewrites it again before breakfast. Each round could make the system more capable and harder for humans to keep up with.
Kokotajlo’s fear is that if this process pushes models all the way to superintelligence, researchers could lose control of the technology entirely.
Why it matters now
So should you panic and unplug your smart speaker? Not quite. Superintelligence is still a possibility rather than a reality, and researchers continue to disagree about whether today’s technology can get there.
But the conversation is clearly shifting. It’s no longer just about what these systems can do. It’s also about who steers them and whether anyone can keep a firm hand on the wheel. As companies push ahead, warnings like Kokotajlo’s serve as a reminder that the biggest question may not be how smart machines can get, but who’s in charge when they do.
