King Charles warns AI leaders of existential dangers

King Charles told leaders of major artificial intelligence companies that the technology could pose “existential dangers” if it falls into the wrong hands, urging them to keep AI “firmly in the service of humanity” as debate intensifies over how to regulate fast-moving systems.

The King made the remarks at a summit he convened at Dumfries House in Ayrshire, Scotland, where executives and officials gathered to discuss how AI can benefit society and whether a shared set of principles can guide its future use. The meeting included UK AI Minister Kanishka Narayan, the Pope’s adviser, and participants from Nvidia, OpenAI, Anthropic and Google DeepMind.

“There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways,” Charles told guests at the opening of the conference.

He said developers themselves were increasingly warning about the technology’s risks. “Those who have created these technologies are now increasingly warning that AI risks developing darker capacities — perhaps even to take life,” he said.

The King added that people who value “our humanity and its vital moral component” were seeking reassurance from industry leaders “that we will not lose control of our destiny.” He said the work before them was “not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.”

The summit came amid a wave of warnings from inside the AI industry about the growing capabilities of advanced, or “frontier,” AI models. Al Jazeera reported that the gathering was intended to address safety concerns before the technology became “too late” to rein in, though the meeting was not expected to produce binding agreements.

Jensen Huang, chief executive of Nvidia, was among the senior industry figures in attendance. Nvidia supplies chips to leading AI developers including OpenAI and Meta, and the BBC reported that it is the world’s most valuable company, with a stock market value of about $5.15 trillion.

Huang told participants that safety was “paramount” and said companies should hold back a product and “keep engineering” if it was not safe enough. He also defended open models, which give users direct access to software, as a way to make sure “people and countries are not left behind.” Such systems, he said, should be protected so “researchers, universities, start-ups and countries” can build applications “none of us would have imagined alone.”

“Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world,” Huang said.

Sir Demis Hassabis, co-founder of DeepMind, struck a more cautious note. He said artificial general intelligence — systems that could be as good as or better than humans at multiple tasks — was “probably only a few short years away” and could have an impact “ten times that of the Industrial Revolution.” The chance of something going wrong was “definitely non-zero,” he said, while calling for “a sensible middle way.”

“The future is not yet written,” Hassabis told delegates.

Sarah Friar, OpenAI’s chief financial officer, said AI could help solve some of society’s “hardest problems,” but that its growing capabilities raised questions about safety and “how we ensure it always serves people.” She added: “No one company or government can do that alone and I’m grateful to His Majesty for convening this discussion at such an important moment.”

The broader debate has drawn political and religious leaders. Al Jazeera reported that Pope Leo XIV said in May that “the world does not need an artificial intelligence that diminishes humanity” and must be “disarmed.” US President Donald Trump has dismissed AI safety concerns as a “hoax,” while the BBC reported that critics including Senator Bernie Sanders and Steve Bannon have called for stronger government oversight.