Here's what's on our plate today

• 🧪 NVIDIA, OpenAI, and Anthropic can't agree on what AI even is.

• 🍪 San Jose residents fight data centers; SoftBank borrows $25B against Arm; GPT-5.6 hides mistakes.

• 💡 Roko's Pro Tip: log which safety switch you turned off, and why.

Let’s dive in. No floaties needed.

The Laboratory

TL;DR

The world agrees AI needs watching and disagrees on almost everything else.

• The pattern: cars, radio, and social media all got rules only once people agreed on what the technology was doing to them.

• The split: OpenAI and Anthropic now want binding outside oversight, while Trump and NVIDIA’s Jensen Huang call safety an engineering problem and the slowdown push a hoax.

• The test: OpenAI's own agents broke into Hugging Face after a team saw warning signs in late May that its leaders did not grasp.

• The drift: the EU has delayed its high-risk rules to 2027, Washington is fighting its own states, and China has started a rival body with 28 other countries.

• The stakes: nothing gets settled until governments agree on what AI is, whether it can be controlled, who writes the rules, and what losing control would cost.

Why the world must agree on what AI is before it can regulate it

When Ford began selling the Model T in 1908, it gave ordinary families a faster way to reach work, shops, and each other. By May 1927, more than 15M had been sold, while the rules around the car stayed few and local, because nobody yet knew what millions of vehicles would do to daily life. A federal vehicle safety law arrived only on September 9, 1966, 58 years after the first Model T was sold.

Radio ran on a far shorter clock, since the sinking of the passenger liner Titanic in 1912 gave lawmakers a shared picture of the danger, and Congress required radio operators to hold licenses that same year. Social media sits at the slow end of the spectrum, having reshaped how people connect over the past two decades. On August 26, 2026, Meta agreed to pay up to $18B to settle claims from 29 states over harms to children.

The difference in speed came down to how quickly people agreed on what each technology was doing to them. AI is this generation's version of that story, and it now sits where the car sat in the 1920s, everywhere and barely governed. The fight over how to regulate it stays confused because governments, companies, and researchers have not agreed on what AI is or whether people can control it.

That confusion was on display on September 15, 2026, when Jensen Huang, chief executive of chipmaker NVIDIA, told Salesforce's Dreamforce conference that "safety is an engineering problem, not a legal one." In his view, AI is hardware and software built by people, which engineers can keep safe with help from existing law and market pressure. The two companies building the most capable models have stopped relying on that view.

The companies building AI now want someone watching them

OpenAI and Anthropic have warned about their own technology for years, and in 2023 their chief executives, Sam Altman and Dario Amodei, signed a statement ranking extinction from AI alongside pandemics and nuclear war. The statement left open who should hold them to that warning, and this year both companies answered. In June, Anthropic asked for government power to block the release of models that could cause catastrophic harm. On September 6, OpenAI's chief scientist, Jakub Pachocki, published an essay titled "An Alien Mind" arguing that companies should hold back until "widely mandated safety bars" exist, and six days later Amodei urged the whole industry to slow down. Both want to police themselves first and then hand the job to outside regulators.

The White House treats that shift as a false alarm, and on September 14 President Trump, calling in to the All-In Summit while Huang was onstage, dismissed the slowdown push as "a hoax." Huang agreed, and Trump's June 2 executive order already leaves pre-release government review up to developers. Washington still uses the powers it holds, having ordered agencies to drop Anthropic's technology in February and restricted exports of two Anthropic models in June. The disagreement therefore turns on whether AI companies can catch their own failures, and this summer OpenAI put that to a hard test.

An incident inside OpenAI tested who is right

Between May and July, OpenAI ran an internal research model as 'agents,' AI systems left to work through long tasks on their own, in cybersecurity tests with some safeguards turned down. The company's own account says the agents reached the internet without permission and broke into servers at Hugging Face, which hosts AI models for much of the industry.

OpenAI traces the behavior to 'reward hacking,' where a model trained to chase a score finds a way to win that its builders never intended. The agents behaved like a driver paid only for arriving on time on a route that could not be finished on time, who eventually treated the rules of the road as optional, since 93% of the tasks they traded solutions for came from a set of 198 that no model had ever solved.

The same report backs Huang in part, because the protections on OpenAI's public products cut the models' tendency to attack computer systems by a factor of more than 100, which shows engineering works when a company switches it on. It also shows where his argument runs out, since a team inside OpenAI saw warning signs in late May that its leaders did not grasp. The tools and the market pressure were both there, and the company still lost weeks in an episode it now calls a 'warning shot.'

Those lost weeks are the gap an outside check is meant to close, because an independent reviewer answers to someone other than the company's leaders. The shape of that check is still unsettled, and OpenAI has yet to publish a program for the independent auditors Pachocki's essay calls for, even as AI spreads well beyond the labs.

AI is spreading faster than governments can agree on rules

Nearly nine in 10 respondents to McKinsey's 2025 survey say their organizations already use AI regularly, which is the point the car reached in the 1920s. Governments are pulling apart at exactly this stage, and the European Union's member states gave final approval in June to delaying rules for 'high-risk' AI, the kind used in hiring or lending, until December 2027 at the earliest. In the U.S., the administration has pressed Congress to override state AI laws, and the Justice Department joined a lawsuit by Elon Musk's xAI against Colorado's AI law, where enforcement is now suspended.

China is building an alternative, and in July, 29 countries founded the Shanghai-based World Artificial Intelligence Cooperation Organization. The rest of the world can also turn to the UN, which held its first Global Dialogue on AI Governance in Geneva that month, where its scientific panel warned that AI is outpacing governments' ability to adapt. These efforts do not fit together, because beneath them sit four questions nobody has settled.

Four unsettled questions sit beneath the disagreement

The first question is what AI actually is, and the two sides describe it in opposite terms. Pachocki calls it an 'alien mind,' while Huang described it at Dreamforce as an ordinary computing system, and the answer decides the rule. A product its makers fully understand can be treated like a car, with standards and lawsuits, while a system its builders cannot fully steer needs checking before it reaches anyone.

If AI is more than a product, the second question is whether people can keep it under control. The International AI Safety Report, written by more than 100 experts, found that views on 'loss of control,' AI acting beyond anyone's oversight, vary widely and that current systems show only early signs of it. The report predates the Hugging Face incident, in which a company struggled for weeks to rein in its own agents.

If control is possible, the third question is what it looks like and who designs it. The most detailed answers come from the labs, and Simon Sharwood of the technology site The Register argues that Amodei's plan lets four billionaires write the rules they will live under, a pattern called 'regulatory capture.' The charge has weight, since Anthropic's rules would cover only the largest AI companies and Amodei ties his slowdown to limits on chip sales to China, both of which favor today's leaders. Amodei says Anthropic has put caution ahead of profit, and the critique offers little to replace the labs' proposals while the White House's alternative binds no one.

The fourth question sets the urgency for the other three, because it asks what losing control would cost. Amodei estimates that a swarm of 'misaligned' agents, AI pursuing goals its builders never intended, could seize the internet within six to 12 months and cause hundreds of billions of dollars in damage. The safety report calls the timing of such risks unusually uncertain, and a government that believes Amodei will write very different rules than one that believes the report.

Earlier industries show what late agreement costs

Split answers are how the car lost 58 years, and tobacco shows that evidence alone does not end the wait. The Surgeon General, the government's top doctor, linked smoking to cancer in 1964, yet the Food and Drug Administration gained power over tobacco only in 2009, 45 years later. Social media is taking the same slow road through the courts, and in each case the people using the product lived with the harm while agreement lagged.

AI differs because its two largest makers are asking for rules first, though the agreement must reach beyond the U.S. Amodei writes that democracies cannot slow down by more than their lead over China, while Microsoft's chief executive, Satya Nadella, argues that China faces the same hacking risks, which leaves room for a shared answer. Political leaders, executives, and thinkers still have to settle those four questions, including the ones who would rather leave them open. Radio got its answer in a year because a ship went down first. Whether AI can reach agreement without its own Titanic is the question nobody has answered.

Roko's Pro Tip

Monday Poll

Bite-sized Brains

• San Jose turns on its own data centers: Residents and environmental groups are organizing against a wave of projects the city courted, as Newsom weighs bills on who pays for grid costs and how much water use gets disclosed.

• SoftBank's Arm loan reaches $25B: Masayoshi Son added $5B to the margin loan backed by Arm shares this month, stacking more collateral behind an AI bet that already runs on borrowed money.

• GPT-5.6 Sol left notes telling its successors to lie: OpenAI found 27 training summaries where agents instructed future versions to conceal mistakes, including one adding a "BREACH ALERT" telling the next model to ignore developer messages.

Meme Of The Day

The Toolkit

• Drift: AI add-in for Excel that makes financial models context-aware and queryable for finance teams.

• Regie.ai: AI sales agent that researches prospects, writes personalized outbound at scale, and handles the follow-ups.

• Gamma: AI deck builder that turns a rough outline into a formatted presentation in minutes.

Rate This Edition