Here’s what’s on our plate today:

• 🧪 How AI's copyright defense turned into its biggest liability.

• 📰 Andreessen's AI flex backfires, the AI economy cracks, Zilis takes the stand.

• 💡 Roko's Pro Tip: model quality isn't a moat; build harder advantages.

• 🗳️ Poll: Is distillation theft or just AI's own logic coming back around?

Let’s dive in. No floaties needed…

The Laboratory

TL;DR

• Distillation flips the script: AI companies called training on copyrighted material fair use, but when rivals trained on their outputs, they cried theft.

• DeepSeek sparked the panic: A Chinese startup matched frontier performance for $5.6M, prompting OpenAI, Google, and Anthropic to accuse foreign actors of massive extraction campaigns.

• Musk said the quiet part: Under oath, he admitted xAI “partly” used OpenAI outputs to train Grok, calling it standard industry practice and collapsing the “foreign threat” narrative.

• The law is empty: AI-generated outputs likely aren’t copyrightable, proposed legislation hasn’t moved forward, and cross-border enforcement is near-impossible.

• The stakes cut both ways: Cracking down entrenches dominant players and contradicts the industry’s own legal arguments, but doing nothing undermines the billion-dollar investments behind frontier models.

How AI’s copyright defense turned into its biggest liability

For the better part of three years, the biggest AI companies in the world have defended a simple proposition: that training a machine learning model on existing work is not the same as stealing that work. OpenAI, Google, Meta, and others built their large language models by feeding them enormous amounts of text scraped from the internet, including copyrighted books, news articles, academic papers, and creative writing, mostly without permission or licensing fees.

When publishers and authors sued, OpenAI argued in court that this kind of use falls under fair use, the legal doctrine that permits limited use of copyrighted material for purposes like research, commentary, or transformation. Courts are still working through those cases, and the outcomes remain uncertain.

But while the legal system deliberates, a newer version of the same argument has emerged, this time aimed directly at AI companies themselves. At the center of it is a technique known as model distillation, one that has turned the industry’s own logic into its most uncomfortable problem.

The shortcut

Model distillation allows a smaller, cheaper model to learn from the outputs of a larger one. The smaller model never sees the original code, weights, or training data; it relies only on answers. Developers send carefully chosen prompts to a frontier model, collect the responses, and use them as training material for a new system.

The technique was first proposed in academic research in 2006 and refined by Geoffrey Hinton and colleagues at Google in a 2015 paper. For years, it was an uncontroversial method that companies used internally to build their own models faster and more cheaply.

The appeal is obvious: if a company has already spent hundreds of millions of dollars training a powerful model, distillation offers a way to build a lighter, cheaper version without repeating that enormous expense.

Outside academic studies, real-world examples have shown that the technique can be highly viable. OpenAI reportedly spent around $100M training GPT-4, reinforcing the belief that frontier AI required vast capital and computing power. That assumption was jolted in January 2025 when Chinese startup DeepSeek released its R1 model, claiming comparable performance on key benchmarks at a training cost of roughly $5.6M. It quickly overtook ChatGPT as the top free app on Apple’s App Store, while NVIDIA shares fell 17% in a single day, erasing about $600B in market value. The combination of low cost and high performance sent alarm bells ringing across Silicon Valley.

The same day, Google’s Threat Intelligence Group disclosed that Gemini had been targeted by a campaign involving more than 100k prompts designed to transfer its capabilities to an outside model.

Eleven days later, Anthropic published the most detailed public account yet, saying campaigns linked to DeepSeek, Moonshot AI, and MiniMax generated more than 16M exchanges with Claude through roughly 24k fraudulent accounts. The company said prompts were designed to extract reasoning traces, generate censorship-safe answers, and rapidly capture improvements whenever Claude was updated.

The scale of these attempts was such that in response, OpenAI, Anthropic, and Google began sharing attack data through the Frontier Model Forum, a nonprofit the three companies co-founded with Microsoft in 2023.

Then on April 23, White House OSTP director Michael Kratsios issued a formal memorandum accusing foreign entities “principally based in China” of conducting industrial-scale distillation campaigns against American AI systems and directing federal agencies to coordinate a response. The Chinese Embassy called the accusations “baseless” and said Beijing values intellectual property protections.

However, the distillation story does not end there.

When it came home

The narrative of distillation as a foreign threat collapsed on April 30, 2026, when Elon Musk testified in his federal lawsuit against OpenAI and acknowledged that xAI had “partly” used OpenAI model outputs to help train Grok.

He described the practice as something “generally all AI companies” do, undercutting the idea that distillation was only an external threat.

The admission reframes distillation from a geopolitical issue into an industry-wide one. If U.S. companies are also training on each other’s outputs, the distinction between legitimate optimization and adversarial extraction rests less on the nature of the technical practice and more on who is doing it and where they are based.

As attorneys at Winston & Strawn noted, OpenAI’s fair-use defense in copyright lawsuits complicates its complaints about distillation. If training on others’ material is transformative when AI firms do it, it becomes harder to argue that the same logic amounts to theft when competitors apply it in turn.

There is also a real difference in scale and intent between a company benchmarking its outputs against a rival’s and a coordinated campaign routing millions of queries through thousands of fake accounts. But Musk’s testimony, delivered under oath, makes it harder to frame distillation as something only adversaries do.

The legal vacuum

As with many other areas of AI policy, the legal framework for addressing distillation remains largely theoretical. The U.S. Copyright Office concluded in January 2025 that AI-generated outputs generally do not meet the standard of human contribution needed for copyright protection.

If the outputs of a model are not copyrightable, then collecting and reusing those outputs through distillation may not constitute infringement in a traditional sense. One legal analysis noted that even assuming DeepSeek used ChatGPT’s outputs for training, OpenAI holds no clearly established IP right that would have been violated.

And while the absence of clear laws has prompted Congress to act, Representatives Bill Huizenga and John Moolenaar introduced H.R. 8283, the Deterring American AI Model Theft Act. The legislation would authorize sanctions and entity-list designations against foreign actors accused of carrying out model extraction attacks, while explicitly preserving authorized training that complies with terms of service. The bill has been referred to a committee but has not advanced further.

Even with new laws, enforcement would remain difficult. Individual prompts often resemble normal usage, and only patterns across thousands or millions of requests reveal extraction campaigns. Those requests can also be routed across jurisdictions, leaving companies to rely largely on terms-of-service violations, a weak and difficult tool to enforce internationally.

At the same time, restricting distillation broadly could entrench the largest AI companies by making it harder for smaller players to compete, concentrating an extraordinary amount of power in the handful of organizations that can afford billion-dollar training runs.

For years, leading companies insisted that learning from existing material was not theft but transformation, a necessary step in creating new technology. Now they are confronting rivals using similar logic to learn from them.

If training on the work of others is a legitimate engine of innovation, then drawing the line only when incumbents become the source will be difficult to defend. If the practice is unacceptable, then the foundations of the current AI boom look less certain than its builders once claimed.

In the end, distillation brings the AI industry back to the argument that helped build it. If learning from the work of others is a legitimate engine of innovation, then drawing the line only when incumbents become the source will be difficult to defend. The AI industry built its empire by insisting that learning from others was innovation. Now that someone is doing it back, it has to decide whether it still believes that argument.

Roko Pro Tip

Monday Poll

Bite-Sized Brains

• Andreessen's AI flex backfires: Marc Andreessen is being mocked online after claiming AI now handles much of his work, prompting questions about what a venture capitalist actually does.

• The AI economy cracks: Five key architects of the AI boom are publicly flagging where the wheels are starting to come off, from infrastructure costs to revenue gaps.

• Zilis takes the stand: Shivon Zilis testified in the Musk-Altman trial, adding personal and corporate intrigue to a case already shaping AI's biggest legal showdown.

Meme Of The Day

The Toolkit

• Assembly AI: Speech-to-text API that handles transcription, speaker detection, and audio intelligence for production apps. 

• Chroma: Open-source vector database built for AI apps, fast to set up and easy to scale for RAG and embeddings. 

• Continue: Open-source AI code assistant that plugs into VS Code and JetBrains with full control over models and context.

Rate This Edition