PROBABLY PRIVATE

Differential Privacy in Today's AI: What's hard?

What data needs to be protected in the ML lifecycle?

Often companies use models already trained on publicly available data and then finetune using differential privacy, but Tramèr et al. (2024) released a position paper calling for a more nuanced approach to what data is considered public. In their paper, they enumerate several cases where publicly available data violated individual privacy.

• A phone number was memorized by an LLM based on publicly scraped data used during pretraining. DP finetuning did not fix this.
• A cryptocurrency key was memorized by Copilot and the wallet was emptied.

When applying differential privacy, you need to think through not just fine tuning but data exposure in the entire model lifecycle, from collection, training, evaluation and inference. If there is sensitive data at any of those stages, the same privacy questions apply.

This doesn't mean you can't or shouldn't fine tune with differential privacy (I give you several examples of how to do this in my book). It just means that you might still have memorization and overexposure, especially in overparametrized models.

Is your data representative enough to learn privately?

Tramèr and Boneh (2021) discovered to reach the same deep learning accuracy of a normally trained model, a differentially private model might need an "order of magnitude" more data.

How come? Well, if the model cannot memorize or learn from just one example, it must process many examples of that concept to learn it privately.

There's been significant evolution of the overlap between learning theory and differential privacy. For example, there are bounds as to what can be learned (PAC theory) when it comes to using differential privacy on certain complex distributions. There's also mathematical proof that all distributions are differentially privately learnable although they might not be efficient.

In an applied setting: If you don't have enough different and diverse data, or if you don't have a well-defined problem space and can't find data that adequately represents that problem, you probably won't be able to learn privately.

My advice: think through your problem and task deeply. Figure out what you actually really need to learn and what is superfluous. Then determine if you can actually simulate, collect or produce data that matches that requirement. This will help you not only learn privately, but also more efficiently.

Can some tasks ever be private?

Brown et al (2022) asked this question about today's language models. The authors argue that Nissenbaum's contextual integrity should apply to language data.

Nissembaum's theory says each user should have autonomy about how, where and in what context their data appears. The authors argue the only data that matches how LLMs are used today is data intended to be freely available for the general public.

Text origin and ownership is often difficult to define, which is a key decision to appropriately apply differential privacy. For example, to do appropriate privacy accounting, you define how much one person can contribute to the training. This is surprisingly difficult for text data because sometimes someone is quoting another person, or paraphrasing or referencing. Or someone may use different accounts or handles but be the same person. How can you define authorship well enough to apply differential privacy?

For those of us working on this problem: what large language models can truly be "privacy preserving"? When can you ensure the guarantees match the real-world concerns and context? Is example-level good enough for the problem at hand? Do we need to think through attribution at a higher level?

How can you work interdisciplinary to develop real use cases?

Applying differential privacy so that it actually fits the organization's understanding of privacy and risk can be a challenge. Handoffs almost always lead to confusion. This can result in missed product expectations, poor performance and also privacy and security mistakes. Interdisciplinary processes can help build knowledge, understanding and communication that helps smooth handoffs and clarify these expectations.

Ideally the AI/ML product lifecycle includes product, privacy, security, machine learning and risk stakeholders from the beginning. It could look something like this:

• Start product ideation as a multidisciplinary team
• Threat model and provide privacy engineering input based on early architecture, data and design choices (nothing built yet)
• Begin model, data and software development using identified privacy technologies and best practices
• Evaluate models based on specs from product, privacy and security
• Finalize model candidates and integrate them into stack
• Purple team models and perform privacy testing
• Tweak guardrails, controls and model choice based on attack and evaluation success
• Launch after sign-off from the multidisciplinary team
• Check-in and re-evaluate based on changing risk and model landscape and learnings from other products

I'm curious: does your model lifecycle look like this?

If your organization is ready to explore AI privacy and security, you can work with me on the harder parts of applying differential privacy to today's AI.

I've summarized these points from a longer article, in case you'd like to read more.

Links for new years learning

If you are a resolutions, goals or new practices in the new year kind of person, I thought I'd share a few things I would recommend if you're looking to learn and grow.

• Mary Wooters' Coding theory course: I took this course last year (self-guided via YouTube and the Stanford website) and learned so much! Professor Wooters is an amazing teacher and it definitely expanded my understanding of how coding theory influenced machine learning.
• David MacKay's Information theory course: The open-source book and lecture videos available are such a treat. It was also fun to see the course progression (information theory basics -> Monte-Carlo and Bayesian learning -> early deep learning). Highly recommend!
• Sebastian Raschka's Build an LLM from scratch: Last year I read Sebastian's book and dove into some of the videos and GitHub. I learned more about attention mechanisms and better understood how parts of the Transformer architecture contribute to problems like memorization.
• Introduction to Technical Privacy Course: Last year I released an O'Reilly learning course based on my book for a more general technical audience. If you haven't made it through my book, try spending 4 hours with me on video!
• Introduction to Purple Teaming AI Models: In case you missed it, my hands-on introduction to attacking and defending AI models wrapped up this week. I walk through basic to advanced attacks alongside defenses like guardrails, prompt sanitization, and LLM as a judge (or pseudonymizer).

In 2026, I'll be sharing my local-first personal AI use cases, developing materials and videos for multidisciplinary thinking in AI privacy and security and show you how to apply my book directly into your AI product lifecycle. Requests, open questions and suggestions very welcome!

FOSDEM presentation on Sovereign AI

I'm presenting in Brussels at FOSDEM this year on Sovereign AI.

Here's a few questions I'm noodling on. If you have feedback, articles and thoughts you're willing to share, send them my way!

• Why do many sovereign AI/cloud conversations automatically assume a zero sum game? (i.e. someone must win and lose)
• What do individuals actually want in terms of sovereignty?
• How can marketing and propaganda messages be cut through for real conversations around the challenges, risks and rewards of building out new types of computing and cloud infrastructure?
• Does sovereign AI represent a chance to encode human rights into AI/ML practices?
• What does "Sovereign AI" mean to you?

If you'll be at FOSDEM and want to meet up or chat, please drop me a line! It's a pleasure to read from you; either by a quick reply or a postcard or letter to my PO Box:

Postfach 2 12 67 10124 Berlin Germany

Until next time!