PROBABLY PRIVATE
Issue # 28: Privacy Evals
We are officially in September, meaning I've done my appropriate European duty of forgetting August existed. 😂 I will use this as proof of my integration.
Jokes aside, I'm excited to be back and have many plans for this Fall/Winter. I'm already having a great time learning from the students in my InfoQ Privacy and Security Engineering Certification. There's another round starting in mid-October if you missed this one! But I'm also asking you all to chime in on what I should be teaching online and what's interesting to you... (read on!)
Aside from that, I'm also diving into the murky waters of privacy evaluations for AI/ML systems. Let's take a look together, shall we?
As I mentioned in the last newsletter, I had some great conversations at IWPE around how we might develop new ways of evaluating privacy in AI/ML systems.
Inspired by those chats and much research I've been reading for months, I put together a mega-post on privacy evaluations. I'll summarize some main bits here, but would love your feedback and questions on the longer post if you have interest.
Note: some people misconstrue "privacy evaluations" with privacy advice. They are not the same thing. Evaluations (in this newsletter and the post) are about evaluating the model capability for a particular task (in this case related to privacy), not "giving legal advice". It is always important to seek human legal counsel for compliance advice. :)
There are a few main approaches to privacy evaluations that I have seen via my work and network:
LLM as a Judge: Here an LLM is prompted to "evaluate the privacy" of a given piece of text and perhaps to sanitize or pseudonymize the text before using it for a task. Depending on the setup, this may also reroute the request based on the perceived sensitivity.
Privacy Benchmarks: There are several benchmark datasets focused on a variety of privacy-related tasks including PII removal, memorization detection, privacy reasoning and even some enterprise-task and agentic context-aware benchmarks.
Custom, Task-Specific Evaluations: These work best when implemented or guided by the teams actually using AI for their work. It also is a great way to better define privacy requirements and outcomes, but requires support for privacy engineering on a use case basis (or controls widespread enough to implement easily).
Static Testing: Traditional testing mechanisms that aren't ML-based work for specific privacy engineering tasks, like pseudonymizing found entities in text (names, people, places) or appropriately routing based on policy and data access decisions. Having standardized tests for privacy can also help in a coding assistant harness to validate AI-written code meets the organization's privacy patterns and standards.
Here are a few points that I think are important to discuss in the machine learning/AI and privacy engineering community:
Not many companies are doing privacy evaluations of AI vendors and models, and that means very few companies are choosing models based on privacy requirements. A trend I see is that C_O makes a decision about which AI vendor the company is going to use and then the privacy team is tasked with "figuring out how to make it fit" into the privacy principles or requirements. Sometimes without technical assistance or additional budget for governance controls.
Without data on how models (and guardrails) perform, it's difficult to understand what threats are realistic and which ones are not. This is why threat modeling and red teaming go hand-in-hand. Setting up basic privacy engineering testing for your AI vendors and models can be a useful first step into being able to reason about what controls actually are needed and how.
Many privacy evaluation datasets and benchmarks are focused just on PII removal. Although input sanitization is an important task for AI workflows, it is not the only privacy-related problem of AI systems, and it may or may not actually be the control your organization needs for sanitization (depending on your data and use cases).
As with all things in privacy, getting more specific means better evaluations! An instructive example from the CONFAIDE research is labeling what data should be used by a transcription workflow and what data should not be used due to sensitivity (or irrelevance). Making use-case specific evaluations and working with federating governance to teams can support privacy goals across the organization and lead to better conversations around handling personal data in AI systems.
Even if you don't have machine learning engineers, evaluation software and suites are likely already in the tools teams are using for AI workflows. This means it's a matter of priority and interest. As more companies move to mixed-model and mixed-provider (and some local model usage) settings, it's the perfect time to rethink evaluation and make it a common part of system and product design as well as privacy requirements and controls.
If you're new to evaluations I also highly recommend checking out Sebasitan Raschka's LLM evaluation article, Hamel Husain's FAQs on evaluations and this Vanishing Gradients podcast on failure analysis.
If you work in research, my longer post describes what evaluation suites don't tell us yet and why we need more use-case specific datasets in order to better evaluate AI workflows and models. I wish for a day where more of the AI privacy conversations are focused on better classifiers, models and guardrails for determining: is this ick or not? And for more conversations about where sensitive data should live, what should get sent and what we can run locally or distributed.
Happy to hear your feedback and thoughts and if I missed some interesting new approaches or research to evaluating privacy in AI systems. I'll be updating the post regularly, so your input improves it for everyone.
I occasionally get a reply to this newsletter, which I really love! But there's plenty of folks here who probably enjoy or read this without ever emailing me.
I'm planning some public courses for next year and also deciding what content to feature and write in the coming months.
For my public courses, like my Practical AI Privacy Course, I'm investigating if I can offer them for a lower price point with an increased class size.
If you have considered joining one of my courses, but couldn't, I'd love to learn what would have made it possible (lower price, different time, different format, different outcomes...).
Here's a few things that I have planned to dive deeper into:
I'd love to get an idea of which topics are most interesting for you, other things you'd like me to cover and what you've enjoyed thus far. You can:
Of course, no promises that I can/will prioritize everything, but it's lovely to get your thoughts.
I'll be presenting at the following events in the coming weeks/months. If you aren't attending the event but are in the city, I'd be very happy to say hi and drink an espresso or tea. Just reply to this email.
There's a few other public dates in the works so stay tuned! Hope to see you somewhere this year... or to read from you on what you'd like to hear/learn/see from me online. :)
It's a pleasure to read from you; either by a quick reply or a postcard or letter to my PO Box:
Postfach 2 12 67 10124 Berlin Germany
Until next time!
With Love and Privacy, kjam