[ Web Proxy ]
URL:
Viewing: https://brave.com/research/kd/ [Back]  [Original]
Membership and Memorization in LLM Knowledge Distillation | Brave

Yes Dismiss
Brave Software home [Brave Software home]
Get Brave Search the Web

Membership and Memorization in LLM Knowledge Distillation

Ziqi Zhang (Peking University), Ali Shahin Shamsabadi (Brave Software), Hanxiao Lu (Purdue University), Yifeng Cai (Purdue University), Hamed Haddadi (Brave Software, Imperial College London) | Privacy, LLM

EMNLP 2025

Recent advances in Knowledge Distillation (KD) aim to mitigate the high computational demands of Large Language Models (LLMs) by transferring knowledge from a large teacher to a smaller student model. However, students may inherit the teachers privacy when the teacher is trained on private data. In this work, we systematically characterize and investigate membership and memorization privacy risks inherent in six LLM KD techniques. Using instruction-tuning settings that span seven NLP tasks, together with three teacher model families (GPT-2, LLAMA-2, and OPT), and various size student models, we demonstrate that all existing LLM KD approaches carry membership and memorization privacy risks from the teacher to its students. However, the extent of privacy risks varies across different KD techniques. We systematically analyse how key LLM KD components (KD objective functions, student training data and NLP tasks) impact such privacy risks. We also demonstrate a significant disagreement between memorization and membership privacy risks of LLM KD techniques. Finally, we characterize per-block privacy risk and demonstrate that the privacy risk varies across different blocks by a large margin.

View paper

Links

Arxiv Code

Ready for a better Internet?

Brave’s easy-to-use browser blocks ads by default, making the Web faster, safer, and less cluttered for people all over the world.

Get Brave
Google Play Store button [Google Play Store button]
Apple App Store button [Apple App Store button]
Get Brave

User resources

Privacy policy

Blogs

For businesses

Beginner guides

Downloads

Social media

Company

Web3

Contact

Support
Advertising
adsales@brave.com Please only use this email address if you are interested in purchasing advertising with Brave. For support, please visit community.brave.app.
Business
bizdev@brave.com
Press
press@brave.com
legal@brave.com

Offices

Brave San Francisco
48 2nd St, Floor 3
San Francisco, CA 94105

Choose a language
English
Brave Software home [Brave Software home]
Terms of use | Report a security issue 2015 - 2026 Brave Software, Inc. | All rights reserved
Web Proxy Viewer  |  New URL  |  Original Page