Privacy-Preserving Machine Learning

Systematization of Knowledge (SoK)

In SoK: Cryptographic Neural-Network Computation [NC@SP23], my student and I delivered a cross-community synthesis of cryptographic approaches to privacy-preserving machine learning (PPML). An SoK at a flagship venue requires deep command of a fast-moving field. We undertook a large-scale study of 53 seminal PPML papers from 2016–2022, dissecting their cryptographic designs and their “love–hate relationships” with machine learning. The paper brings structure and critical insight to a field often seen as fragmented, giving newcomers a clear entry point and helping experts build without needless reinvention. By demystifying cryptography for ML researchers and vice versa, it fosters true interdisciplinary work. We also created a dedicated website to serve as an evolving resource for the community.

Secure and Efficient Transformer Inference

Building on the SoK insights, we targeted one of PPML’s most urgent problems: secure inference for transformer-based models, the foundation of modern large language models such as GPT-3 and BERT. Secure multi-party computation (SMC) is efficient for linear layers but struggles with the non-linear softmax and Gaussian error linear unit (GELU) functions critical to transformers. Prior solutions were too slow and complex for deployment. Our work, SHAFT: Secure, Handy, Accurate, and Fast Transformer Inference [KC@NDSS25], breaks this barrier with two cryptographic firsts: (1) the first constant-round private softmax protocol for transformers, eliminating the logarithmic growth in communication rounds and unlocking scalability; and (2) a high-accuracy Fourier-based GELU protocol far more efficient than existing polynomial methods. SHAFT reduces communication costs by 25-41%, achieves up to 5.3× speedups, and matches the accuracy of non-private models. It earned the Distinguished Artifact Award, recognizing our well-documented, reproducible software, a validation of the strong, practical training my student receives.

This work improves our earlier secure softmax/sigmoid protocol [ZZCPTLY@ACSAC23] and furthers our line of works including GForce [NC@USS21], a GPU-friendly system for oblivious neural network prediction that, as noted in the SoK, remains on the Pareto frontier for selected settings. My group also pioneered Goten [NCWWZ@AAAI21], the first method to securely outsource neural network training to untrusted GPUs, achieving a major performance leap. In addition, we developed sublinear evaluation of multi-client decision trees [MTZC@NDSS21], an interpretable model valued for its explainability, and exposed vulnerabilities in privately trained models [WMWNC@IJCAI20] in a top AI venue.

Differentially Private Large Language Models (LLMs)

Beyond cryptographic methods, we present a complementary approach based on differential privacy (DP) that limits the information leaked by outputs. In DP-Forward [DCWHS@CCS23], we introduced a new DP training paradigm over LLMs. The standard DP approach, DP-SGD, adds noise to gradients during backpropagation. This is computationally and memory intensive, especially for fine-tuning massive pre-trained models, and fails to protect inference queries from attacks like embedding inversion.

DP-Forward shifts the paradigm: it perturbs embedding matrices directly during the forward pass, leading to: (1) Comprehensive privacy: rigorous local DP guarantees for both training data and user inputs at inference. (2) Superior efficiency: ~3× reduction in time and memory compared to the fastest DP-SGD implementations. (3) Stronger security and utility: up to 88 percentage point lower embedding inversion attack success rates (where DP-SGD fails entirely) and higher accuracy at equivalent privacy levels.

The above extends to our prior work on natural text sanitization, replacing sensitive words with semantically similar, non-sensitive alternatives to protect privacy while preserving utility for text analytics [YDWLSC@ACL21], and on sanitizing sentence embeddings and their labels, enabling private release of high-dimensional text representations that remain effective for downstream ML tasks [DYCS@WWW23].

We also advance federated learning (FL), developing a highly efficient secure aggregation protocol [BRECPH@ACSAC24] that reduces the communication overhead, a key bottleneck in practice, and cryptographic methods that enable differential privacy in FL for complex models such as generative adversarial networks and meta-learning [ZSDCLZW@ADMA23]. Foundational to these collaborations is private set intersection (PSI) [MC@ASIACCS22, ZC@WPES18], which aligns distributed, vertically partitioned data while ensuring that items unique to one dataset remain hidden from other data owners