Deep Learning with Differential Privacy¶
Why this mattered¶
TBD
Abstract¶
Machine learning techniques based on neural networks are achieving remarkable results in a wide variety of domains. Often, the training of models requires large, representative datasets, which may be crowdsourced and contain sensitive information. The models should not expose private information in these datasets. Addressing this goal, we develop new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy. Our implementation and experiments demonstrate that we can train deep neural networks with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.
Related¶
- cite → Learning representations by back-propagating errors — Deep Learning with Differential Privacy trains neural networks with differentially private stochastic gradient descent based on backpropagation.
- cite → Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — Deep Learning with Differential Privacy uses rectifier-based deep networks like those improved by PReLU initialization as target models for private training.
- cite → Going deeper with convolutions — Deep Learning with Differential Privacy evaluates private optimization on deep convolutional architectures related to GoogLeNet.
- cite → Gradient-based learning applied to document recognition — Deep Learning with Differential Privacy applies private stochastic gradient training to convolutional neural networks descended from LeNet.
- cite → ImageNet classification with deep convolutional neural networks — Deep Learning with Differential Privacy uses AlexNet-style ImageNet convolutional networks as representative deep models for private learning.
- cite → Mastering the game of Go with deep neural networks and tree search — Differentially private deep learning is motivated by privacy risks in high-performing neural-network systems such as the deep policy and value networks used for Go.
- enables ← Learning representations by back-propagating errors — Backpropagation supplies the gradient computations that Abadi et al. privatize with clipped, noise-added stochastic gradient descent.
- enables ← Gradient-based learning applied to document recognition — LeCun et al.'s gradient-based neural-network training is the optimization setting that Deep Learning with Differential Privacy adapts by making SGD differentially private.