- 1. Alleviating Privacy Attacks via Causal Learning Shruti Tople, Amit Sharma, Aditya V. Nori Microsoft Research https://arxiv.org/abs/1909.12732 https://github.com/microsoft/robustdg
- 2. Motivation: ML models leak information about data points in the training set Neural Network TrainingHealth Records (HIV/AIDS patients) ML-as-a-service Member of Train Dataset Non-member Membership Inference Attacks [SP’17][CSF’18][NDSS’19][SP’19]
- 3. The likely reason is overfitting Output 85% Output 95% Overfitting to dataset • Neural networks or associational models overfit to the training dataset • Membership inference adversary exploits differences in prediction score for training and test data [CSF’18]
- 4. Overfitting to distribution The likely reason is overfitting • Neural networks or associational models overfit to the training dataset • Membership inference attacks exploit differences in prediction score for training and test data [CSF’18] • Privacy risk can increase when model is deployed to different distributions • E.g., Hospital in one region shares the model to other regions Output 85% Output 95% Overfitting to dataset Output 75% Poor generalization across distributions exacerbates membership inference risk.
- 5. Can causal ML models help?
- 6. Can causal ML models help? Contributions 1. Causal models provide stronger (differential) privacy guarantees than associational models. • Due to their better generalizability on new distributions. 2. And hence are more robust to membership inference attacks. • As the training dataset size → ∞, membership inference attack’s accuracy drops to a random guess. 3. We empirically demonstrate privacy benefits of causal models across 5 datasets. • Associational models exhibit up to 80% attack accuracy whereas causal models exhibit attack accuracy close to 50%. Causal Learning Privacy
- 7. Disease Severity Background: Causal Learning 𝒀 Blood Pressure Heart Rate 𝑿 𝒑𝒂𝒓𝒆𝒏𝒕 𝑿 𝒑𝒂𝒓𝒆𝒏𝒕 𝑿 𝟏 𝑿 𝟐 Weight Age Use a structural causal model (SCM) that defines what conditional probabilities are invariant across different distributions [Pearl’09].
- 8. Background: Causal Learning Use a structural causal model (SCM) that defines what conditional probabilities are invariant across different distributions [Pearl’09]. Causal Predictive Model: A prediction model based only on the parents of the outcome Y. What if SCM is not known? Learn an invariant feature representation across distributions [ABGD’19, MTS’20]. For ML models, causal learning can be useful for fairness [KLRS’17] explainability [DSZ’16, MTS’19] privacy [this work] Disease Severity 𝒀 Blood Pressure Heart Rate 𝑿 𝒑𝒂𝒓𝒆𝒏𝒕 𝑿 𝒑𝒂𝒓𝒆𝒏𝒕 𝑿 𝟏 𝑿 𝟐 Weight Age
- 9. 𝒀 𝑋𝑆0 𝑋 𝑃𝐴 𝑋𝑆2 𝑋𝑆1 𝑋 𝐶𝐻 𝑋𝑐𝑝 Intervention Why is a model based on causal parents invariant across data distributions?
- 10. Why is a model based on causal parents invariant across data distributions? 𝒀 𝑋𝑆0 𝑋 𝑃𝐴 𝑋𝑆2 𝑋𝑆1 𝑋 𝐶𝐻 𝑋𝑐𝑝 Intervention 𝒀 𝑋𝑆0 𝑋 𝑃𝐴 𝑋𝑆2 𝑋𝑆1 𝑋 𝐶𝐻 𝑋𝑐𝑝 𝑃(𝑌|𝑋 𝑃𝐴) is invariant across different distributions, unless there is a change in true data-generating process for Y.
- 11. Result 1: Worst-case out-of-distribution error of a causal model is lower than an associational model.
- 12. For any model ℎ, and 𝑃∗ such that 𝑃∗ 𝑌 𝑋 𝑃𝐴 = 𝑃(𝑌|𝑋 𝑃𝐴), In-Distribution Error (IDE)= 𝐈𝐃𝐄 𝐏 𝒉, 𝒚 = 𝐋 𝑷 𝒉, 𝒚 − 𝐋 𝑺∼P(𝒉, 𝒚) Expected loss on the same distribution as the train data Out-of-Distribution Error (ODE)=𝐎𝐃𝐄 𝐏,𝐏∗ 𝒉, 𝒚 = 𝐋 𝑷∗ 𝒉, 𝒚 − 𝐋 𝑺∼P 𝒉, 𝒚 Expected loss on a different distribution 𝑃∗ than the train data Result 1: Worst-case out-of-distribution error of a causal model is lower than an associational model.
- 13. For any model ℎ, and 𝑃∗ such that 𝑃∗ 𝑌 𝑋 𝑃𝐴 = 𝑃(𝑌|𝑋 𝑃𝐴), In-Distribution Error (IDE)= 𝐈𝐃𝐄 𝐏 𝒉, 𝒚 = 𝐋 𝑷 𝒉, 𝒚 − 𝐋 𝑺∼P(𝒉, 𝒚) Expected loss on the same distribution as the train data Out-of-Distribution Error (ODE)=𝐎𝐃𝐄 𝐏,𝐏∗ 𝒉, 𝒚 = 𝐋 𝑷∗ 𝒉, 𝒚 − 𝐋 𝑺∼P 𝒉, 𝒚 Expected loss on a different distribution 𝑃∗ than the train data Proof Idea. Simple case: Assume 𝑦 = 𝑓(𝒙) is deterministic. 𝐎𝐃𝐄 𝐏,𝐏∗ 𝒉 𝐜, 𝒚 ≤ 𝐈𝐃𝐄 𝐏(𝒉 𝒄, 𝒚) + 𝒅𝒊𝒔𝒄 𝐋 𝑷, 𝑷∗ Discrepancy b/w 𝑷 and 𝑷∗ distributions Causal Model Result 1: Worst-case out-of-distribution error of a causal model is lower than an associational model.
- 14. For any model ℎ, and 𝑃∗ such that 𝑃∗ 𝑌 𝑋 𝑃𝐴 = 𝑃(𝑌|𝑋 𝑃𝐴), In-Distribution Error (IDE)= 𝐈𝐃𝐄 𝐏 𝒉, 𝒚 = 𝐋 𝑷 𝒉, 𝒚 − 𝐋 𝑺∼P(𝒉, 𝒚) Expected loss on the same distribution as the train data Out-of-Distribution Error (ODE)=𝐎𝐃𝐄 𝐏,𝐏∗ 𝒉, 𝒚 = 𝐋 𝑷∗ 𝒉, 𝒚 − 𝐋 𝑺∼P 𝒉, 𝒚 Expected loss on a different distribution 𝑃∗ than the train data Proof Idea. Simple case: Assume 𝑦 = 𝑓(𝒙) is deterministic. 𝐎𝐃𝐄 𝐏,𝐏∗ 𝒉 𝐜, 𝒚 ≤ 𝐈𝐃𝐄 𝐏(𝒉 𝒄, 𝒚) + 𝒅𝒊𝒔𝒄 𝐋 𝑷, 𝑷∗ 𝐎𝐃𝐄 𝐏,𝐏∗ 𝒉 𝒂, 𝒚 ≤ 𝐈𝐃𝐄 𝐏 𝒉 𝒂, 𝒚 + 𝒅𝒊𝒔𝒄 𝐋 𝑷, 𝑷∗ + 𝐋 𝑷∗(𝒉 𝒂,𝑷 𝑶𝑷𝑻 , 𝒚) ⇒ max 𝐏∗ 𝐎𝐃𝐄𝐁𝐨𝐮𝐧𝐝 𝐏,𝐏∗ 𝒉 𝐜, 𝒚 ≤ max 𝐏∗ 𝐎𝐃𝐄𝐁𝐨𝐮𝐧𝐝 𝐏,𝐏∗ 𝒉 𝒂, 𝒚 Discrepancy b/w 𝑷 and 𝑷∗ distributions Optimal 𝒉 𝒂 on P is not optimal on 𝑷∗ Causal Model Assoc. Model Result 1: Worst-case out-of-distribution error of a causal model is lower than an associational model.
- 15. And better generalization results in lower sensitivity for a causal model Sensitivity: If a single data point 𝒙, 𝑦 ∼ 𝑃∗ is added to the train dataset 𝑆 to create 𝑆′, how much does the learnt model h 𝑆 min change? Since the optimal causal model is the same across all 𝑃∗ , adding any 𝒙, 𝑦 ∼ 𝑃∗ has less impact on a trained causal model. Sensitivity for a causal model Sensitivity for an associational model
- 16. Main Result: A causal model has stronger Differential Privacy guarantees Let M be a mechanism that returns a ML model trained over dataset 𝑆, M(𝑆) = ℎ. Differential Privacy [DR’14]: A learning mechanism M satisfies 𝜖-differential privacy if for any two datasets, 𝑆, 𝑆′ that differ in one data point, Pr(M 𝑆 ∈𝐻) Pr(M 𝑆′ ∈𝐻) ≤ 𝑒 𝜖. (Smaller 𝜖 values provide better privacy guarantees) Since lower sensitivity ⇒ lower 𝜖, Theorem: When equivalent Laplace noise is added and models are trained on same dataset, causal mechanism MC provides 𝜖 𝐶-DP and associational mechanism MA provides 𝜖 𝐴-DP guarantees such that: 𝝐 𝒄 ≤ 𝝐 𝑨
- 17. Therefore, causal models are more robust to membership inference (MI) attacks Advantage of an MI adversary: (True Positive Rate – False Positive Rate) in detecting whether 𝑥 is from training dataset or not. [From Yeom et al. CSF’18] Membership advantage of an adversary is bounded by 𝑒 𝜖 − 1. Since the optimal causal models are the same for 𝑃 and 𝑃∗, As 𝑛 → ∞, membership advantage of causal model → 0. Theorem: When trained on the same dataset of size 𝑛, membership advantage of a causal model is lower than the membership advantage for an associational model.
- 19. Goal: Compare MI attack accuracy between causal and associational models [BN] When true causal structure is known Datasets generated from Bayesian networks: Child, Sachs, Water, Alarm Causal model: MLE estimation based on Y’s parents Associational model: Neural networks with 3 linear layers 𝑃∗: Noise added to conditional probabilities (uniform or additive) [MNIST] When true causal structure is unknown Colored MNIST dataset (Digits are correlated with color) Causal Model: Invariant Risk Minimization that utilizes 𝑃 𝑌 𝑋 𝑃𝐴 is same across distributions [ABGD’19] Associational Model: Empirical Risk Minimization using the same NN architecture 𝑃∗: Different correlations between color and digit than the train dataset Attacker Model: Predict whether an input belongs to train dataset or not
- 20. [BN] With uniform noise, MI attack accuracy for a causal model is near a random guess 80% 50% For associational models, the attacker can guess membership in training set with 80% accuracy.
- 21. [BN-Child] With uniform noise, MI attack accuracy for a causal model is near a random guess 80% 50% For associational models, the attacker can guess membership in training set with 80% accuracy. Privacy without loss in utility: Causal & DNN models achieve same prediction accuracy.
- 22. [BN-Child] MI Attack accuracy increases with amount of noise for associational models, but stays constant at 50% for causal models
- 23. [BN] Consistent results across all four datasets High attack accuracy for associational models when 𝑃∗ (Test2) has uniform noise. Same classification accuracy between causal and associational models.
- 24. [MNIST] MI attack accuracy is lower for invariant risk minimizer compared to associational model IRM model motivated by causal reasoning has 53% attack accuracy, close to random. Associational model also fails to generalize: 16% accuracy on test set. Model Train Accuracy (%) Test Accuracy (%) Attack Accuracy (%) Causal Model (IRM) 70 69 53 Associational Model (ERM) 87 16 66
- 25. Conclusion • Established theoretical connection between causality and differential privacy. • Demonstrated the benefits of causal ML models for alleviating privacy attacks, both theoretically and empirically. • Code available at https://github.com/microsoft/robustdg Future work: Investigate robustness of causal models with other kinds of adversarial attacks. Causal Learning Privacy thank you! Amit Sharma Microsoft Research
- 26. References • [ABGD’19] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. • [CSF’18] Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. CSF 2018. • [DR’14] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4):211–407, 2014. • [DSZ’16] Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In Security and Privacy (SP), 2016 IEEE Symposium on, pp. 598–617. IEEE, 2016 • [KLRS’17] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems, pp. 4066–4076, 2017. • [MTS’19] Mahajan, Divyat, Chenhao Tan, and Amit Sharma. "Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers." arXiv preprint arXiv:1912.03277 (2019). • [MTS’20] Mahajan, Divyat, Shruti Tople and Amit Sharma. “Domain Generalization using Causal Matching”. arXiv preprint arXiv:2006.07500, 2020. • [NDSS’19] Salem, A., Zhang, Y., Humbert, M., Fritz, M., and Backes, M. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. NDSS 2019. • [SP’17] Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. Security and Privacy (SP), 2017. • [SP’19] Nasr, M., Shokri, R., and Houmansadr, A. Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks. Security and Privacy (SP), 2019.