Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Battista Biggio, Invited Keynote @ AISec 2014 - On Learning and Recognition of Secure Patterns

Learning and recognition of secure patterns is a well-known problem in nature. Mimicry and camouflage are widely-spread techniques in the arms race between predators and preys. All of the information acquired by our senses is therefore not necessarily secure or reliable. In machine learning and pattern recognition systems, we have started investigating these issues only recently, with the goal of learning to discriminate between secure and hostile patterns. This phenomenon has been especially observed in the context of adversarial settings like biometric recognition, malware detection and spam filtering, in which data can be adversely manipulated by humans to undermine the outcomes of an automatic analysis. As current pattern recognition methods are not natively designed to deal with the intrinsic, adversarial nature of these problems, they exhibit specific vulnerabilities that an adversary may exploit either to mislead learning or to avoid detection. Identifying these vulnerabilities and analyzing the impact of the corresponding attacks on pattern classifiers is one of the main open issues in the novel research field of adversarial machine learning.

In the first part of this talk, I introduce a general framework that encompasses and unifies previous work in the field, allowing one to systematically evaluate classifier security against different, potential attacks. As an example of application of this framework, in the second part of the talk, I discuss evasion attacks, where malicious samples are manipulated at test time to avoid detection. I then show how carefully-designed poisoning attacks can mislead learning of support vector machines by manipulating a small fraction of their training data, and how to poison adaptive biometric verification systems to compromise the biometric templates (face images) of the enrolled clients. Finally, I briefly discuss our ongoing work on attacks against clustering algorithms, and sketch some possible future research directions.

  • Be the first to comment

  • Be the first to like this

Battista Biggio, Invited Keynote @ AISec 2014 - On Learning and Recognition of Secure Patterns

  1. 1. Pa#ern Recogni-on and Applica-ons Lab On Learning and Recognition of Secure Patterns University of Cagliari, Italy Department of Electrical and Electronic Engineering BaAsta Biggio Dept. Of Electrical and Electronic Engineering University of Cagliari, Italy Sco#sdale, Arizona, AISec 2014 US, Nov., 7 2014
  2. 2. Secure Patterns in Nature • Learning of secure patterns is a well-known problem in nature – Mimicry and camouflage – Arms race between predators and preys http://pralab.diee.unica.it 2
  3. 3. Secure Patterns in Computer Security • Similar phenomenon in machine learning and computer security – Obfuscation and polymorphism to hide malicious content Spam emails Malware Start 2007 with a bang! Make WBFS YOUR PORTFOLIO’s first winner of the year ..." http://pralab.diee.unica.it <script type="text/javascript" src=”http:// palwas.servehttp.com//ml.php"></ script>" ... var PGuDO0uq19+PGuDO0uq20; EbphZcei=PVqIW5sV.replace(/jTUZZ/ g,"%"); var eWfleJqh=unescape;" Var NxfaGVHq=“pqXdQ23KZril30”;" q9124=this; var SkuyuppD= q9124["WYd1GoGYc2uG1mYGe2YnltY".r eplace(/[Y12WlG:]/g, "")];SkuyuppD.write(eWfleJqh(Ebph Zcei));" ..." 3
  4. 4. Arms Race • Adaptation/evolution is crucial to survive! http://pralab.diee.unica.it Attackers Evasion techniques System designers Design of effective countermeasures text analysis on spam emails visual analysis of attached images Image-based spam … 4
  5. 5. Machine Learning in Computer Security http://pralab.diee.unica.it 5
  6. 6. Design of Learning-based Systems Training phase http://pralab.diee.unica.it xx x x x x x x x x x x x x x x x x1 x2 ... xd pre-­‐processing and feature extrac-on training data (with labels) classifier learning Linear classifiers: assign a weight to each feature and classify a sample based on the sign of its score f (x) = sign(wT x) +1, malicious −1, legitimate "# $ %$ start" bang portfolio winner" year ... university campus" Start 2007 with a bang! Make WBFS YOUR PORTFOLIO’s first winner of the year ..." start" bang portfolio winner" year ... university campus" 1 1" 1" 1" 1" ..." 0" 0" SPAM x start" bang portfolio winner" year ... university campus" w +2 +1" +1" +1" +1" ..." -3" -4" 6
  7. 7. Design of Learning-based Systems Test phase http://pralab.diee.unica.it x1 x2 ... xd pre-­‐processing and feature extrac-on test data x x x x x x x x x x x x x x x x x classifica-on and performance evalua-on e.g., classifica-on accuracy Start 2007 with a bang! Make WBFS YOUR PORTFOLIO’s first winner of the year ..." start" bang portfolio winner" year ... university campus" 1 1" 1" 1" 1" ..." 0" 0" start" bang portfolio winner" year ... university campus" +2 +1" +1" +1" +1" ..." -3" -4" +6 > 0, SPAM" (correctly classified) x w Linear classifiers: assign a weight to each feature and classify a sample based on the sign of its score f (x) = sign(wT x) +1, malicious −1, legitimate "# $ %$ 7
  8. 8. Can Machine Learning Be Secure? • Problem: how to evade a linear (trained) classifier? Start 2007 with a bang! Make WBFS YOUR PORTFOLIO’s first winner of the year ..." http://pralab.diee.unica.it start" bang portfolio winner" year ... university campus" 1 1" 1" 1" 1" ..." 0" 0" f (x) = sign(wT x) +6 > 0, SPAM" (correctly classified) x start" bang portfolio winner" year ... university campus" w +2 +1" +1" +1" +1" ..." -3" -4" x’ St4rt 2007 with a b4ng! Make WBFS YOUR PORTFOLIO’s first winner of the year ... campus! start" bang portfolio winner" year ... university campus" 0 0! 1" 1" 1" ..." 0" 1! f (x) = sign(wT x) +3 -4 < 0, HAM" (misclassified email) 8
  9. 9. Can Machine Learning Be Secure? • Underlying assumption of machine learning techniques – Training and test data are sampled from the same distribution • In practice – The classifier generalizes well from known examples (+ random noise) – … but can not cope with carefully-crafted attacks! • It should be taught how to do that – Explicitly taking into account adversarial data manipulation – Adversarial machine learning • Problem: how can we assess classifier security in a more systematic manner? http://pralab.diee.unica.it (1) M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar. Can machine learning be secure? ASIACCS 2006 (2) B. Biggio, G. Fumera, F. Roli. Security evaluation of pattern classifiers under attack. IEEE Trans. on Knowl. and Data Engineering, 2014 9
  10. 10. Security Evalua>on of Pa@ern Classifiers http://pralab.diee.unica.it (1) B. Biggio, G. Fumera, F. Roli. Security evaluation of pattern classifiers under attack. IEEE Trans. on Knowl. and Data Engineering, 2014 (2) B. Biggio et al., Security evaluation of SVMs. SVM applications. Springer, 2014 10
  11. 11. Security Evaluation of Pattern Classifiers [B. Biggio, G. Fumera, F. Roli, IEEE Trans. KDE 2014] Adversary model • Goal of the attack • Knowledge of the attacked system • Capability of manipulating data • Attack strategy as an optimization problem Performance of more secure classifiers should degrade more gracefully under attack C2 http://pralab.diee.unica.it Bounded adversary! C1 accuracy 0 performance degradation of text classifiers in spam filtering against a different number of modified words in spam emails Security Evaluation Curve Bound on knowledge / capability (e.g., number of modified words) 11
  12. 12. Adversary’s Goal 1. Security violation [Barreno et al. ASIACCS06] – Integrity: evade detection without compromising system operation – Availability: classification errors to compromise system operation – Privacy: gaining confidential information about system users 2. Attack’s specificity [Barreno et al. ASIACCS06] – Targeted/Indiscriminate: misclassification of a specific set/any sample http://pralab.diee.unica.it Integrity Availability Privacy Spearphishing vs phishing 12
  13. 13. Adversary’s Knowledge TRAINING DATA • Perfect knowledge - Learning algorithm - Parameters (e.g. feature weights) - Feedback on decisions x x – upper bound on the performance degradation under attack http://pralab.diee.unica.it FEATURE REPRESENTATION LEARNING ALGORITHM e.g., SVM x1 x2 ... xd xx x x x x x x x x x x x x x 13
  14. 14. Adversary’s Capability • Attack’s influence [Barreno et al. ASIACCS06] – Manipulation of training/test data • Constraints on data manipulation – maximum number of samples that can be added to the training data • the attacker usually controls only a small fraction of the training samples – maximum amount of modifications • application-specific constraints in feature space • e.g., max. number of words that are modified in spam emails http://pralab.diee.unica.it x ' d(x, x!) ≤ dmax x2 x1 f(x) x Feasible domain 14
  15. 15. Main Attack Scenarios • Evasion attacks – Goal: integrity violation, indiscriminate attack – Knowledge: perfect / limited – Capability: manipulating test samples e.g., manipulation of spam emails at test time to evade detection • Poisoning attacks – Goal: availability violation, indiscriminate attack – Knowledge: perfect / limited – Capability: injecting samples into the training data e.g., send spam with some ‘good words’ to poison the anti-spam filter, which may subsequently misclassify legitimate emails containing such ‘good words’ http://pralab.diee.unica.it 15
  16. 16. Targeted classifier: SVM • Maximum-margin linear classifier f (x) = sign(g(x)), g(x) = wT x + b min w,b 1 2 http://pralab.diee.unica.it Σ wTw+C max(0,1− yi f (xi )) i 1/margin classifica-on error on training data (hinge loss) 16
  17. 17. Kernels and Nonlinearity Σ →g(x) = αi w = αi Σ + b • Enables learning and classification using only dot products between samples • Kernel functions for nonlinear classification – e.g., RBF −2 Kernel ( 2 ) −1.5 −1 −0.5 0 0.5 1 1.5 http://pralab.diee.unica.it yixi i yi x, xi i support vectors k(x, xi ) = exp −γ x − xi 17
  18. 18. Evasion A@acks http://pralab.diee.unica.it 18 1. B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Srndic, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. ECML PKDD, 2013. 2. B. Biggio et al., Security evaluation of SVMs. SVM applications. Springer, 2014
  19. 19. A Simple Example • Problem: how to evade a linear (trained) classifier? – We have seen this already… St4rt 2007 with a b4ng! Make WBFS YOUR PORTFOLIO’s first winner of the year ... campus! • But… what if the classifier is nonlinear? f (x) = sign(wT x) – Decision functions can be arbitrarily complicated, with no clear relationship between features (x) and classifier parameters (w) http://pralab.diee.unica.it start" bang portfolio winner" year ... university campus" 0 0! 1" 1" 1" ..." 0" 1! start" bang portfolio winner" year ... university campus" +2 +1" +1" +1" +1" ..." -3" -4" +3 -4 < 0, HAM" (misclassified email) x’ w 19
  20. 20. Gradient-descent Evasion Attacks • Goal: maximum-confidence evasion • Knowledge: perfect • Attack strategy: • Non-linear, constrained optimization – Gradient descent: approximate solution for smooth functions • Gradients of g(x) can be analytically computed in many cases – SVMs, Neural networks http://pralab.diee.unica.it −2 −1.5 −1 −0.5 0 0.5 1 x f (x) = sign(g(x)) = +1, malicious −1, legitimate "# $ %$ min x' g(x') s.t. d(x, x') ≤ dmax x ' 20
  21. 21. Computing Descent Directions Support vector machines Σ xi )+ b, ∇g(x) = αi g(x) = αi RBF kernel gradient: ∇k(x, xi ) = −2γ exp −γ || x − xi ||2 { }(x − xi ) Neural networks x1 xd http://pralab.diee.unica.it δ1 δk δm xf g(x) w1 wk wm v11 vmd vk1 … … … … yik(x, i yi∇k(x, xi ) Σ i mΣ g(x) = 1+ exp − wkδk (x) k=1 # $ % & ' ( ) * + −1 , - . ∂g(x) ∂xf mΣ = g(x)(1− g(x)) wkδk (x) 1−δk ( (x))vkf k=1 21
  22. 22. An Example on Handwritten Digits • Nonlinear SVM (RBF kernel) to discriminate between ‘3’ and ‘7’ • Features: gray-level pixel values – 28 x 28 image = 784 features Before attack (3 vs 7) 5 10 15 20 25 5 10 15 20 25 http://pralab.diee.unica.it After attack, g(x)=0 5 10 15 20 25 5 10 15 20 25 After attack, last iter. 5 10 15 20 25 5 10 15 20 25 0 500 2 1 0 −1 −2 g(x) nNuummbbeerr ooff imteoradtiifoiends gray-level values Few modifications are enough to evade detection! … without even mimicking the targeted class (‘7’) 22
  23. 23. Bounding the Adversary’s Knowledge Limited knowledge attacks • Only feature representation and learning algorithm are known • Surrogate data sampled from the same distribution as the classifier’s training data • Classifier’s feedback to label surrogate data PDd(aXta,Y ) http://pralab.diee.unica.it Surrogate training data f(x) Send queries Get labels Learn surrogate classifier f’(x) 23
  24. 24. Experiments on PDF Malware Detection • PDF: hierarchy of interconnected objects (keyword/value pairs) • Adversary’s capability – adding up to dmax objects to the PDF – removing objects may compromise the PDF file (and embedded malware code)! http://pralab.diee.unica.it /Type 2 /Page 1 /Encoding 1 … 13 0 obj << /Kids [ 1 0 R 11 0 R ] /Type /Page ... >> end obj 17 0 obj << /Type /Encoding /Differences [ 0 /C0032 ] >> endobj Features: keyword count min x' g(x') s.t. d(x, x') ≤ dmax x ≤ x' 24
  25. 25. Experiments on PDF Malware Detection • Dataset: 500 malware samples (Contagio), 500 benign (Internet) – Targeted (surrogate) classifier trained on 500 (100) samples • Evasion rate (FN) at FP=1% vs max. number of added keywords – Averaged on 5 repetitions – Perfect knowledge (PK); Limited knowledge (LK) 1 0.8 0.6 0.4 0.2 0 0 10 20 30 40 50 http://pralab.diee.unica.it d max FN SVM (Linear), λ=0 PK (C=1) LK (C=1) Linear SVM 1 0.8 0.6 0.4 0.2 0 Nonlinear SVM (RBF Kernel) 0 10 20 30 40 50 d max FN SVM (RBF), λ=0 PK (C=1) LK (C=1) Number Number of added keywords to each PDF of added keywords to each PDF 25
  26. 26. Poisoning A@acks against SVMs http://pralab.diee.unica.it 26 1. B. Biggio, B. Nelson, P. Laskov. Poisoning attacks against SVMs. ICML, 2012 2. B. Biggio et al., Security evaluation of SVMs. SVM applications. Springer, 2014
  27. 27. Poisoning Attacks against SVMs [B. Biggio, B. Nelson, P. Laskov, ICML 2012] malicious h#p://www.vulnerablehotel.com/components/ com_hbssearch/longDesc.php?h_id=1& id=-­‐2%20union%20select%20concat%28username, 0x3a,password%29%20from%20jos_users-­‐-­‐ http://pralab.diee.unica.it Classifier Web Server Tr HTTP requests h#p://www.vulnerablehotel.com/login/ legitimate ✔ ✔ 27
  28. 28. Poisoning Attacks against SVMs [B. Biggio, B. Nelson, P. Laskov, ICML 2012] • Poisoning classifier to cause denial of service ✖ malicious legitimate h#p://www.vulnerablehotel.com/login/ h#p://www.vulnerablehotel.com/components/ com_hbssearch/longDesc.php?h_id=1& id=-­‐2%20union%20select%20concat%28username, 0x3a,password%29%20from%20jos_users-­‐-­‐ http://pralab.diee.unica.it Classifier Web Server Tr HTTP requests poisoning a#ack ✖ h#p://www.vulnerablehotel.com/login/components/ h#p://www.vulnerablehotel.com/login/longDesc.php?h_id=1& 28
  29. 29. Adversary Model and Attack Strategy • Adversary model – Goal: to maximize classification error (availiability, indiscriminate) – Knowledge: perfect knowledge (trained SVM and TR set are known) – Capability: injecting samples into TR • Attack strategy – optimal attack point xc in TR that maximizes classification error http://pralab.diee.unica.it xc classifica-on error = 0.022 classifica- on error = 0.039 29
  30. 30. Adversary Model and Attack Strategy • Adversary model – Goal: to maximize classification error (availiability, indiscriminate) – Knowledge: perfect knowledge (trained SVM and TR set are known) – Capability: injecting samples into TR • Attack strategy – optimal attack point xc in TR that maximizes classification error classifica-on error = 0.022 http://pralab.diee.unica.it classifica-on error as a func-on of xc xc 30
  31. 31. Poisoning Attack Algorithm • Max. classification error L(xc) w.r.t. xc through gradient ascent • Gradient is not easy to compute – The training point affects the classification function – Details of the derivation are in the paper http://pralab.diee.unica.it (0) xc xc (0) xc xc 1. B. Biggio, B. Nelson, P. Laskov. Poisoning attacks against SVMs. ICML, 2012 31
  32. 32. Experiments on the MNIST digits Single-point attack • Linear SVM; 784 features; TR: 100; VAL: 500; TS: about 2000 – ‘0’ is the malicious (attacking) class – ‘4’ is the legitimate (attacked) one xc http://pralab.diee.unica.it (0) xc 32
  33. 33. Experiments on the MNIST digits Multiple-point attack • Linear SVM; 784 features; TR: 100; VAL: 500; TS: about 2000 – ‘0’ is the malicious (attacking) class – ‘4’ is the legitimate (attacked) one http://pralab.diee.unica.it 33
  34. 34. http://pralab.diee.unica.it A@acking Clustering 34 1. B. Biggio, I. Pillai, S. R. Bulò, D. Ariu, M. Pelillo, and F. Roli. Is data clustering in adversarial settings secure? AISec, 2013 2. B. Biggio, S. R. Bulò, I. Pillai, M. Mura, E. Z. Mequanint, M. Pelillo, and F. Roli. Poisoning complete-linkage hierarchical clustering. S+SSPR, 2014
  35. 35. Attacking Clustering • So far, we have considered supervised learning – Training data consisting of samples and class labels • In many applications, labels are not available or costly to obtain – Unsupervised learning • Training data only include samples – no labels! • Malware clustering – To identify variants of existing malware or new malware families http://pralab.diee.unica.it xx x x x x x x x x x x x x x x x x1 x2 ... xd feature extraction (e.g., URL length, num. of parameters, etc.) data collection (honeypots) clustering of malware families (e.g., similar HTTP requests) for each cluster if … then … else … data analysis / countermeasure design (e.g., signature generation) 35
  36. 36. Is Data Clustering Secure? • Attackers can poison input data to subvert malware clustering http://pralab.diee.unica.it x x x x x x x x x x x x x x x x x x1 x2 ... xd feature extraction (e.g., URL length, num. of parameters, etc.) data collection (honeypots) clustering of malware families (e.g., similar HTTP requests) for each cluster if … then … else … data analysis / countermeasure design (e.g., signature generation) Well-­‐cra1ed HTTP requests to subvert clustering h#p://www.vulnerablehotel.com/… h#p://www.vulnerablehotel.com/… h#p://www.vulnerablehotel.com/… h#p://www.vulnerablehotel.com/… … is significantly compromised … becomes useless (too many false alarms, low detection rate) 36
  37. 37. Our Work • A framework to identify/design attacks against clustering algorithms – Poisoning: add samples to maximally compromise the clustering output – Obfuscation: hide samples within existing clusters • Some clustering algorithms can be very sensitive to poisoning! – single- and complete-linkage hierarchical clustering can be easily compromised by creating heterogeneous clusters • Details on the attack derivation and implementation are in the papers Clustering on untainted data (80 samples) Clustering after adding 10 attack samples http://pralab.diee.unica.it 37 1. B. Biggio et al. Is data clustering in adversarial settings secure? AISec, 2013 2. B. Biggio et al. Poisoning complete-linkage hierarchical clustering. S+SSPR, 2014
  38. 38. Conclusions and Future Work • Learning-based systems can be vulnerable to well-crafted, sophisticated attacks devised by skilled attackers – … that exploit specific vulnerabilities of machine learning algorithms! • Future (and ongoing) work – Privacy Attacks – Secure Learning, Clustering and Feature Selection/Reduction http://pralab.diee.unica.it Secure learning algorithms Attacks against learning 38
  39. 39. Joint work with … http://pralab.diee.unica.it ? Any questions Thanks for your a#en-on! 39 … and many others
  40. 40. References 1. B. Biggio, G. Fumera, and F. Roli. Security evaluation of pattern classifiers under attack. IEEE Trans. on Knowl. and Data Eng., 26(4):984–996, April 2014. 2. M. Barreno, B. Nelson, R. Sears, A. D. Joseph, and J. D. Tygar. Can machine learning be secure? In Proc. ACM Symp. Information, Computer and Comm. Sec., ASIACCS ’06, pages 16–25, New York, NY, USA, 2006. ACM. 3. M. Barreno, B. Nelson, A. Joseph, and J. Tygar. The security of machine learning. Machine Learning, 81:121–148, 2010. 4. B. Biggio, I. Corona, B. Nelson, B. Rubinstein, D. Maiorca, G. Fumera, G. Giacinto, and F. Roli. Security evaluation of support vector machines in adversarial environments. In Y. Ma and G. Guo, eds, Support Vector Machines Applications, pages 105–153. Springer International Publishing, 2014. 5. B. Biggio, G. Fumera, and F. Roli. Pattern recognition systems under attack: Design issues and research challenges. Int’l J. Patt. Recogn. Artif. Intell., 2014, In press. 6. B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Srndic, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In H. Blockeel et al., editors, European Conf. on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), Part III, volume 8190 of LNCS, pp. 387–402. Springer, 2013. 7. B. Biggio, B. Nelson, and P. Laskov. Poisoning attacks against support vector machines. In J. Langford and J. Pineau, eds, 29th Int’l Conf. on Machine Learning, pp.1807–1814. Omnipress, 2012. 8. B. Biggio, I. Pillai, S. R. Bulò, D. Ariu, M. Pelillo, and F. Roli. Is data clustering in adversarial settings secure? In Proc. 2013 ACM Workshop on Artificial Intell. and Security, AISec ’13, pages 87–98, New York, NY, USA, 2013. ACM. 9. B. Biggio, S. R. Bulò, I. Pillai, M. Mura, E. Z. Mequanint, M. Pelillo, and F. Roli. Poisoning complete-linkage hierarchical clustering. In P. Franti, G. Brown, M. Loog, F. Escolano, and M. Pelillo, editors, Joint IAPR Int’l Workshop on Structural, Syntactic, and Statistical Patt. Recogn., volume 8621 of LNCS, pages 42–52, Joensuu, Finland, 2014. Springer Berlin Heidelberg. http://pralab.diee.unica.it 40

    Be the first to comment

    Login to see the comments

Learning and recognition of secure patterns is a well-known problem in nature. Mimicry and camouflage are widely-spread techniques in the arms race between predators and preys. All of the information acquired by our senses is therefore not necessarily secure or reliable. In machine learning and pattern recognition systems, we have started investigating these issues only recently, with the goal of learning to discriminate between secure and hostile patterns. This phenomenon has been especially observed in the context of adversarial settings like biometric recognition, malware detection and spam filtering, in which data can be adversely manipulated by humans to undermine the outcomes of an automatic analysis. As current pattern recognition methods are not natively designed to deal with the intrinsic, adversarial nature of these problems, they exhibit specific vulnerabilities that an adversary may exploit either to mislead learning or to avoid detection. Identifying these vulnerabilities and analyzing the impact of the corresponding attacks on pattern classifiers is one of the main open issues in the novel research field of adversarial machine learning. In the first part of this talk, I introduce a general framework that encompasses and unifies previous work in the field, allowing one to systematically evaluate classifier security against different, potential attacks. As an example of application of this framework, in the second part of the talk, I discuss evasion attacks, where malicious samples are manipulated at test time to avoid detection. I then show how carefully-designed poisoning attacks can mislead learning of support vector machines by manipulating a small fraction of their training data, and how to poison adaptive biometric verification systems to compromise the biometric templates (face images) of the enrolled clients. Finally, I briefly discuss our ongoing work on attacks against clustering algorithms, and sketch some possible future research directions.

Views

Total views

777

On Slideshare

0

From embeds

0

Number of embeds

112

Actions

Downloads

28

Shares

0

Comments

0

Likes

0

×