BlueHat Seattle 2019 || Building Secure Machine Learning Pipelines: Security Patterns and Challenges
•
1 like•173 views
Anamitra Dutta Majumdar & Anubhav Saini Increasing adoption of Machine Learning and Artificial Intelligence by data-driven organizations like LinkedIn is posing some important challenges related to data security and privacy. On the one hand, member data is an asset that unlocks unlimited business potential whereas, on the other hand, the consumption of the data must happen in a secure and privacy-preserving manner. This poses an interesting challenge for security and operations teams in the organization. In this presentation, we will walk through all the well-known use cases of machine learning at LinkedIn and also the phases of a machine learning pipeline. We will identify key security gaps and the corresponding security controls to address the gaps at each phase of any machine learning pipeline. The associated scalability and operational challenges for the application of security control will be explained. Controls in each phase would be put into the perspective of the Productive Machine Learning pipeline phases being built at LinkedIn There will be a section on how Blueshift will impact the application of security controls once compute and data have been decoupled. By the end of the talk, we would have described what a secure machine learning pipeline looks like and what are the key security patterns to be put in place to secure the pipeline.
More Related Content
What's hot
What's hot (19)
Similar to BlueHat Seattle 2019 || Building Secure Machine Learning Pipelines: Security Patterns and Challenges
Similar to BlueHat Seattle 2019 || Building Secure Machine Learning Pipelines: Security Patterns and Challenges (20)
More from BlueHat Security Conference
More from BlueHat Security Conference (20)
Recently uploaded
Recently uploaded (20)
BlueHat Seattle 2019 || Building Secure Machine Learning Pipelines: Security Patterns and Challenges
- 2. Anamitra Dutta Majumdar Anubhav Saini LinkedIn House Security Team Building Secure Machine Learning Pipelines: Challenges and security Patterns
- 3. Talk Outline ML Use cases at LinkedIn Phases in ML pipelines and Infrastructure Security Risks and Security Solution Patterns Scalability and Security Control challenges.
- 4. Acknowledgements SRE, Data APA, Foundation and AI teams House Security Team
- 5. Major Machine Learning Use Cases at LinkedIn Build Network: Connect members Find Jobs: Connect members to jobs Feed: Predict and recommend contents to members Anti-abuse: Detect member Abuse on the platform.
- 6. Phases in Machine Learning Pipeline and ProML Primary security concerns • Data Exfi ltration • Unautho rized Data Access Feature Engineering Mode; Artifac t Store Model Health Assurance Model Training Compute Model Deployment Ececutor Model scoring and Selection Hyperpara meter tuning Model Explorer Model Registry Model Inference Engine AI Meta data Hub Proml Work Space App Plugins/SDKs/Pip eline framework Model Runtime Environment Data Store Experimentation: Model Training and Evaluation Model Deployment Model Serving and Inference
- 7. ML Flow Architecture Model Artifact Store Member activity event bus Model Serving on Model Runtime Model Deployment Executor Model Registration and publish ONLINE Experimentation Training Scoring and evaluation OFFLINE Data
- 8. ML Pipeline Security Challenges Experimentation Phase: Unauthorized data Access, Sensitive data leakage Model Training Phase:, Sensitive data generation and leakage Deployment Phase: Unauthorized Model actions Leakage of sensitive models Inference Phase: DoS Security Misconfiguration, Member Inference, Vulnerabilities
- 9. Security Controls and Patterns • Access Controls • Encryption • Privacy Preserving Libraries: • Feature sensitivity annotation Experimentation • Encryption • Authenticated, authorized and automated flows • Cleanup Sensitive intermediate dataset • ML Classifiers Model Training • Dual Verification and Multi-factor Authentication • Model Randomization • Use of synthetic data Model Deployment • Visibility into workload • Segregation of workloads • Secure configuration • Model Health Assurance Model Inference
- 10. Challenges: Heterogenous authentication controls for offline and online world OFFLINE GRID Name Node Data Node Distributed ML JobsML Job Scheduler Kerberos Delegation Token Block Access TokenBlock Access Token Kerberos ONLNE Service A Service BMutual TLS Kerberos X509 Certificates
- 11. Security Control Pattern: Heterogenous Authentication and authorization control pattern Translator Service Identity Management System Secret Store Distributed Compute Distributed Storage Service B Service A
- 12. Segregation of Compute and Storage to remove Tight Coupling Web Server
- 14. The case for containerizatio n in ML Pipelines Run Entity Embedding and Convolutional Neural Network(CNN) Specialized Hardware and Software for computational demands Use Deep learning containerized software for fast-paced innovation and scale. Distributed Workload with decoupled compute and storage
- 16. ML Container Pipeline Build: Base image + System Dep + Application Artifact Storage in Image Registry: storage and retrieval of images Container Deployment : Orchestrator, Custom deployment tools Container Management :Availability and scale at runtime Scanning and monitoring: Log monitoring, Vulnerability scanning
- 19. Container Security Challenges • Vulnerable components in internal images. • Vulnerable base image from third party Untrusted and Vulnerable Image • Insufficient Authn and AuthZ Insecure access • Lack of access controls • Poorly Separated inter-container network traffic Orchestrator Risks : • Runtime Software vulnerabilities • Insecure container Runtime configuration Container Risks :
- 20. Container Control Patterns • Certify Bases Image, System dependencies, Application artifacts • Trusted Sources Image • Encrypted channels • Certificate or token based authentication • Use Scrubber Registry • Segregate cluster based on workflow sensitivity • Enable Audit Logging • Compare with benchmark checks • Use Least Privilege access model Orchestrator • Certify Container runtime configuration • Resource Isolation • User Attribution Container
- 23. Key Take ways for building ML Pipeline Security Patterns Segregation of Infrastructure Segregation of storage and computation Segregation based workload sensitivity Control plane and data plane components AI Metadata system Model training and inference time security threats and requirements Centralized Feature Metadata System Monitoring Continuous monitoring Scanning Security metrics Security Infrastructure Efficient Identity Management platform wrappers and access layers Scalable Key Management System Security Control Scaling Engineer and operationalize the Automation of the security controls
- 24. Thank You Contact the presenters at • amajumdar@linkedin.com • ansaini@linkedin.com