Recommended
More Related Content
What's hot
What's hot (20)
Similar to Modeling application risk at scale @ netflix
Similar to Modeling application risk at scale @ netflix (20)
Recently uploaded
Recently uploaded (20)
Modeling application risk at scale @ netflix
- 1. Modeling Application Risk At Scale @ Netflix Shannon Morrison / Scott Behrens
- 3. Problems Broad risk categorization influenced by coffee and sleep Inconsistencies in risk language with how we talk about risks Uncertainty if we're recommending the most important security things
- 4. Our Goals Better decision making Improved security self-service Improved strategic decision making More confident incident response
- 5. Risk Assessment At Netflix we partner with risk practitioners who perform risk assessments Risk assessment mitigate many of these concerns Risk assessments are manual efforts that require time and people to conduct
- 6. FAIR Primer
- 7. Introducing Sage: An Asset Risk Modeling Framework For Netflix Enable at scale risk measurement across a broad range of risk scenarios, so Netflix can achieve more efficient prioritization of security work
- 8. Example Risk Scenario Analyze the risk of an external actor(s) impact the confidentiality of sensitive data via [application] attack?
- 9. What Features Do We Use for Modeling? Assets Anything of value Risk Factors for a given asset, that contribute to likelihood or magnitude of risk Paved Road Controls for a given asset, that reduce to likelihood or magnitude of risk Security Incidents/Vulnerabilities provide prior evidence of loss event frequency Observations provide the expert with additional information to improve the accuracy of their forecasts
- 10. Asset Inventory Where do we pour store these features?
- 11. Asset Inventory Asset Inventory provides a way to navigate and query relationships between disparate infrastructure data sources such as application metadata, laptops, databases, etc. to enable us to operate confidently on challenges that span our complex environment.
- 12. Asset Inventory Application Feature Examples Example Risk Factors: Internet facing Missing authentication Insecure ports Outdated AMI Critical vulnerabilities Non Employee access Example Paved Road Controls: SSO mTLS Secure database proxy Application firewall Secure application proxy
- 13. Asset Inventory Demo Data Model
- 14. How Do We Survey? We introduce forecasters to the process Risk team offers optional calibration training to improve accuracy In Person Meeting Calibration On their own time, they fill out a 50 question survey Survey We revisit results, work through outliers, and collect feedback on the process Refine
- 16. Loss Event Frequency One time in the next 5 years: 0.2
- 17. Review Loss Event Frequency by Application and Forecaster
- 18. Step 2 Outliers
- 19. Step 2 Build Model of Loss Event Frequency
- 21. Step 2 Magnitude
- 22. Step 2 Magnitude Distribution Mean x Frequency = $Annualized Loss
- 23. Step 2 Magnitude Distribution Mean x Frequency = $Annualized Loss Impact: [$100,000, $1,000,000] Lognormal distribution mean: $400,400
- 24. Step 2 Magnitude Distribution Mean x Frequency = $Annualized Loss One time in the next 5 years: 0.2
- 25. Step 2 Magnitude Distribution Mean x Frequency = $Annualized Loss $400,400 x 0.2 = $80,800
- 26. Step 2 Write to Asset Inventory
- 27. Step 2 Security Guide and Dashboard Demo
- 28. 94% App risk model accuracy from 2020 to 2021 10% Every application with an incident or bug bounty payout was in the top 10% by frequency from 2020 to 2021
- 29. What Did We Get Wrong? Too precise Too many features Not enough coffee Unclear scenario Data reliability Feature confusion
- 30. Sage Limitations Emphasis on Paved Road Magnitude uncertainty Data incongruities Experts can be wrong Many risk scenarios happen infrequently
- 31. Conclusion Strategic Decision Making Operational and Tactical Decision Making Sage
- 32. Thanks Dave King Paul McMillan Tony Martin-Vegue Aubrey Sharwarko Jai Balani Felipe Munera Savino Amit Patil Markus De Shon
- 33. Questions? Riskquant library Quantifying Risk QCon Presentation
Editor's Notes
- For 5,500 applications, we forecasted risk scores based in frequency and dollars for application compromise events that lead to data exposure. We did this by training a machine learning model, with risk forecasters who were presented features to inform their forecast based on facts we've collected on our environment. This mirrors a traditional risk assessment approach but allows us to scale the function to thousands of risk scenarios and assets.
- Inconsistency with how teams/orgs discuss risk, which leads to bad prioritization of security work Risk is bucketed into very broad categories: High, medium, low which loses the details. Which applications expose us to the most risk and can we measure that more quantitatively (ratio) vs qualitatively (ordinal) How can we detect application changes that drastically increase or decrease our risk exposure How do we measure the success of our security program (with regards to risk factors and controls which mitigate those risk factors)? Incident Response and Triage; which applications do we fix first?
- Security Self Service Sage quantifies your asset's risks, and the model's results help explain which paved road controls mitigate the most risk. We can use the assets risk model with our self-service tooling, such as go/green, to present security paved road controls prioritized by risk. Strategic Decision Making Sage's annualized loss exposure forecasts can be aggregated by facets such as organization, technology, paved road practice adoption, and risk factors. Inform prioritization and help measure security partnerships Inform how well our paved road products are doing at reducing risks in dollars over time Surface technologies or teams with rapidly evolving risk Metrics Asset inventory, Sage, etc. all provide measurement of paved road, risk factors, risk scores over time and are surfaceable on technology, org, factor, etc. Could be the start of a really cool product We use all this to scale security LEVERAGE Unlocks the ability for recommending security controls based on measured risk factors prioritized by risk reduction in DOLLARS! No more laundry lists of security tasks Unlocks powerful analytical features to steer leaders on where to invest How well is our paved road working? Are our security partnership programs reducing risk? What orgs have seen the most significant increase/decrease in risk?
- to the sub-risks that encompass the 1,000's of assets we manage where it’s not cost effective to perform manual assessment.
- FAIR is a model that codifies and monetizes risk Risk is represented in frequency and dollars Fair provides a manual risk assessment methodology practitioners can use
- Threat Effect Asset Method
- Assets exp. applications, laptops, mobile devices Risk factors for a given asset, that contribute to likelihood or magnitude of risk events exp. application is internet facing, critical vulnerability, missing OS security patches Paved road practices for a given asset, that reduce to likelihood or magnitude of risk events exp. Meechum single sign on, laptop encryption Security Incidents for a given asset, that provide prior evidence of loss event frequency (exp. Application compromised) Observations for a given asset, that provide the expert with additional information to improve the accuracy of their forecasts (exp. Asset programming language, customer segments)
- We got feedback on more ‘human readable’ features that we’re planning to use on future surveys We have a feature definition doc that was helpful too
- The forecasters provide loss frequency for each application, which is a number from 0 to infinity, not limited to 0 to 1
- This visualization shows the frequency of loss on the y axis and the app on the x axis, with individual colors per forecaster. Review survey data for team understanding… do we have the same view of the controls and risks? Which features increase risk? Which controls reduce risk? For internet-facing applications, the secure gateway decreases the frequency of a loss event
- Once we have consistent opinions about the features, remove individual outliers. This spike in green was likely a typo on my part :)
- Once we have clean labeled data from our survey, we fit a regression model using xgboost, a library that builds gradient boosted trees. We tried several models; linear regression, random forest, SVM, and xgboost, and xgboost had the lowest RMSE (root mean squared error). Then we pulled the features from the Asset Inventory for all active applications, and predicted a frequency for each.
- We used the feature importance as we were training the model as one way to understand which features were most useful to the forecasters. We added and removed features as we went through the process. Generally, importance provides a score that indicates how useful or valuable each feature was in the construction of the boosted decision trees within the model. The more an attribute is used to make key decisions with decision trees, the higher its relative importance. This importance is calculated explicitly for each attribute in the dataset, allowing attributes to be ranked and compared to each other. Importance is calculated for a single decision tree by the amount that each attribute split point improves the performance measure, weighted by the number of observations the node is responsible for. The performance measure may be the purity (Gini index) used to select the split points or another more specific error function. The feature importances are then averaged across all of the the decision trees within the model.
- That covers frequency. For magnitude, we use the type of data that might be lost. The Risk team shared their loss tables with us; these tables have loss data per data type like PII and PCI, and include response costs. We use that, along with frequency, to run simulations using the riskquant library. The resource slide will include a link to this library, and a presentation from QCon about risk quantification at Netflix. Riskquant maps loss to a lognormal distribution, which can’t go below 0 and has a long tail to represent extreme loss events {'minimum': 0, 'tenth_percentile': 0,'mode': 0, 'median': 0, 'ninetieth_percentile': 80, 'maximum': 900}
- The mean of the magnitude distribution times the loss event frequency gives us an annualized loss in US dollars.
- Impact is a low loss and a high loss
- A number from 0 to infinity, not limited to 0 to 1
- This moves us from an ordinal scale of low, medium, high, to a ratio scale; a number that can have descriptive and inferential statistics applied to it
- Tableau demo
- 94% accuracy - percent of applications that had bug bounty or incidents payout within the range we estimated.
- Expect to do a few iterations We did 3 models and worked through feedback/concerns until we had confidence to use it
- Paved road: We dont necesasrly serve the unknown unknowns or teams which role their own secuirty capaiblities (aka they have thier own paved road defined which does mitigate risk factors)