Tidyrisk Workshop

David Severski
David SeverskiData Scientist
#SIRAcon
Risk Analysis with
OpenFAIR and Tidyrisk
A Hands-On Introduction
https://tidyrisk.org
#SIRAcon
Pre-Flight Checklist
Get workshop materials from https://github.com/davidski/tidyrisk-
workshop
> install.packages(“usethis”)
> usethis::use_course(“http://bit.ly/tidyrisk-workshop”)
- or (if you’re fancy) -
# git clone https://github.com/davidski/tidyrisk-workshop.git
#SIRAcon
Our Path this Afternoon
• Why Analyze Risk?
• Defining Risk Scenarios
• Estimating Risk Factors
• Quantifying Risk Exposure with Tidyrisk
#SIRAcon
Disclaimer
#SIRAcon
Why Analyze Risk?
Understand why we’re here
#SIRAcon
Problem
How do we prioritize risk management
efforts?
Given 5 minutes or $5 dollars to spend,
where should I spend it?
There are 1,000 worthy projects, but we
can only focus on 5. Which ones matter
most?
#SIRAcon
Quantitative
Answers
Given a risk scenario, what does the
future hold?
- Value at Risk
- Probable Future Loss
- Maximum Future Loss
#SIRAcon
Hello, Tidyrisk!
Collector Evaluator
#SIRAcon
Tidyrisk Flows – Single Scenario
Define
Scenario
Gather
Inputs
Run
Simulations
Report
#SIRAcon
Tidyrisk Flows – Multi-Scenario
Qualitative Analysis
Define
Scenarios
•Create
Templates
•Update
Scenarios
Gather Inputs
•Generate
Materials
Create
Quantitative
Scenarios
•Map Inputs
•Prepare
Scenarios
Run
Simulations
Report
#SIRAcon
Tidyrisk Flows – Full Quantitative
Analysis
Define
Scenarios
Gather
Inputs
• Generate
Materials
• Run
Interviews
Fit
Parameters
Create
Quantitative
Scenarios
• Combine
• Prepare
Scenarios
Run
Simulations
Report
#SIRAcon
Defining a Risk Scenario
Orient to the OpenFAIR framework
Describe a structured risk scenario
#SIRAcon
OpenFAIR Ontology
RISK
LEF
TEF VULN
TC DIFF
LM
PLM SR
SLEF SLM
#SIRAcon
Today’s Focus
RISK
LEF
TEF VULN
TC DIFF
LM
PLM SR
SLEF SLM
#SIRAcon
A Well-Formed Risk Scenario
Narrative OpenFAIR Element Scale
Who/What – How frequently may attempts be made? Threat Community (TEF) Events per year
What action – How much force is acting against the
asset?
Threat Capability (TC) Percentage
What asset – How well are we pushing back against the
attack?
Difficulty (DIFF) Percentage
What loss – How bad does it hurt, per event, when the
bad thing occurs?
Loss Magnitude (LM) Dollars (or other
currency) per event
occurrence
Who/What is doing what action against what asset that results in what loss?
#SIRAcon
Incomplete “Risks”
• Our password rotation policies are not strong enough.
• Cyber adversaries may compromise administrative credentials on our
public web site, allowing access to customer information.
• The company’s consumer web app is vulnerable to resilience
outages.
• A failure in the primary region of our cloud provider may result in a
prolonged outage on our primary web application.
• We are not competitive for talent in the local job market.
• We are unable to compete with the local job market, causing us to
loose key talent on our major initiative.
#SIRAcon
Exercise 1 – Identifying Factors
• Think of a risk situation applicable to your work
• Identify the OpenFAIR factors of this risk scenario
• Discuss with your neighbor
#SIRAcon
Gathering Inputs
Creating estimates for a risk scenario
#SIRAcon
Creating Estimates
How do we estimate?
Looking for a 90% confidence interval
Each estimate has two values:
A high (95 percentile)
A low (5 percentile)
Questions to Ask
What data sources could we use?
To whom could we talk?
#SIRAcon
Exercise 2 – Estimating Factors
• For each of the OpenFAIR factors you identified in your
scenario, create a 90% confidence High-Low estimate
• Discuss your reasoning with your neighbor
Remember
• Frequency (TEF): # of opportunities per year
• Threat Capability (TC): Strength of the adverse force (%)
• Difficulty (DIFF): Strength of our capabilities (%)
• Loss Magnitude (LM): $ per event
#SIRAcon
Working with Tidyrisk
Build a Tidyrisk scenario
Running simulations
#SIRAcon
Exercise 3 – Interactive Exploration
• Run the OpenFAIR Example application
• > library(evaluator)
• > openfair_example()
• Enter your estimates
• Run a simulation
• Explore the results
• What happens if your controls get 10% better?
• What happens if the threat strength increases by 10%?
#SIRAcon
Exercise 4 – Building a Scenario
• Translate your scenario into a tidyrisk_scenario object
• Use the RStudio Add-in
• Tools:Addins:Browse Addins…
• Print the scenario
• > print(my_scenario)
• Display the scenario parameters
• > as_tibble(my_scenario)
#SIRAcon
Running a Simulation
> my_results <- run_simulation(my_scen, iterations = 1000)
#SIRAcon
Review the Results
> glimpse(my_results)
Variable Meaning
Iteration Identifier of the simulated year
# of threat events Number of threat events that occurred in this iteration
# of loss events Number of threat events that results in losses
Vulnerability Percentage of threat events that became losses
TC exceedance Margin by which the threat action ”won” over the controls
DIFF exceedance Margin by which the controls “won” over the threat
ALE – annual loss exposure Total annual losses for that iteration
SLE – single loss exposure Single loss event size (min, max, median, mean)
#SIRAcon
Summarizing the Results
> summarize_scenario(my_results)
Variable Meaning
# of loss events Number of threat events that results in losses
ALE Total annual losses (median, max, VaR 95)
SLE Single loss event size (min, max, median, mean)
TC exceedance Margin by which the threat action ”won” over the controls
DIFF exceedance Margin by which the controls “won” over the threat
Vulnerability Percentage of threat events that became losses
#SIRAcon
Graphical Representations
Histogram of possible future losses
> exposure_histogram(my_results)
Generate a scatterplot of loss events
> loss_scatterplot(my_results)
#SIRAcon
Exercise 5
• Expanding from one to many
• From evaluator.tidyrisk.org
• Review the sample risk report
• https://evaluator.tidyrisk.org/reports/evaluator_risk_analysis.html
• Review the sample risk dashboard
• https://evaluator.tidyrisk.org/reports/evaluator_risk_dashboard.html
• Review the sample scenario explorer
• https://davidski.shinyapps.io/scenario_explorer
#SIRAcon
Look At What We’ve
Done Today!
• Defined a risk scenario
• Estimated parameters for risk
factors
• Simulated a scenario with
evaluator
• Interpreted risk results
#SIRAcon
Oh, The Places You’ll Go!
• Multiple scenario analysis
• Building a library of scenarios
• Collecting subject matter experts input
• https://tidyrisk.org
#SIRAcon
Enjoy the rest of
conference!
• Collector and Tidyrisk hex stickers
available!
1 of 31

More Related Content

What's hot(20)

ISO 27001 In The Age Of PrivacyISO 27001 In The Age Of Privacy
ISO 27001 In The Age Of Privacy
ControlCase415 views
bsi-cyber-resilience-presentationbsi-cyber-resilience-presentation
bsi-cyber-resilience-presentation
Ajai Srivastava221 views
LogSentinel Next-Gen SIEMLogSentinel Next-Gen SIEM
LogSentinel Next-Gen SIEM
Denitsa Dimova100 views
ISMS implementation challenges-KASYSISMS implementation challenges-KASYS
ISMS implementation challenges-KASYS
Reza Teynia ISMS, ITSM, MSc786 views
Introduction to CybersecurityIntroduction to Cybersecurity
Introduction to Cybersecurity
Krutarth Vasavada473 views

Similar to Tidyrisk Workshop(20)

Business cases for software securityBusiness cases for software security
Business cases for software security
Marco Morana4.4K views
DevSecOps : an IntroductionDevSecOps : an Introduction
DevSecOps : an Introduction
Prashanth B. P.275 views
ITD BSides PDX SlidesITD BSides PDX Slides
ITD BSides PDX Slides
EricGoldstrom489 views
Data Driven Risk ManagementData Driven Risk Management
Data Driven Risk Management
Resolver Inc.1.8K views
Incident Response - Eyes EverywhereIncident Response - Eyes Everywhere
Incident Response - Eyes Everywhere
Amazon Web Services275 views
Incident Response - Eyes EverywhereIncident Response - Eyes Everywhere
Incident Response - Eyes Everywhere
Amazon Web Services688 views

More from David Severski(14)

Tidyrisk - EARL Seattle 2018Tidyrisk - EARL Seattle 2018
Tidyrisk - EARL Seattle 2018
David Severski225 views
Evaluator - SiRAcon 2018 PresentationEvaluator - SiRAcon 2018 Presentation
Evaluator - SiRAcon 2018 Presentation
David Severski773 views
Data-Driven SecurityData-Driven Security
Data-Driven Security
David Severski599 views
AWS Logging and Monitoring OverviewSAWS Logging and Monitoring OverviewS
AWS Logging and Monitoring OverviewS
David Severski649 views
CISM AWS Overview (Sanitized)CISM AWS Overview (Sanitized)
CISM AWS Overview (Sanitized)
David Severski852 views
Crawl, walk...run!Crawl, walk...run!
Crawl, walk...run!
David Severski631 views
When Mallory Met Alice - A FableWhen Mallory Met Alice - A Fable
When Mallory Met Alice - A Fable
David Severski437 views
CISM IS Leadership Presentation   CISM IS Leadership Presentation
CISM IS Leadership Presentation
David Severski666 views
Building a Log Analysis PipelineBuilding a Log Analysis Pipeline
Building a Log Analysis Pipeline
David Severski2.1K views
We Have Met the EnemyWe Have Met the Enemy
We Have Met the Enemy
David Severski464 views
Even Giants Start SmallEven Giants Start Small
Even Giants Start Small
David Severski429 views

Recently uploaded(20)

ColonyOSColonyOS
ColonyOS
JohanKristiansson69 views
RIO GRANDE SUPPLY COMPANY INC, JAYSON.docxRIO GRANDE SUPPLY COMPANY INC, JAYSON.docx
RIO GRANDE SUPPLY COMPANY INC, JAYSON.docx
JaysonGarabilesEspej6 views
How Leaders See Data? (Level 1)How Leaders See Data? (Level 1)
How Leaders See Data? (Level 1)
Narendra Narendra10 views
PROGRAMME.pdfPROGRAMME.pdf
PROGRAMME.pdf
HiNedHaJar7 views
Journey of Generative AIJourney of Generative AI
Journey of Generative AI
thomasjvarghese4917 views
RuleBookForTheFairDataEconomy.pptxRuleBookForTheFairDataEconomy.pptx
RuleBookForTheFairDataEconomy.pptx
noraelstela164 views
Data structure and algorithm. Data structure and algorithm.
Data structure and algorithm.
Abdul salam 12 views
PTicketInput.pdfPTicketInput.pdf
PTicketInput.pdf
stuartmcphersonflipm286 views
MOSORE_BRESCIAMOSORE_BRESCIA
MOSORE_BRESCIA
Federico Karagulian5 views
3196 The Case of The East River3196 The Case of The East River
3196 The Case of The East River
ErickANDRADE909 views

Tidyrisk Workshop

  • 1. #SIRAcon Risk Analysis with OpenFAIR and Tidyrisk A Hands-On Introduction https://tidyrisk.org
  • 2. #SIRAcon Pre-Flight Checklist Get workshop materials from https://github.com/davidski/tidyrisk- workshop > install.packages(“usethis”) > usethis::use_course(“http://bit.ly/tidyrisk-workshop”) - or (if you’re fancy) - # git clone https://github.com/davidski/tidyrisk-workshop.git
  • 3. #SIRAcon Our Path this Afternoon • Why Analyze Risk? • Defining Risk Scenarios • Estimating Risk Factors • Quantifying Risk Exposure with Tidyrisk
  • 6. #SIRAcon Problem How do we prioritize risk management efforts? Given 5 minutes or $5 dollars to spend, where should I spend it? There are 1,000 worthy projects, but we can only focus on 5. Which ones matter most?
  • 7. #SIRAcon Quantitative Answers Given a risk scenario, what does the future hold? - Value at Risk - Probable Future Loss - Maximum Future Loss
  • 9. #SIRAcon Tidyrisk Flows – Single Scenario Define Scenario Gather Inputs Run Simulations Report
  • 10. #SIRAcon Tidyrisk Flows – Multi-Scenario Qualitative Analysis Define Scenarios •Create Templates •Update Scenarios Gather Inputs •Generate Materials Create Quantitative Scenarios •Map Inputs •Prepare Scenarios Run Simulations Report
  • 11. #SIRAcon Tidyrisk Flows – Full Quantitative Analysis Define Scenarios Gather Inputs • Generate Materials • Run Interviews Fit Parameters Create Quantitative Scenarios • Combine • Prepare Scenarios Run Simulations Report
  • 12. #SIRAcon Defining a Risk Scenario Orient to the OpenFAIR framework Describe a structured risk scenario
  • 15. #SIRAcon A Well-Formed Risk Scenario Narrative OpenFAIR Element Scale Who/What – How frequently may attempts be made? Threat Community (TEF) Events per year What action – How much force is acting against the asset? Threat Capability (TC) Percentage What asset – How well are we pushing back against the attack? Difficulty (DIFF) Percentage What loss – How bad does it hurt, per event, when the bad thing occurs? Loss Magnitude (LM) Dollars (or other currency) per event occurrence Who/What is doing what action against what asset that results in what loss?
  • 16. #SIRAcon Incomplete “Risks” • Our password rotation policies are not strong enough. • Cyber adversaries may compromise administrative credentials on our public web site, allowing access to customer information. • The company’s consumer web app is vulnerable to resilience outages. • A failure in the primary region of our cloud provider may result in a prolonged outage on our primary web application. • We are not competitive for talent in the local job market. • We are unable to compete with the local job market, causing us to loose key talent on our major initiative.
  • 17. #SIRAcon Exercise 1 – Identifying Factors • Think of a risk situation applicable to your work • Identify the OpenFAIR factors of this risk scenario • Discuss with your neighbor
  • 19. #SIRAcon Creating Estimates How do we estimate? Looking for a 90% confidence interval Each estimate has two values: A high (95 percentile) A low (5 percentile) Questions to Ask What data sources could we use? To whom could we talk?
  • 20. #SIRAcon Exercise 2 – Estimating Factors • For each of the OpenFAIR factors you identified in your scenario, create a 90% confidence High-Low estimate • Discuss your reasoning with your neighbor Remember • Frequency (TEF): # of opportunities per year • Threat Capability (TC): Strength of the adverse force (%) • Difficulty (DIFF): Strength of our capabilities (%) • Loss Magnitude (LM): $ per event
  • 21. #SIRAcon Working with Tidyrisk Build a Tidyrisk scenario Running simulations
  • 22. #SIRAcon Exercise 3 – Interactive Exploration • Run the OpenFAIR Example application • > library(evaluator) • > openfair_example() • Enter your estimates • Run a simulation • Explore the results • What happens if your controls get 10% better? • What happens if the threat strength increases by 10%?
  • 23. #SIRAcon Exercise 4 – Building a Scenario • Translate your scenario into a tidyrisk_scenario object • Use the RStudio Add-in • Tools:Addins:Browse Addins… • Print the scenario • > print(my_scenario) • Display the scenario parameters • > as_tibble(my_scenario)
  • 24. #SIRAcon Running a Simulation > my_results <- run_simulation(my_scen, iterations = 1000)
  • 25. #SIRAcon Review the Results > glimpse(my_results) Variable Meaning Iteration Identifier of the simulated year # of threat events Number of threat events that occurred in this iteration # of loss events Number of threat events that results in losses Vulnerability Percentage of threat events that became losses TC exceedance Margin by which the threat action ”won” over the controls DIFF exceedance Margin by which the controls “won” over the threat ALE – annual loss exposure Total annual losses for that iteration SLE – single loss exposure Single loss event size (min, max, median, mean)
  • 26. #SIRAcon Summarizing the Results > summarize_scenario(my_results) Variable Meaning # of loss events Number of threat events that results in losses ALE Total annual losses (median, max, VaR 95) SLE Single loss event size (min, max, median, mean) TC exceedance Margin by which the threat action ”won” over the controls DIFF exceedance Margin by which the controls “won” over the threat Vulnerability Percentage of threat events that became losses
  • 27. #SIRAcon Graphical Representations Histogram of possible future losses > exposure_histogram(my_results) Generate a scatterplot of loss events > loss_scatterplot(my_results)
  • 28. #SIRAcon Exercise 5 • Expanding from one to many • From evaluator.tidyrisk.org • Review the sample risk report • https://evaluator.tidyrisk.org/reports/evaluator_risk_analysis.html • Review the sample risk dashboard • https://evaluator.tidyrisk.org/reports/evaluator_risk_dashboard.html • Review the sample scenario explorer • https://davidski.shinyapps.io/scenario_explorer
  • 29. #SIRAcon Look At What We’ve Done Today! • Defined a risk scenario • Estimated parameters for risk factors • Simulated a scenario with evaluator • Interpreted risk results
  • 30. #SIRAcon Oh, The Places You’ll Go! • Multiple scenario analysis • Building a library of scenarios • Collecting subject matter experts input • https://tidyrisk.org
  • 31. #SIRAcon Enjoy the rest of conference! • Collector and Tidyrisk hex stickers available!

Editor's Notes

  1. Confession and story time Cat lover, and SIRAcon 2012
  2. Pilot cat Cat with checklist
  3. Not OpenFAIR training and I don’t speak on behalf of my employer.
  4. Why do any of this stuff anyways? Andrew Lang – Using statistics as a drunk uses a lamppost, more for support than illumination. In risk – Jason Leuenberger - are we using this as a crutch or a club?
  5. Ultimately risk analysis is a decision making tool. Are we doing the most important things right now? Risk analysis allows us to prioritize scenarios based upon quantified risk (VaR)
  6. Simulation based approach These values go into our risk register and GRC tools Risk budget
  7. Collector helps define scenarios, calibrates SMEs, gathers inputs, generates data inputs Evaluator runs simulations and produces reports, also can handle qualitative inputs! We’ll cover only Evaluator in today’s session
  8. Today’s focus
  9. A full quantitative process
  10. So what does a risk scenario look like? Remember that we’re looking to make a decision. THIS IS MY FAVORITE (non cat) SLIDE – Defining the problem is really important
  11. Quick discussion of OpenFAIR There are many risk frameworks, this one is ours.  Structured way of decomposing the factors of risk
  12. Evaluator’s primary out of the box level of details is TEF_TC_DIFF_PLM Where I find most of the value, but your experience may differ
  13. Before we can run simulations, we need to identify what we’re analyzing
  14. Identify the factors in a scenario and discuss with your neighbor
  15. Now that we have a scenario, let’s get some data
  16. Reference sources Data element
  17. Estimate each element State that this is not an analyst-in-a-dark-room exercise, but a collaborative work-with-SMEs conversation. (Side note, this is where Collector comes in!  ) This also helps with calibration!
  18. Let’s do some analysis!
  19. Hands on with the shiny app Record your parameters
  20. Hands on with code
  21. Run a simulation - given a scenario object - number of iterations to perform
  22. Show the results as a dataframe/ (aka a data table) What do you get? Do this on your scenario!
  23. What’s the bottom line? It’s VAR!
  24. Describe what these are and do.
  25. May go into more or less on this in the interest of time.
  26. Amazed cat, smart cat
  27. Grab a sticker!