Evaluator - SiRAcon 2018 Presentation
•3 likes•773 views
Download to read offline
Report
A high level introduction to the Evaluator toolkit for open source quantified information risk management, as presented at SiRAcon 2018 in Seattle on February 8, 2018.
Recommended
More Related Content
Similar to Evaluator - SiRAcon 2018 Presentation
Similar to Evaluator - SiRAcon 2018 Presentation(20)
More from David Severski
More from David Severski(13)
Recently uploaded
Recently uploaded(20)
Evaluator - SiRAcon 2018 Presentation
- 1. Evaluator OPEN SOURCE QUANTITATIVE RISK MANAGEMENT MADE EASY EASIER SIRACon 2018 – Data > Dogma 1
- 2. Poll SIRACon 2018 – Data > Dogma 2
- 3. Hypothesis Risk Management is Haaaaaaaaaaard… SIRACon 2018 – Data > Dogma 3
- 4. Problem Statement How do we get people with large qualitative investments comfortable with strategic quantitative risk analysis? SIRACon 2018 – Data > Dogma 4
- 5. Introducing Evaluator Providing a bridge between qualitative data to OpenFAIR quantitative risk simulation SIRACon 2018 – Data > Dogma 5
- 6. What Does Evaluator Give Me? SIRACon 2018 – Data > Dogma 6
- 7. Quick Review of OpenFAIR RISK LEF TEF VULN TC DIFF LM PLM SL SLEF SLM SIRACon 2018 – Data > Dogma 7
- 8. Evaluator’s Default Model RISK LEF TEF VULN TC DIFF LM PLM SL SLEF SLM SIRACon 2018 – Data > Dogma 8
- 9. Evaluator Default Process Flow Excel-based data acquisition Encode qualitative data into quantitative scenarios Monte Carlo scenario simulation Summarize results across multiple dimensions Default reports for jump starting analysis SIRACon 2018 – Data > Dogma 9
- 10. Flow of an Evaluator Analysis Prepare Load Encode Simulate Summarize Report SIRACon 2018 – Data > Dogma 10
- 11. Prepare SIRACon 2018 – Data > Dogma 11 Prepare Load Encode Simulate Summarize Report • Survey Instrument • Domain Dictionary • Risk Tolerances • Qualitative Mappings
- 12. Survey Instrument SIRACon 2018 – Data > Dogma 12
- 13. Load SIRACon 2018 – Data > Dogma 13 Prepare Load Encode Simulate Summarize Report • Capabilities Table • Validated Qualitative Scenarios
- 14. Encode SIRACon 2018 – Data > Dogma 14 Prepare Load Encode Simulate Summarize Report • Quantitative Scenarios
- 15. Simulate SIRACon 2018 – Data > Dogma 15 Prepare Load Encode Simulate Summarize Report Dataframe of Results • Threat Event Count • Loss Event Count • ALE/SLE • VULN - TC and DIFF Exceedance
- 16. Summarize SIRACon 2018 – Data > Dogma 16 Prepare Load Encode Simulate Summarize Report Per-scenario and per-domain summary files Ready for Analysis with R, Tableau, etc.
- 17. Report SIRACon 2018 – Data > Dogma 17 Prepare Load Encode Simulate Summarize Report
- 18. Risk Dashboard SIRACon 2018 – Data > Dogma 18
- 19. Risk Report SIRACon 2018 – Data > Dogma 19
- 20. Scenario Explorer SIRACon 2018 – Data > Dogma 20
- 21. Ugh…that’s too much typing! SIRACon 2018 – Data > Dogma 21
- 22. MVA (Minimum Viable Analysis) > evaluator::create_templates() > base_dir <- “~/evaluator” > source(“~/evaluator/run_analysis.R”) SIRACon 2018 – Data > Dogma 22
- 23. davidski/evaluator-docker Your Container is Ready SIRACon 2018 – Data > Dogma 23
- 24. Advanced Options • Write your own model • Try different distributions SIRACon 2018 – Data > Dogma 24
- 25. Evaluator in the Wild • Strategic Technology Risk • HIPAA • PCI-DSS • Binary Risk Analysis (BRA) SIRACon 2018 – Data > Dogma 25
- 26. Future • Export scenarios to other tools • Increase performance and ease of modelling • Sensitivity analysis SIRACon 2018 – Data > Dogma 26
- 27. Call to Action • Try out Evaluator! • Find the rough edges! • Provide feedback! • Do more quantitative risk! SIRACon 2018 – Data > Dogma 27
- 28. Would You Like to Know More? SIRACon 2018 – Data > Dogma 28
- 29. Q&A https://evaluator.severski.net SIRACon 2018 – Data > Dogma 29
Editor's Notes
- Super excited to be speaking to such a great group in such a great location with such great beverages! In my daily work, constant trade offs between accuracy and precision. This also applied to talk synopsis. This is accurate, but perhaps not precise. Talk is more about how to make quant. risk management easier But first…
- Who is doing risk management via: some form via qualitative methods (count) some form of quantitative methods? Expected results - Even in this audience, qualitative methods dominate! This crowd understands the problems with qualitative risk management Range compression Heat Maps Peanut butter x Jet engine = shiny (ir-)reproducibility (changing staff often means changing inputs) Despite SIRA being around for over 7 years [2011- whoa, I was at the first SIRACon back in 2012], mainstream adoption of quant is still lagging
- People think quant risk management is haaaaaaaaard. Read the Jack2 book (Measuring and Managing Information Risk) and give a thumbs up, but still aren’t sure how to move forward Not ready to invest for a commercial solution There's a large procedural and psychological investment in qualitative processes Personal background - came from a well established qualitative program and wanted to level up to do more quant
- Shout out to Jay Jacobs & Chris Hayes, 2012, OpenPert Jay called OpenPert a ”gateway drug” I flatter myself by trying to continue that trend Wrapping qualitative analysis in a big fuzzy hug
- Goal: Provide a bridge from qualitative data to an OpenFAIR-based quantitative risk simulation Move the conversation from bikeshedding over methods to discussing data inputs. R Library Why R? Free & Open Powerful tools for working with simulations and statistics Powerful and beautiful visualization framework (ggplot)
- Building blocks for: Gathering structured qualitative data Converting to quantitative estimates Running an OpenFAIR analysis Reporting Easy Out of the box path, with the ability to get more sophisticated as your program changes
- Quick refresher on principal elements of the OpenFAIR taxonomy. Among all the risk frameworks, OpenFAIR is still one I come back to again and again
- The Evaluator Out of Box Experience (OOBE) Implementing other models is made possible through simple primitives like `sample_lm` and `sample_tc`.
- High level view Excel – the universal solvent for data in enterprises Structured method for converting qualitative data into quantitative estimates
- DEMO – Show Spreadsheet create_templates() DEMO – Load Data Read domains.csv import_spreadsheet
- DEMO – Show Spreadsheet create_templates() DEMO – Load Data Read domains.csv import_spreadsheet
- DEMO - Show conversion to quant Transform scenarios from qualitative to quantitative Customizable translation table from qualitative to quantitative parameters (BetaPert out of box)
- DEMO – Run a small simulation, but load a more full set of results
- DEMO – Run summarization Multi-dimensional roll up of results Scenario-level Domain-level
- DEMO – Scenario Explorer, Risk Dashboard, Report
- But, David…this is way too much typing. My analyst community aren’t programmers!
- Absolute minimum to get up and running Create starter files Edit starter files Set the home directory Run analysis Profit!
- BETA-quality Docker images are available on the Docker Hub (they work, but are less tested) docker image pull davidski/evaluator-docker
- Out of the box model TEF, TC, DIFF, LM Multiple controls are averaged (mean) Other possibilities SLM & SLF Controls Weakest link Strongest link
- Evaluator has been used in healthcare, cloud providers, finance, and the retail sector. All over $1 billion USD – Which is a surprise to me! Need more not huge orgs! Used for Strategic risk HIPAA risk analysis PCI-DSS analysis BRA-to-OpenFAIR translation guide (Recently built, but not yet part of Evaluator) …may be a future blog post
- Replace modelling engine with either MC2D or Stan
- Rough edges - Currently optimized for Excel -> Quant -> Analysis