Successfully reported this slideshow.
Your SlideShare is downloading. ×

Evaluator - SiRAcon 2018 Presentation

Ad

Evaluator
OPEN SOURCE QUANTITATIVE RISK MANAGEMENT MADE
EASY EASIER
SIRACon 2018 – Data > Dogma 1

Ad

Poll
SIRACon 2018 – Data > Dogma 2

Ad

Hypothesis
Risk Management is Haaaaaaaaaaard…
SIRACon 2018 – Data > Dogma 3

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Upcoming SlideShare
Tidyrisk Workshop
Tidyrisk Workshop
Loading in …3
×

Check these out next

1 of 29 Ad
1 of 29 Ad

Evaluator - SiRAcon 2018 Presentation

Download to read offline

A high level introduction to the Evaluator toolkit for open source quantified information risk management, as presented at SiRAcon 2018 in Seattle on February 8, 2018.

A high level introduction to the Evaluator toolkit for open source quantified information risk management, as presented at SiRAcon 2018 in Seattle on February 8, 2018.

Advertisement
Advertisement

More Related Content

Advertisement

Evaluator - SiRAcon 2018 Presentation

  1. 1. Evaluator OPEN SOURCE QUANTITATIVE RISK MANAGEMENT MADE EASY EASIER SIRACon 2018 – Data > Dogma 1
  2. 2. Poll SIRACon 2018 – Data > Dogma 2
  3. 3. Hypothesis Risk Management is Haaaaaaaaaaard… SIRACon 2018 – Data > Dogma 3
  4. 4. Problem Statement How do we get people with large qualitative investments comfortable with strategic quantitative risk analysis? SIRACon 2018 – Data > Dogma 4
  5. 5. Introducing Evaluator Providing a bridge between qualitative data to OpenFAIR quantitative risk simulation SIRACon 2018 – Data > Dogma 5
  6. 6. What Does Evaluator Give Me? SIRACon 2018 – Data > Dogma 6
  7. 7. Quick Review of OpenFAIR RISK LEF TEF VULN TC DIFF LM PLM SL SLEF SLM SIRACon 2018 – Data > Dogma 7
  8. 8. Evaluator’s Default Model RISK LEF TEF VULN TC DIFF LM PLM SL SLEF SLM SIRACon 2018 – Data > Dogma 8
  9. 9. Evaluator Default Process Flow Excel-based data acquisition Encode qualitative data into quantitative scenarios Monte Carlo scenario simulation Summarize results across multiple dimensions Default reports for jump starting analysis SIRACon 2018 – Data > Dogma 9
  10. 10. Flow of an Evaluator Analysis Prepare Load Encode Simulate Summarize Report SIRACon 2018 – Data > Dogma 10
  11. 11. Prepare SIRACon 2018 – Data > Dogma 11 Prepare Load Encode Simulate Summarize Report • Survey Instrument • Domain Dictionary • Risk Tolerances • Qualitative Mappings
  12. 12. Survey Instrument SIRACon 2018 – Data > Dogma 12
  13. 13. Load SIRACon 2018 – Data > Dogma 13 Prepare Load Encode Simulate Summarize Report • Capabilities Table • Validated Qualitative Scenarios
  14. 14. Encode SIRACon 2018 – Data > Dogma 14 Prepare Load Encode Simulate Summarize Report • Quantitative Scenarios
  15. 15. Simulate SIRACon 2018 – Data > Dogma 15 Prepare Load Encode Simulate Summarize Report Dataframe of Results • Threat Event Count • Loss Event Count • ALE/SLE • VULN - TC and DIFF Exceedance
  16. 16. Summarize SIRACon 2018 – Data > Dogma 16 Prepare Load Encode Simulate Summarize Report Per-scenario and per-domain summary files Ready for Analysis with R, Tableau, etc.
  17. 17. Report SIRACon 2018 – Data > Dogma 17 Prepare Load Encode Simulate Summarize Report
  18. 18. Risk Dashboard SIRACon 2018 – Data > Dogma 18
  19. 19. Risk Report SIRACon 2018 – Data > Dogma 19
  20. 20. Scenario Explorer SIRACon 2018 – Data > Dogma 20
  21. 21. Ugh…that’s too much typing! SIRACon 2018 – Data > Dogma 21
  22. 22. MVA (Minimum Viable Analysis) > evaluator::create_templates() > base_dir <- “~/evaluator” > source(“~/evaluator/run_analysis.R”) SIRACon 2018 – Data > Dogma 22
  23. 23. davidski/evaluator-docker Your Container is Ready SIRACon 2018 – Data > Dogma 23
  24. 24. Advanced Options • Write your own model • Try different distributions SIRACon 2018 – Data > Dogma 24
  25. 25. Evaluator in the Wild • Strategic Technology Risk • HIPAA • PCI-DSS • Binary Risk Analysis (BRA) SIRACon 2018 – Data > Dogma 25
  26. 26. Future • Export scenarios to other tools • Increase performance and ease of modelling • Sensitivity analysis SIRACon 2018 – Data > Dogma 26
  27. 27. Call to Action • Try out Evaluator! • Find the rough edges! • Provide feedback! • Do more quantitative risk! SIRACon 2018 – Data > Dogma 27
  28. 28. Would You Like to Know More? SIRACon 2018 – Data > Dogma 28
  29. 29. Q&A https://evaluator.severski.net SIRACon 2018 – Data > Dogma 29

Editor's Notes

  • Super excited to be speaking to such a great group in such a great location with such great beverages! 

    In my daily work, constant trade offs between accuracy and precision. This also applied to talk synopsis. This is accurate, but perhaps not precise.
    Talk is more about how to make quant. risk management easier

    But first…
  • Who is doing risk management via:
    some form via qualitative methods (count)
    some form of quantitative methods?
    Expected results - Even in this audience, qualitative methods dominate!

    This crowd understands the problems with qualitative risk management
    Range compression
    Heat Maps
    Peanut butter x Jet engine = shiny
    (ir-)reproducibility (changing staff often means changing inputs)

    Despite SIRA being around for over 7 years [2011- whoa, I was at the first SIRACon back in 2012], mainstream adoption of quant is still lagging

  • People think quant risk management is haaaaaaaaard.
    Read the Jack2 book (Measuring and Managing Information Risk) and give a thumbs up, but still aren’t sure how to move forward
    Not ready to invest for a commercial solution
    There's a large procedural and psychological investment in qualitative processes

    Personal background - came from a well established qualitative program and wanted to level up to do more quant
  • Shout out to Jay Jacobs & Chris Hayes, 2012, OpenPert
    Jay called OpenPert a ”gateway drug”
    I flatter myself by trying to continue that trend

    Wrapping qualitative analysis in a big fuzzy hug
  • Goal: Provide a bridge from qualitative data to an OpenFAIR-based quantitative risk simulation

    Move the conversation from bikeshedding over methods to discussing data inputs.

    R Library
    Why R?
    Free & Open
    Powerful tools for working with simulations and statistics
    Powerful and beautiful visualization framework (ggplot)
  • Building blocks for:
    Gathering structured qualitative data
    Converting to quantitative estimates
    Running an OpenFAIR analysis
    Reporting

    Easy Out of the box path, with the ability to get more sophisticated as your program changes
  • Quick refresher on principal elements of the OpenFAIR taxonomy.

    Among all the risk frameworks, OpenFAIR is still one I come back to again and again
  • The Evaluator Out of Box Experience (OOBE)

    Implementing other models is made possible through simple primitives like `sample_lm` and `sample_tc`.
  • High level view
    Excel – the universal solvent for data in enterprises

    Structured method for converting qualitative data into quantitative estimates
  • DEMO – Show Spreadsheet
    create_templates()

    DEMO – Load Data
    Read domains.csv
    import_spreadsheet
  • DEMO – Show Spreadsheet
    create_templates()

    DEMO – Load Data
    Read domains.csv
    import_spreadsheet
  • DEMO - Show conversion to quant


    Transform scenarios from qualitative to quantitative
    Customizable translation table from qualitative to quantitative parameters (BetaPert out of box)
  • DEMO – Run a small simulation, but load a more full set of results
  • DEMO – Run summarization

    Multi-dimensional roll up of results
    Scenario-level
    Domain-level
  • DEMO – Scenario Explorer, Risk Dashboard, Report
  • But, David…this is way too much typing. My analyst community aren’t programmers!
  • Absolute minimum to get up and running

    Create starter files

    Edit starter files

    Set the home directory

    Run analysis

    Profit!
  • BETA-quality Docker images are available on the Docker Hub (they work, but are less tested)

    docker image pull davidski/evaluator-docker
  • Out of the box model
    TEF, TC, DIFF, LM
    Multiple controls are averaged (mean)

    Other possibilities
    SLM & SLF
    Controls
    Weakest link
    Strongest link
  • Evaluator has been used in healthcare, cloud providers, finance, and the retail sector.

    All over $1 billion USD – Which is a surprise to me! Need more not huge orgs!

    Used for
    Strategic risk
    HIPAA risk analysis
    PCI-DSS analysis
    BRA-to-OpenFAIR translation guide (Recently built, but not yet part of Evaluator) …may be a future blog post
  • Replace modelling engine with either MC2D or Stan
  • Rough edges - Currently optimized for Excel -> Quant -> Analysis

×