9. Evaluator Default Process Flow
Excel-based data acquisition
Encode qualitative data into quantitative scenarios
Monte Carlo scenario simulation
Summarize results across multiple dimensions
Default reports for jump starting analysis
SIRACon 2018 – Data > Dogma 9
10. Flow of an Evaluator Analysis
Prepare Load Encode Simulate Summarize Report
SIRACon 2018 – Data > Dogma 10
Super excited to be speaking to such a great group in such a great location with such great beverages!
In my daily work, constant trade offs between accuracy and precision. This also applied to talk synopsis. This is accurate, but perhaps not precise.
Talk is more about how to make quant. risk management easier
Who is doing risk management via:
some form via qualitative methods (count)
some form of quantitative methods?
Expected results - Even in this audience, qualitative methods dominate!
This crowd understands the problems with qualitative risk management
Peanut butter x Jet engine = shiny
(ir-)reproducibility (changing staff often means changing inputs)
Despite SIRA being around for over 7 years [2011- whoa, I was at the first SIRACon back in 2012], mainstream adoption of quant is still lagging
People think quant risk management is haaaaaaaaard.
Read the Jack2 book (Measuring and Managing Information Risk) and give a thumbs up, but still aren’t sure how to move forward
Not ready to invest for a commercial solution
There's a large procedural and psychological investment in qualitative processes
Personal background - came from a well established qualitative program and wanted to level up to do more quant
Shout out to Jay Jacobs & Chris Hayes, 2012, OpenPert
Jay called OpenPert a ”gateway drug”
I flatter myself by trying to continue that trend
Wrapping qualitative analysis in a big fuzzy hug
Goal: Provide a bridge from qualitative data to an OpenFAIR-based quantitative risk simulation
Move the conversation from bikeshedding over methods to discussing data inputs.
Free & Open
Powerful tools for working with simulations and statistics
Powerful and beautiful visualization framework (ggplot)
Building blocks for:
Gathering structured qualitative data
Converting to quantitative estimates
Running an OpenFAIR analysis
Easy Out of the box path, with the ability to get more sophisticated as your program changes
Quick refresher on principal elements of the OpenFAIR taxonomy.
Among all the risk frameworks, OpenFAIR is still one I come back to again and again
The Evaluator Out of Box Experience (OOBE)
Implementing other models is made possible through simple primitives like `sample_lm` and `sample_tc`.
High level view
Excel – the universal solvent for data in enterprises
Structured method for converting qualitative data into quantitative estimates
DEMO – Show Spreadsheet
DEMO – Load Data
DEMO – Show Spreadsheet
DEMO – Load Data
DEMO - Show conversion to quant
Transform scenarios from qualitative to quantitative
Customizable translation table from qualitative to quantitative parameters (BetaPert out of box)
DEMO – Run a small simulation, but load a more full set of results
DEMO – Run summarization
Multi-dimensional roll up of results
DEMO – Scenario Explorer, Risk Dashboard, Report
But, David…this is way too much typing. My analyst community aren’t programmers!
Absolute minimum to get up and running
Create starter files
Edit starter files
Set the home directory
BETA-quality Docker images are available on the Docker Hub (they work, but are less tested)
docker image pull davidski/evaluator-docker
Out of the box model
TEF, TC, DIFF, LM
Multiple controls are averaged (mean)
SLM & SLF
Evaluator has been used in healthcare, cloud providers, finance, and the retail sector.
All over $1 billion USD – Which is a surprise to me! Need more not huge orgs!
HIPAA risk analysis
BRA-to-OpenFAIR translation guide (Recently built, but not yet part of Evaluator) …may be a future blog post
Replace modelling engine with either MC2D or Stan
Rough edges - Currently optimized for Excel -> Quant -> Analysis