![“More than 91 percent [of] clients
victimized by ransomware”](https://image.slidesharecdn.com/fi-crowdsourcedprobabilityestimates-180514164731/85/Crowdsourced-Probability-Estimates-A-Field-Guide-FAIR-Institute-4-320.jpg)
![13
“More than 91
percent [of] clients
victimized by
ransomware”1
• First Google result
• “Clients” are MSP’s
• Probably not
statistically
significant (not
disclosed)
Source: Datto’s 2016 Global Ransomware Report
http://pages.datto.com/rs/572-ZRG-001/images/DattoStateOfTheChannelRansomwareReport2016_RH.pdf](https://image.slidesharecdn.com/fi-crowdsourcedprobabilityestimates-180514164731/85/Crowdsourced-Probability-Estimates-A-Field-Guide-FAIR-Institute-13-320.jpg)
![14
“More than 91
percent [of] clients
victimized by
ransomware”1
• First Google result
• “Clients” are MSP’s
• Probably not
statistically
significant (not
disclosed)
Source: Datto’s 2016 Global Ransomware Report
http://pages.datto.com/rs/572-ZRG-001/images/DattoStateOfTheChannelRansomwareReport2016_RH.pdf](https://image.slidesharecdn.com/fi-crowdsourcedprobabilityestimates-180514164731/85/Crowdsourced-Probability-Estimates-A-Field-Guide-FAIR-Institute-14-320.jpg)
Uploaded byTony Martin-Vegue
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
The document discusses the prevalence of ransomware, asserting that over 91% of companies are victimized, and critiques the methodologies behind such claims, emphasizing the need for rigorous research and expert calibration. It outlines strategies for crowdsourcing expert opinions while controlling for biases and provides insights into expert elicitation techniques. Additionally, it touches on the importance of integrating expert assessments into risk evaluations while keeping a critical eye on their calibration and the diversity of opinions.
More Related Content
Similar to Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
![[DSC Europe 25] Ekaterina Bubenko - Behind the Curtain: How Data Roles Collab...](https://cdn.slidesharecdn.com/ss_thumbnails/anmv6x8dstqbbzchoklr-ekaterina-bubenko-behind-the-curtain-how-data-roles-collaborate-in-the-ai-era-a-260123083019-4b252ec7-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Raul Cruz Bonilla - Harnessing GEN AI in Fashion, Luxury and ...](https://cdn.slidesharecdn.com/ss_thumbnails/me7nvup5thwqzwzblbvw-raul-cruz-harnessing-ai-en-luxury-260123083019-32ac5a43-thumbnail.jpg?width=640&height=640&fit=bounds)
Crowdsourced Probability Estimates: A Field Guide (FAIR Institute)
- 1.
- 2. 2 I’m Bob Best city ever! FinTechfirm Risk!
- 3.
- 4. “More than 91percent [of] clients victimized by ransomware”
- 5. 5 The research showsthat the ransomware epidemic affects over 91% of companies. What is the probability of this happening here?
- 6.
- 7.
- 8. 8 •Collect and vet research quality •Identifyand control for bias •Crowdsource expert assessments The Fix
- 9.
- 10. 10 •Gauge the level ofcalibration among experts •Weigh the expert opinions, combine (math or group) •Integrate into risk assessment Assessment
- 11. Expert Judgement • Interdisciplinary •Acquired knowledge • Predictive judgements • Use sparingly
- 12. Expert elicitation shouldbuild on the best available research and analysis and be undertaken only when, given those, the state of knowledge will remain insufficient to support timely informed assessment and decision making. - M. Granger Morgan
- 13. 13 “More than 91 percent[of] clients victimized by ransomware”1 • First Google result • “Clients” are MSP’s • Probably not statistically significant (not disclosed) Source: Datto’s 2016 Global Ransomware Report http://pages.datto.com/rs/572-ZRG-001/images/DattoStateOfTheChannelRansomwareReport2016_RH.pdf
- 14. 14 “More than 91 percent[of] clients victimized by ransomware”1 • First Google result • “Clients” are MSP’s • Probably not statistically significant (not disclosed) Source: Datto’s 2016 Global Ransomware Report http://pages.datto.com/rs/572-ZRG-001/images/DattoStateOfTheChannelRansomwareReport2016_RH.pdf
- 15. 1 infection lastyear
- 16.
- 17. 17 Literature Review • Commonin Social Sciences • Cornerstone of any research project • Vet quality of sources
- 18. 18 • Gathered andread 12 reports on Ransomware • Vary from surveys, empirical studies, research • Excluded 6 Source: Measuring Ransomware, Part 3: Prevalence https://cyentia.com/2017/07/25/ransomware-p3-prevalence/
- 19.
- 20.
- 21. What actually happens inthe cyber threat landscape What we read about Availability Bias
- 22. Overconfident professionals sincerely believethey have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion. - Daniel Kahneman Overconfidence Effect
- 23.
- 24. “60% of smallcompanies that suffer a cyber attack are out of business within six months.” “80% of all cyber attacks originate from the inside” “75 percent of companies have experienced a data breach in the past 12 months” 24
- 25. Eliciting Experts • Crowdsourcing •Identified some bias • Using 6 sources
- 26. 15 respondents (SIRA andFAIR Institute) Self-selected InfoSec experts Finding Experts
- 27.
- 28. Control for Bias Determine Calibration Seed Questions General Trivia Questions Over Confident Discardor Weigh Lower Perfectly Calibrated Use Estimate Under Confident Discard or Weigh Lower 28
- 29.
- 30. Tally the Responses •Convert percentages to a decimal • Add up – this is “expected” number correct • Compare against total number correct* *From “How to Measure Anything in Cyber Risk,” | Doug Hubbard, Richard Seiersen
- 31. Name Question score Calibration Score Calibration Respondent 1 8/108.2 Perfectly calibrated Respondent 2 7/10 8.2 Slightly overconfident Respondent 3 8/10 8.6 Perfectly calibrated Respondent 4 7/10 7.4 Perfectly calibrated Respondent 5 8/10 7.7 Perfectly calibrated Respondent 6 6/10 7.2 Slightly overconfident Respondent 7 9/10 7.2 Underconfident Respondent 8 6/10 7.8 Overconfident Respondent 9 7/10 6.9 Perfectly calibrated Respondent 10 5/10 6.7 Overconfident Respondent 11 8/10 7.7 Perfectly calibrated Respondent 12 5/10 7.4 Overconfident Respondent 13 7/10 7.1 Perfectly calibrated Respondent 14 8/10 7.1 Perfectly calibrated Respondent 15 8/10 8.7 Perfectly calibrated Results
- 32. Respondent Calibrated MinMode Max Respondent 1 Yes 10 25 35 Respondent 2 No 27 32 34 Respondent 3 Yes 15 35 65 Respondent 4 Yes 1 5 36 Respondent 5 Yes 1 2 65 Respondent 6 No 20 25 40 Respondent 7 Yes 10 20 60 Respondent 8 Yes 1 50 100 Respondent 9 No 27 30 34 Respondent 10 No 25 31 35 Respondent 11 Yes 0 5 40 Respondent 12 No 5 10 20 Respondent 13 No 1 5 20 Respondent 14 No 5 35 80 Respondent 15 Yes 20 30 40 Probability Estimates
- 33.
- 34. Are they calibrated? •Discard probability estimates; or • Coach on ranges and calibration; or • Integrate into final assessment, but weigh lower Misunderstood the question, research or assumptions • Follow-up with the expert; review their understanding of the request • If a misunderstanding, ask for a reassessment Different world-view • Let the expert challenge your assumptions • Consider multiple risk assessments Checklist for Vastly Differing Opinions
- 35. 35 Source: Doran &Zimmermann 2009, Anderegg et al 2011 and Cook et al 2013.
- 36. Science is nota matter of majority vote. Sometimes it is the minority outlier who ultimately turns out to have been correct. Ignoring that fact can lead to results that do not serve the needs of decision makers. - M. Granger Morgan
- 37. Respondent Calibrated MinMode Max Respondent 1 Yes 10 25 35 Respondent 2 No 27 32 34 Respondent 3 Yes 15 35 65 Respondent 4 Yes 1 5 36 Respondent 5 Yes 1 2 65 Respondent 6 No 20 25 40 Respondent 7 Yes 10 20 60 Respondent 8 Yes 1 50 100 Respondent 9 No 27 30 34 Respondent 10 No 25 31 35 Respondent 11 Yes 0 5 40 Respondent 12 No 5 10 20 Respondent 13 No 1 5 20 Respondent 14 No 5 35 80 Respondent 15 Yes 20 30 40 Probability Estimates
- 38. Behavioral • Delphi Technique •Nominal Group Technique • Negotiation to reach a consensus Mathematical • Averaging • Linear Opinion Pool • Methods Using Bayes Methods for Combining
- 39. All Respondents, EqualWeight
- 40. 40 All Respondents, Weightedon Calibration
- 41.
- 42.
- 43. Free • Excalibur • R •Excel Paid • Model Risk • Crystal Ball • @Risk Software for Combining
- 44.
- 45.
- 46. • Everyone thatparticipated in my exercise • Wade Baker • Jay Jacobs • Cynentia Institute • Calibratedprobabilityestimates.org • Doug Hubbard • Richard Seiersen • SIRA • The FAIR Institute Thank You
Editor's Notes
- #2 Hi everyone – My name is tony – I work as the director, tech tisk at lending club in san Francisco Also involved with SF FI I’m going to be talking about probability estimates. This talk is part field guide – practical tools for eliciting many expert opinions – but it’s also a part cautionary tale. Sometimes risk analysts use expert opinion as a shortcut for data gathering, but don’t consider all the associalted problems that can undermine even the best intended risk assessment
- #3 Meet Bob. Bob works for a mid-sized, San Francisco-based FinTech company. Bob works as a risk analyst in his firm’s information security department. Bob’s happy. Bob is trained in quanatitive risk analysis and frequently provides reports that allows his management to make better decision and priortize risk treatment in meaningful ways.
- #4 Bob’s Board of Directors is very concerned with the risk of Ransowmare and the negative effects an infection would have on the company. Several members of the Board recently attended a cybersecurity conference and learned about the “Ransowmare Epidemic” and would like to know how this ranks with other risks in the enterprise. asked bob to perform risk analysis, he decided to enlist the help of an expert – someonein his information security department -- to assist him with the probability portion
- #5 In order to ascertain the probability of such an incident, Bob does a few things. He partners up with Natalie, an incident analyst that works in the company’s Security Operations Center - They gathers media reports on the ransomware epidemic, such as the headlines shown here - Find research reports that sais 91% of clients have been victimized by Ransowmare -Discover their own company had 1 infection last year Bob asks Natalie to review the research. Later that day, they meet again and he is ready to ask her for a probability assessment.
- #6 Bob asks natalie this question: The research shows that the ransomware epidemic affects over 91% of companies. What is the probability of this happening here? Natalie reviews the research report the 91% figure is from, reads some news articles, thinks on it then gives her assessment, there is a 100% probability of this occurring at bob and natalie’s company Bob is satisfied., packages up his research – reports the probability as 100% - finished the risk assessment -- and send it up to the board.
- #7 Bob present his assessment to the Board, and he gets some pretty tpugh questions. They arew really focusing in on the Probability portion of the assessment and are incredlous that a ransomware infection probability is 100% - a sure thing. Some Board memebrs are experienced in risk analysis, other are familiar with probability theory, so they ask Bob for his workpapers. They want to see the research, assumptions and detailed analysis. Bob shows them the reearch eport that cits the 91% figure, and recounts his workshop with Natalie. The Board agrees that this is not sufficient – the risk analysis does not have the rigor they expect, does not have enough data to back up the probability claim . also question why only one person on the Information Security team was consulted. What do the other analysts on the team think? what does the CISO think? They asked him to go back, examine the fundamental problems in his analysis, and perform it again.
- #8 Bob is sad. Bob is Sad Bob. But we like bob. We’re going to help him. What are the problems with Bob’s risk assessment; there are some serious issues with the research performed. He used 1 source for research, the 91% figure -- and it’s not good, not defendable - Didn’t recognize, identify or control for cognitive bias Relying on 1, expert to inform analysis. Bob didn’t elicit expert opinion of several experts, to challenge both his and Natalie’s assumptions common mistakes but even just one of them can really skew a risk analysis and misrepresent the entire range of risk. There are ways to fix this.
- #9 So How do we fix this? Gather and vet research – Take a very critical look at the sources we use for research; needs to be vetted for quality, identify bias, weed out junk Try to control for bias of the people giving expert judement Crowdsource probability estimates – what I mean by this is asking many people for their opinion, instead of just the risk analyst doing it their self (which often happens) or one person. I have done many, many many thousands of risk assessment and over the years I’ve really come to appreciate diversity of opinion that comes from asking many people for probability estimates. The more people you talk to, the better your underlying assumptions are, and the better your assumptions are, the better your risk assessment is---
- #10 One of the hallmarks of the good risk assessment is this: hand the same data, same research, similar experts from the same field, to a totally different risk analyst and they come up with the same or very similar results. That is a good, defendible risk assesment, THAT should be the bar we aim for.
- #11 I’m going to demo this for you all – I sent out a call for help to SIRA and FI and asked for a few things. 15 people answered the call, 15 experts – first I gauged their level of calibration, and asked them for a probability assessment on a the ransomware question. We’ll go over the results – and show a way to combine them together for use in a risk assessment Now this entire process I’m talking about – gathering research, gathering experts, calibrating them, combining the results – this is part of a larger discipline called
- #12 …eliciting expert judgement. It is interdisciplinary. medicine, biology, climate science, engineering, and, risk management many many more. Most interesting work is being done in medicine & climate science. This is not a one-sized-fits all approach however. Using expert judtement makes sense when you are trying to forecast something and there’s a high level of conjecture or interpretation of incomplete, missing data or there’s a degree of uncertainty,. For example -- There are risk managers – I’m one of them - now that are perforing assessments on data breaches and other cyber incidents on blockchain tech. Naysaysers say this is impossible because historical models on blockchain tech are not available, however we do have data on data breaches in general, distributed databases and also data on incidents re. certain business cases, such as payments or forex. One can use expert judgement to use the current data available and fill in the gaps. example of then you do want to use it An example of when you would not want to use this method – when assessing the risk of a group of bowwoers defaulting in loans. There is reliable data, statistical models for this
- #13 If there a single piece of advice to keep in mind when using expert judgement, it’s this quote from m. granger morgan. Morgan is currently a professor at Carnegie mellon and spend his career performing analysis on climate change, the energy system and forecasting oil and gas prices. He’s one of the leading experts on expert judgement. Morgan teaches us that expert opinion is not a replacement for research and analysis – it builds upon current research and analysis and is used for that last mile in decision making In other words, Use sparingly. Now let’s dig into each of Bob’s data sources and see how we can improve
- #14 This is Bobs primary source – it’s an interesting stat and at is from a Firm called Datto in a 2016 global ransomware report Scary stat, but there’s more than meets the eye impulse is to assume that 91% of companies are ransomware victims. If you reqd the fine print – these are not regular firms like the one Bob works at, they are MSP’s – huge intake of incidents Victimized mean? Pay the ransom? Outages? Extorted for bitcoin? Or does it also include av catching a link to a site that hosts malware. Also, another issue – not stat significant. In survey science, st significant means that people that constructed the survey did their best to randomize the respondents – minimize/control for response bias, enables both the survey firm and the reader to take the results and and apply to the larger and general population. Without that distiction, the survey results only apply to the the respondents themselves
- #15 This is something that’s completely missing in many security and tech surveys Survey science is real! All too often research is based on A twitter poll w/ a chance to win an itunes gift card. this is not defensible for real-world decisions Some people will literally say or do anything for a $5 gift card. Don’t believe me, beer money subredditt. Should scare you enough to take a second look at survey-based research
- #16 Second problem in bob's risk assessment i looking at only what happened last year It’s true that we can learn much about observing one thing, as described in doug Hubbard’s book, how to measure anything. The fact that Bob’s company – or any company - had 1 ransomware infection last year is interesting to me as a risk analyst. What is more interesting to me if what happened in the previous 5 years. And, using the word infection, without context, creates too much room for interpretation for the expert. Was it an old cryptolocker strain that was blocked by antivirus, or was a a muti-day, multi server infection and we had to pay 500 bitcoin to a criminal gang in Belarus?
- #17 Next issue is usage of the term “ransomware epidemic” in the question that bob asked the expert Not a lawyer, but lots of law and order, learned from jack mccoy this concept of leading the witness. If you want the defendant to answer the light was red, you say “idn’t it true the light was red?” instead of “what color was the light” Objection your honor, Leading the witness! If you want the expert to think there’s a ransomware epidemic, you ask them about the epidemic. Are ransomware infections increasing? Sure. Is it a big problem. Of course it is. If you are asking someone to forecast the probability of a future ransomware incident, we create bias by using a terms like this.
- #18 The solution lies in a method used in Social Sciences called the “Literature Review” It’s the cornerstone and kety element of any research project. Read all existing or subeset of research and articleson a certain topic Write this up and provides the direction for beginning your research Similar framework / risk Look at existing research on a subject. Evaluate it. Reserch methodology, applues to the question, trying to sell you something Valuable skill for risk analysts / better than letting expert fend for themselves
- #19 Turned to some exsting research from the cyentia institute Jay Jacobs at the cyentia institute performed a Meta analysis of 12 different studies on ransware prevelance rates Exactly what I’m referring to – vetting research, looking at methodology, turns out he excluded 6 Discussed whether research was based on data or a survey (as a risk analyst I favor data based) Presenting something that a risk analyst can actually use Mentioned that I elicited the help of SIRA and FI
- #20 Aggregated – extracted the most important data points out of each research, it’s an improvement over Bob’s single point research item I would like to see risk analysis trained on this skill; this is a crucial part of a risk analysis. There are many free to low-cost courses on Coursera and other places that teach reseach methodology, not to use and interpret data. I took this entire blog post, all of the data and send it to our 15 expert to read and start thinking abut how they woulduse this data to perform a problability asessment
- #21 Bias is prevelant everywhere, and especially where humans are making decisions. Risk assessment are no exception Identifying and controlling for bias is incredibly hard to do There’s a lot of bias out there – will refer you to evan’s talk – looking forward to an in-depth talk, Bredly want to touch on 3 forms of cognitive bias that that really affect the results of a risk assessment – remember if our bar we have set for ourselves is to give the same body of research to a different group of experts and risk analyst and achieve the same or very similar result, removing as much bias as possible is key The three forms of cognitive bias I want to touch on are availabilby bias, the overconfidence effect and last, and a form/ combination of anchoring/group think and confirmation bias that I have dubbed the “infosec folklore effect:
- #22 First let’s look at availability bias, or availability heuerestic It’s a mental shortcut – when making a decision, we tend to favor very recent information. Daniel kahneman and amos taversky have done an enerous aount of wok in this topic, and if you haven’t read “thnking fast and slow” by kahneman you should Excmple of how we see in in cyber risk: I was conducting a workshop a few weeks back Gathering expert judgement for a variety of incident remediation efforts, but focusing on incidents that cause system outages – availability risk People were laser focused on DDoS, cyber criminals, nation states, n korea Internal and external data shows that the most dangerous threat actor in this areas are sysadmins that dn’t follow change control Don’t read about this in the news, people don’t talk about it, infosec folks don’ blog about it, not focused on it - Control for: encourage people to think in long-term trends; tell them about the bias, some can conceputalize; bring in data and research instead of letting people rely on memory alone
- #23 The next cognitive bias is called the overconfidence effect, also researched by kahneman The the specific caseof risk analysis, overconfidence appears when you are eliciting expert judgement An example: Asking expert for the annualized probability of a data breach You want the best case, worse case and the most likely case You tell the expert however – here’s the catch – I need you to to be 90% confident that the "correct" answer falls in that range. Overconfidence effect come into play here because most people will overestimate their ability to give a correct estimate In other words they think they are right more often than they really are The underconfidence effect is also present – you think you are correct less often than you really are – but does not happen as often as over Good news – possible to correct for with technique called calibration, will touch on this with a real life example from the SIRA and FI volunteers
- #24 Last, there’s something that I’ve been styding for quite some time I call it the infosec folklore effect It’s part availability, part groupthink, part confirmation bias It’s present when thouse of us in InfoSec just believe a statistic to be true – regardless of contradictory evidence, or research that te;lls us otherwise Here are some examples
- #25 Read first: This is a very pervasive idea and the stat is based on nothing. It’s simply not true. Where are these businesses? Ok There’s Mt Gox, even hacking team is still in business. Even HB Gary is still in business. But you will see this quoted over and over and over Next has to do with Insider Threat. 80% of all… This is a perfect example of infosec folklore. There is data driven, empirical research that shows time and time again that this just isn’t true, but it still persists Worst yet, most infosec people believe this – so when they get an email asking them to fill out a survbey on insider threat and win a free iPad, and they get asked where do most of your incidents originate: inside or outside? No one is going to actually open up the incident logs – they want their iPad – availability huerestic kicks in and they answer Inside. So yet research paper based on surveys is published, perputating this myth. Last one: I’ve seen this in many places is is a misquote of several opinion surveys that floated around over the years It seems accurate, it seems plausible, seems like a great way for security vendors to sell you more products Dig into it and it defies logic: according to privacy rights clearing house there were 579 publically reportable data breaches In 2017 According to the us census burearu there are around 6 million companies in the US Avoidoing the infosec folklore effect is difficult, but c an be minimized with good reseatch, a good vetting process
- #26 Let’s go back to our experts now. We have the building block in place. I posted on SIRA and FI asking for help Read the cyentia institute blog post Participate in an exercise so I can see how calibrated they are – control for over/under confidence Answer one question on the prevelance of ransomware So We’re crowdsourcing this – we’re using many experts instead of just one.
- #27 Out of those15 experts – it was mostly sira and FI although I suspect there is a great overlap between the two Most are self-selected an an infosec expert, I actually had this as a question One topic I want to dig into is the 10 question trivia exercise, this is a very interesting technique
- #28 these are seed question. We are trying to control for the overconfidence effect. The vast majority oe people havbe this bias and have to work hard to over come it EXCEPT for three groups of people that show very litlt of this effect 1 – bookies, bookmakers. someone who takes bets, calcu;ates odds, pays put winnings. 2 - meterologist. Even though isn’t mostly computer aided, meterologists even today often have to take weather models of differing data and make a probability estimate such as there’s a 40% chance of rain 3 – Professional bridge players. Bridge has the distinction of being of of the few of not only games that requires a sophisticaled underdstanding of probabilities to win the game and the card plaer must be able to do this very quickly Reason why they are calibrated: they constant feedback on the quality of their prior estimate. For example, a bookmaker will know within hours or days if the odds they gave on a particular horse were good. A weather person will know within days or even hours if their weather forecast was accutate. us, on the other hand in cyber risk – I have to wait 5, 10 20 years to see if any my probability estimates came to fruition and it’s unlikely that I’ll receive any feedback, still work for the company that I provided the estimate to or even remember that I provided an estimate in the first place.
- #29 Self-calibration or calibration in a business setting is achievable. Hubbard and Roger Cooke, another thinker in this area, independently has done a ton of onfluential work on this topic We all know Hubbard from the hot to measure anything series Roger Cooke is a professor at delft university in the Netherlands and has written extensively on the usage of expert judgement in areas such as climate science The idea with seed questions to ask the expert general trivia questions, like what is Chandlers last name in Friends or what was invented first, asprin or the airplane, to determine someones level of calibration They don’t have to be InfoSec related at all to be effective Hubbard advocates these type of questions along with calibration training to improve expert judgement. Cooke takes it a step further and advocates for the use of seed questions – to be used every time you elicit expert judgement. Depending on the number of experts, one can either throw out un-calibrated experts of weigh their opinions lower
- #30 Here are the first two questions that I sent to SIRA and FI members. The first portion is a true/false question. The second is a self-assessment – how confident are you that you got the question right? The options range from the lowest – 50%, which is equivilant of a coin flip to, nearly certain you got the question right. We don’t care about the trivie question. We don’t care if you’re terrible at trivia and bomb out it ok – you just need to know that you bombed out, and people that are calibrated, will reflect this in the test.
- #31 10 questions total Used hubbards method in how to measure anything Convert percentages to a decimal Add up – this is “expected” number correct Compare against total number correct* There’s a caveat to this. Generally speaking, 10 questions is not enough to definitively tell if someone is calibrated - cooke recommends 50 – but according to hubbard, it’s enough to get an idea about over/under confidence
- #32 With that Here are the results – of the 15 people thay responded to the questionnaire and completed it, 9 were perfectly calibrated and 6 that were not, with varying degrees. What do you do with the uncalibrated people – there are several techniques I’ll cover soon. Of those that are not calibrated, over confidence is the majority at 5 1 was under confident
- #33 Next, I asked participants to answer one question on ransomware. Probability og a significant company impacting ransomware event Give a probability estimate – min, aka best case Max, aka worst case Mode, aka most likely value This represents the range of possibilities – improvement over a single point estimate ot a color, but also give the expert an opportunity to express their uncertainty about their estimate Tried to avoid any language that would bias the reader or lead them in a certain direction. Before I move to combining etimates, I want to briefly pivot to the topic of what to do with vastly different opinions
- #34 You might have noticed that most of the probability estimates are very similar to each other. Most experts have roughly the same estimate about what I asked them, and since it was done over the Internet and I don’t think any of you conferred with each other – who has the time for that? – I have every reason to believe that based on the best available research available today and drawing on everyone’s expertise as cyber security experts, experts came to very similar conclusions. Now, there are outliers however. I have seen this in a few risk assessments I’ve done this at Lending Club. One, or a few, have totally different estimates as the majority. What do we do with these?
- #35 Are they calibrated? Discard probability estimates; or Coach on ranges and calibration; or Integrate into final assessment, but weigh lower Misunderstood the question, research or assumptions Follow-up with the expert; review their understanding of the request If a misunderstanding, ask for a reassessment Different world-view Let the expert challenge your assumptions Consider multiple risk assessments With the world-view point, there’s a parallel outside ouf info sec
- #36 The field of climate science heavily relies on epert judgement and collecting probability and other kinds of estimates from scientlists. The vast majority – 97% - of climate scientists agree that humans are causing global warming. 3% do not. This is an example of a different worldview. The 97% of scientists have the same or similar worldviews, so it’s reasonable to combine and aggregate their estimates on the rate of global warming, temperature, etc. If the other 3% are included, they would skew the results – so they are put into their own assessment.
- #37 To finalize this point, quote from m granger morgan Science is not. Amajority vote Sometimes it’s outliers that are correct Us as facilitators need to factor this in – don’t throw away other opinions
- #38 Let’s go back to the probability estimates. How do we take these and combine them? There are two methods, each with pros and cons.
- #39 The first category is behavorial. This usually entails the facilitator working through the problem with people until they reach a concensus. Exact methods vary, from anynymous voting, group discussion, negotiating with each other. Pros: Fast, quickly get people on the same page – if someone misunderatnds a concept, other can help, mathless Cons: Obvious here. Very prone to groupthink. People will subconsciously or consciously adopt the same opitions as their leader/manager. I have also seen overbearing peple shout down others or disrespect diversity of opinions to the point where I questioned the validity of the entire results. You also may lose that different world view person I was talking about. They may not speak up, or if they do, they get flamed The other method is mathematical. The most common here is averaging
- #40 Set a baseline – all respondents equal weight This is a c
- #41 Assigned weights 3x as much value to those that were calibrated
- #42 Assigned weights 3x as much value to those that were calibrated
- #44 Links are long – will put up in a blog post
- #45 There’s a happy story for Bob. Fished risk assessment, presented back to the board and they are happy, and now bob is happy
- #46 To Conclude Two things you can do here. I would like to see quant risk shops, FAIR shops, bring some of these techniques into their programs Severl things we can do with this Many orgs do this now – certainly outside cybersecurity. Perhaps in cyber too Predictin markerts are a much larger, different version of this. Local riskathons – get a group of analysts together and work a problem until we have a good probability estimate for a list of incidents