Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lessons Learned in Automated Decision Making / How to Delay Building Skynet
Similar to Lessons Learned in Automated Decision Making / How to Delay Building Skynet (20)
Recently uploaded
Recently uploaded (20)
Lessons Learned in Automated Decision Making / How to Delay Building Skynet
- 1. #RSAC SESSION ID: Sounil Yu Lessons Learned in Automating Decision Making: Pitfalls and Opportunities (or How to Delay Building Skynet: A Cautionary Tale on Connecting AI with Robotics) MLAI-T06 @sounilyu
- 2. #RSAC @sounilyu Questions to ponder 2 How is AI/ML distinct from automation? How mature are our AI/ML and automated decision making capabilities? How mature do they need to be for security? What can we learn from failed cases of automated decision making in security? What guardrails should be considered for automated decision making until sufficient maturity is achieved? Age 5 Age 13 Age 18 Age 21 = ?
- 3. #RSAC @sounilyu Framework #1: Modified OODA Loop 3 Acting Decision Making Sense Making Sensing • Analytics • Artificial Intelligence • Machine Learning • Execution • Robotic Automation • Response Scripts • Defined Processes • Courses of Action • Orchestration • Sensors • Raw Telemetry • Big Data Artificial intelligence capabilities are distinct from robotic automation capabilities
- 4. #RSAC @sounilyu • A fictional computer system developed for the U.S. military featured in the Terminator movie series • Originally built as a “Global Information Grid / Digital Defense Network” • Later given command over all computerized military hardware and systems including the B-2 stealth bomber fleet and America's entire nuclear weapons arsenal • Strategy behind creation was to remove the possibility of human error and slow reaction time to guarantee a fast, efficient response to enemy attack What is Skynet? 4
- 5. #RSAC @sounilyu What Could Possibly Go Wrong? 5 Acting Decision Making Sense Making Sensing • A fictional computer system developed for the U.S. military featured in the Terminator movie series • Originally built as a “Global Information Grid / Digital Defense Network” • Later given command over all computerized military hardware and systems including the B-2 stealth bomber fleet and America's entire nuclear weapons arsenal • Strategy behind creation was to remove the possibility of human error and slow reaction time to guarantee a fast, efficient response to enemy attack security use case
- 6. #RSAC @sounilyu Framework #2: DARPA’s Perspective on AI https://www.darpa.mil/about-us/darpa-perspective-on-ai 6 Perceiving Learning Reasoning Abstracting RICH, COMPLEX AND SUBTLE INFORMATION ABOUT THE OUTSIDE WORLD TO UNDERSTAND WHAT’S GOING ON WITHIN AN ENVIRONMENT AND ADAPTING TO ITS CONDITIONS AND SITUATIONS BASED ON WHAT IS PERCEIVED TO PLAN / DECIDE BASED ON A SET OF PRESCRIBED OR IMPLIED RULES AND UNDERSTANDING WHY BY TAKING KNOWLEDGE OF ONE DOMAIN AND APPLYING TO OTHER DOMAINS TO CREATE NEW MEANINGS Notional intelligence scale Human Level
- 7. #RSAC @sounilyu Perceiving Learning Reasoning Abstracting Perceiving Learning Reasoning Abstracting First Wave of AI – Handcrafted Knowledge – Enables reasoning over narrowly defined problems – No learning capability and poor handling of uncertainty – Examples: Turbotax, Chess, Logistics, DARPA Cyber Grand Challenge – First generation SIEMs Second Wave of AI – Statistical Learning – Nuanced classification and prediction capabilities – No contextual capability and minimal reasoning capabilities – Examples: Voice recognition, Face recognition, DARPA Grand Challenge – Self Driving Cars – Statistically impressive, individually unreliable – Current generation SIEMs DARPA’s Perspective on AI https://www.darpa.mil/about-us/darpa-perspective-on-ai 7
- 8. #RSAC @sounilyu s/Machine Learning|Deep Neural Networks/Statistics/g 8
- 10. #RSAC @sounilyu Why statistical-based machine learning and neural networks DO NOT work for security… 10 Outside the Closed World: On Using Machine Learning for Network Intrusion Detection Robin Sommer, Vern Paxson, 2010 https://www.icsi.berkeley.edu/icsi/node/4511 “Our main claim is that the task of finding attacks is fundamentally different from these other applications, making it significantly harder for the intrusion detection community to employ machine learning effectively.” • Bounded vs unbounded environments • Inviolable rules vs shifting rules • Human adversaries deliberately try to shift the rules (i.e., novel attacks) “…[ML is generally not] suitable for finding novel attacks … Rather, the strength of machine-learning tools is finding activity that is similar to something previously seen…”
- 11. #RSAC @sounilyu Perceiving Learning Reasoning Abstracting DARPA’s Perspective on AI continued https://www.darpa.mil/about-us/darpa-perspective-on-ai Third Wave of AI – Contextual Adaptation – Systems construct explanatory models for classes of real world phenomena – Models explain decisions (cause and effect) – Understand why and why not leads to an understanding of when the system will succeed or fail leads to when to trust and why mistakes are made 11 Source: Robust Physical-World Attacks on Machine Learning Models By Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song Classified 100% of the time as… How do I know this really is a stop sign? Explainable model: • Red • Octagonal • At intersections
- 12. #RSAC @sounilyu Why Do We Need Better Decision Making? … Because of Deliberate Attempts to Fool Sensing and Sense Making… 12 Source: Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Computer Vision and Pattern Recognition (CVPR ’15), IEEE, 2015
- 13. #RSAC @sounilyu Framework #3: Classical Education Trivium 13 Remember: According to the DARPA Framework, this capability emerges in the Third Wave Rhetoric Convincing and persuading Bearing fruit in wisdom Applying and integrating subjects Dialectic/Logic Investigating the truth of opinions Gaining in understanding Explaining “why” and “how” Grammar Structures and rules Soaking in knowledge Memorizing a broad base of facts
- 14. #RSAC @sounilyu The “Age” of Machine Learning 14 Machine learning has not left this stage yet Rhetoric Convincing and persuading Bearing fruit in wisdom Applying and integrating subjects Dialectic/Logic Investigating the truth of opinions Gaining in understanding Explaining “why” and “how” Grammar Structure and rules Soaking in knowledge Memorizing a broad base of facts GRADES K-6 ELEMENTARY SCHOOL GRADES 7-9 JUNIOR HIGH SCHOOL GRADES 10-12 HIGH SCHOOL RHETORIC RHETORIC RHETORIC DIALECTIC DIALECTIC GRAMMAR DIALECTIC GRAMMAR GRAMMAR
- 15. #RSAC @sounilyu Children vs Fully Grown Adults 15 Acting Decision Making Sense Making Sensing Prefrontal Cortex • Coordinates and adjusts complex behavior • Controls impulse • Prioritizes competing and simultaneous inputs Fully Developed at Age 25 Fully Developed at Birth Amygdala • Fear • Emotion • Impulse • Aggression • Instinct Better Filters and Prioritization • Knowledge and Wisdom • Discernment • Model Based Perception Questionable Inputs • Peers, Memes, Social Media Fully Grown AdultsChildren “Algorithms” • Rational and logical • Explainable “Algorithms” • Irrational • Illogical
- 16. #RSAC @sounilyu Lessons Learned when Skipping Steps 16 Acting Sensing Acting Sense Making Sensing Acting Sense Making Sensing Decision Making
- 17. #RSAC @sounilyu Lessons Learned: Reflexive Stimulus – Response 17 Threat Intel Sharing – Blocking google.com – Null routing 0.0.0.0/0 Automated patching – NotPetya – Windows 10 1809 – The “I hope you made a backup” Update Guardrails: – Ensure sensor sources are trustworthy and reliable – Apply actions that are narrowly scoped – Have a kill switch ready if it goes beyond the scope – Make the action immediately reversible Acting Sensing
- 18. #RSAC @sounilyu Acting Sense Making Sensing Lessons Learned: Conditional response based on analysis, enrichment, and regression testing 18 Threat Intel Sharing – Would this new regex pattern create false positives by matching on anything else over the past 30 days? – Is there a unanimous verdict based on enrichment from multiple other sources? (i.e., what does VirusTotal say?) Automated patching after regression testing – Do all systems in the testbed continue to operate as expected after the patch? – Did any applications stop working after the patch?
- 19. #RSAC @sounilyu Lessons Learned: Conditional response based on analysis, enrichment, and regression testing 19 “The automation process has to leave a trail of logic behind decisions so humans can follow it up,” Sam Hamilton, Chief Scientist BAE Systems, Cyber Tech Group Sources: https://www.darpa.mil/program/cyber-hunting-at-scale https://defensesystems.com/articles/2018/08/17/bae-cyber-ai-tool.aspx The CHASE program seeks to develop automated tools to detect and characterize novel attack vectors, collect the right contextual data, and disseminate protective measures both within and across enterprises Sensing Acting Sense Making
- 20. #RSAC @sounilyu Acting Sense Making Sensing Lessons Learned: Conditional response based on analysis, enrichment, and regression testing 20 Automating funds transfers requests from emails to an electronic transfer system – Assumed that emails were only coming from legitimate sources – Assumed that copying action was purely mechanical and didn’t involve any further analysis or thought beyond mapping email content to the proper fields Guardrails – Use regression testing to ensure the outcomes are fully deterministic – Validate assumptions that no other decision making is actually needed or happening – Ensure entire process is well documented and understood by the operators
- 21. #RSAC @sounilyu Acting Sense Making Sensing Decision Making Lessons Learned: Conditional response based on business considerations 21 Bank of Valletta shuts down all of its operations after hackers broke into its systems and shifted funds overseas DoD responds to the Code Red worm by disconnecting NIPRNet from the Internet, resulting in the Army Corps of Engineers not being able to control the locks on the Mississippi River Guardrails: – Pre-establish thresholds where the costs of inaction are worse than the negative repercussions of action – Pre-determine authorities for actions and accountabilities for outcomes
- 22. #RSAC @sounilyu Comparing Frameworks… How Mature is AI/ML Today? 22 Acting Decision Making Sense Making Sensing Perceiving Abstracting Reasoning Learning Rhetoric Stage Dialectic Stage Grammar Stage OODA Loop DARPA Classical Education
- 23. #RSAC @sounilyu Summary of Guardrails: When might it be okay to enable automated decision-making? 23 AI/ML Robotic Automation Create a conscious mental chasm that you deliberately choose to cross when enabling automated decision-making Sensor Diversity Ensure sensor sources are trustworthy and reliable based on multiple sources of truth Established Thresholds Know when the costs of inaction are worse than the negative repercussions of action Algorithmic Integrity Ensure entire process and all assumptions are well documented and understood by the operators Brakes and Reverse Gear Have a kill switch ready if it goes beyond the scope and make the action immediately reversible Bounded Conditions Ensure decisions are highly deterministic and narrowly scoped using regression testing Authorities and Accountabilities Pre-establish authorities for taking action and accountabilities for outcomes
- 24. #RSAC @sounilyu “Apply” Slide 24 Within the next month – Start inventorying capabilities broken out by Sensing, Sense-Making, Decision-Making, Acting – Determine where automated decision-making may be happening within those capabilities Within the next 90 days – Review potential guardrails for automated decision-making Within the next year – Establish governance processes to ensure that systems with automated decision-making stay within those guardrails
- 25. #RSAC @sounilyu DARPA’s Perspective on Artificial Intelligence https://www.darpa.mil/about-us/darpa-perspective-on-ai AI is now so complex its creators can’t trust why it makes decisions https://qz.com/1146753/ai-is-now-so-complex-its-creators-cant-trust-why-it-makes-decisions/ Automation Should Be Like Iron Man, Not Ultron https://queue.acm.org/detail.cfm?id=2841313 How can we be sure AI will behave? Perhaps by watching it argue with itself. https://www.technologyreview.com/s/611069/how-can-we-be-sure-ai-will-behave-perhaps-by-watching-it-argue-with-itself/ AI is more powerful than ever. How do we hold it accountable? https://www.washingtonpost.com/outlook/ai-is-more-powerful-than-ever-how-do-we-hold-it-accountable/2018/03/20/e867b98a-2705-11e8-bc72- 077aa4dab9ef_story.html Artificial Intelligence Has A Problem With Bias, Here's How To Tackle It https://www.forbes.com/sites/bernardmarr/2019/01/29/3-steps-to-tackle-the-problem-of-bias-in-artificial-intelligence/ The case against understanding why AI makes decisions https://qz.com/1192977/the-case-against-understanding-why-ai-makes-decisions/ When computers decide: European Recommendations on Machine-Learned Automated Decision Making https://dl.acm.org/citation.cfm?id=3185595 Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR https://arxiv.org/abs/1711.00399 Explanation in Artificial Intelligence: Insights from the Social Sciences https://arxiv.org/abs/1706.07269 For further reading 25
Editor's Notes
- There is much talk of topics like artificial intelligence, machine learning, and automation within the security industry. We’re oftentimes led to believe that these capabilities will revolutionize our security practices. However, we need to be conscious of the *limits* of these capabilities before we entrust them with matters of importance. In today’s session, we will discuss how ML and automation are distinct and how they fit together through decision making. Many people combine these capabilities and use the terms almost interchangeably, especially if we are letting the machine also do automated decision making. It’s easy to fall into this trap. Doing so can be quite dangerous and can create many unintended consequences. So before we let machines do automated decision making, let’s understand the limits of these capabilities.
- Using a combination of multiple frameworks, I plan to show you how we can determine how far AI/ML has advanced and understand when it might be appropriate to connect it to automation through automated decision making. These frameworks can help us answer some of the following questions such as what is the difference between AI/ML and automation? How separable are those functions? How mature are these capabilities and how mature do they need be for certain security relevant use cases? There’s a long history of EPIC FAILS when it comes to automated decision making. Using these frameworks, what can we learn from those failures, particularly as it relates to security? And lastly, given the current level of maturity of these capabilities, what guardrails should we consider?
- So the first framework for consideration is the OODA loop, or a slightly reworded modified version of it, courtesy of the IACD program. When we look at most security stacks, they have elements of these four components: Sensing, Sense-Making, Decision-Making, and Acting. Sensing requires sensors, the collection of telemetry from those sensors, and the ability to get them to a point where the data is usable. Sensemaking is where we have analytics, and this is also where AI/ML goes. Next up is decision making. This is where we take information, and based on our knowledge, wisdom, and judgment, we hope to make the right decisions that define the next courses of action. And then lastly, in acting, you have the actual execution of the decision. If we use this construct, then it becomes clear that AI/ML is distinct from automation and there is a Decision-Making function that connects AI/ML with Automation. It’s important to consider these separately. When we conflate these terms together, we also risk connecting these capabilities together without a mature decision-making framework, which then may result in severe unintended consequences. So this first framework is intended to explicitly call out the function of decision making which separates out sensing making from acting, or AI/ML from automation.
- This next framework is from DARPA, and it gives perspectives on what specific factors drive higher levels of capability as it pertains to artificial intelligence. With DARPA’s framework, there are also four components, and they align partially with the OODA loop. The first factor is perceiving. How well can a machine consume large quantities of data about the world and understand what’s going on. That’s sensing and a little bit of sense making. The second factor is learning. How well can a machine take the information and start adapting to that environment. This is squarely in sense making. The third factor is reasoning. How well can a machine decide on what to do next and explain why it made that decision. This is squarely in decision making. Note here that DARPA does call out decision making separately from the other functions, but still bundles it all within the context of AI. The fourth factor is abstracting. How well can a machine take what it has learned and decided and apply it to an entirely different domain. There’s no analogue here with the OODA loop, but note here too that DARPA doesn’t include the action itself or the execution of the decisions as a part of AI.
- Now within DARPA’s view of AI, they talk about three waves. The first wave is based on handcrafted knowledge. This has a little bit of perceiving and lots of reasoning. The examples they give include things like tax preparation software and chess. Even the Cyber Grand Challenge that they ran only a few years ago fit into this category. Capabilities in this wave rely heavily on the handcrafted knowledge of experts who can codify their decisions into software and provide deterministic outcomes. For example, in tax prep software, it can take basic inputs in various formats and make highly deterministic decisions over them. First generation SIEMs perceive their world using common event and logging formats and reason over this data using various alerting rules that are handcrafted. These capabilities fit into this first wave fairly well. The second wave has a lot of perceiving and learning capabilities. Machine Learning and most of what people call AI today is in this space. The second wave can do some amazing things to find patterns similar to what has previously been seen and this pattern recognition has been useful to develop capabilities like voice and facial recognition, somewhat self-driving cars, and the current generation of SIEM products that we see today. But this second wave of capabilities is also very limited in its reasoning capabilities. And remember, this is where decision-making lives. Also, note that DARPA calls this statistical learning.
- And on that point, let’s talk about statistical learning for a moment. Statistics are helpful to spot similarity to known patterns, but statistics without context or explanation can be very dangerous. You may have heard this quote before. There are 3 kinds of lies: Lies, Damn Lies, and Statistics. Or this awesome book entitled, “How to Lie with Statistics” And did you know that 83.7% of all statistics are entirely made up?
- Statistics without context or proper explanation has its limitations. And likewise, systems that use statistical-based machine learning will have similar limitations. As I mentioned previously, statistics are helpful to spot similarity to known patterns and fitting a curve, but without context or explanation, the similarities may mean nothing. For example, here are some known patterns that are similar. It turns out that people who drown after falling out of a fishing boat correlates nicely with the marriage rate in KY. And our spend on science, space, and tech correlates really closely with suicides by hanging, strangulation, and suffocation. But we know that these are absurd, unexplainable correlations, but why do we know that? How would a machine know that?
- And when these techniques are applied to security problems, we have encountered similar challenges. This is a great paper on the use of machine learning for network intrusion detection. Note that this was written in 2010, well before all the craze that we’ve seen recently around AI/ML. This paper specifically talks about why it is so fundamentally different and significantly harder to apply ML to security problems. In a nutshell, it distills down to the differences in the nature of problems and the environment that we are in when it comes to security. ML works greated in closed worlds, where things are bounded and the rules don’t change. However, the Internet, and most enterprise environments, are unbounded environments. And the rules change all the time. Furthermore, attackers are constantly trying to change the rules through novel attacks. But as I mentioned before, ML isn’t really that great for finding novelty and outliers, particularly in open environments like the Internet. There are many things that are outliers, which is why we end up with so many false positives when it comes to machine learning. And when we tune our ML systems, we recognize that the costs and the consequences for a false negative are pretty high. In other words, we don’t want to misclassify a real attack as being normal. But as we try to ensure that we don’t have any false negatives, we typically end up with many false positives, which has its own associated costs in terms of analyst time. By the way, most any vendor in this space that makes a claim about ML in their product will state that they have very few false positives. This is just a play on words. They mean that everything that they alert on will be an anomaly, which they don’t deem to be a false positive, because well, it really is an anomaly. But the vast majority of these anomalies are benign. Only a few are malicious. Ultimately, we need these systems to help us find malicious anomalies that we already know the pattern for, but also those malicious anomalies that are novel, and all the while, filtering out the benign anomalies. But to reduce the false positive problem, these systems also need to explain *WHY* these anomalies are malicious.
- And so that takes us to the third wave of AI, which we haven’t reached yet. This wave is what DARPA calls contextual adaption. Here, we have similar perceiving and learning capabilities as in the second wave, but we see much better reasoning and abstracting capabilities. What this means is that for this third wave, we would expect to see systems that can explain its alerting and decisions, ultimately leading to a clearer understanding of causal factors, not just based on statistical correlation. Ultimately, such a system would give us a better understanding of why it classified something as malicious or why it was okay. This provides more predictability for when a system succeeds or fails and drives towards more trust in that system. And with security, we want more trustworthy systems. Our ML systems today have hundreds of inputs and no clear explanation for how it makes sense of those inputs. As an example, here’s an example of a simple attack that was made against a visual recognition system that made a stop sign look like a speed limit sign. If you asked an ML system why it thought it was a speed limit sign, you would struggle to get a comprehensible answer. But at this third wave, such a system should be able to tell you simply that it can differentiate a stop sign from a speed limit sign because the stop sign is red, octagonal, and at intersections.
- It’s difficult to test machine learning models thoroughly, and they are easy to deceive or confuse. This presents challenging new security problems in it of itself. But when it comes to security use cases, it is doubly important that we get this right because it can undermine our ability to have security at all. ML models are black boxes that can only be tested as a unified whole. They usually have a huge number of inputs, and so it’s not possible to thoroughly test even simple models with every possible combination, which leaves open the question of how a ML model will perform in a given situation. And if you have adversarial behavior that is trying to actively fool the sensing and sense-making capabilities of our security products, then we have all the more reason to ensure that we have a better, more mature, decision making framework.
- And so when it comes to the maturity of a decision making framework, let’s look at this last framework. Now, we homeschool our children, so in the process of learning how to teach kids, we also learned about this thing called the classical education trivium. It lays out the progression of how kids learn. First, you start with the Grammar stage. This is where you just soak up facts and raw information. Think of it as the roots and the trunk of a tree. It sets the foundation for everything that grows above it. Next, you go into the dialetic or logic stage. Here, it’s like the branches of a tree. It connects facts together and compares the truth of opinions to determine what is logically correct and what is not. This is also when kids really can start explaining why and how one truth connects to another truth. The last stage is the rhetoric stage, when kids grow in wisdom and can convince and persuade others, having integrated multiple subjects and discplines together. Now if you remember, according to the DARPA framework, the ability to explain why and how happens in the third wave, which means, we’re not quite there yet.
- Now, if you match these three stages of childhood learning to actual ages, you get this breakdown, where during the elementary school years, the focus is primarily on grammar with a little bit on rhetoric and a little bit on dialectic. This is also why some people call elementary schools… grammar schools. In junior high school, we focus more on dialectic, and then in high school, we focus more on rhetoric. But note again, if we need machines to be able to explain why and how, and our current technologies aren’t able to do that adequately, then we have to realize that our machines haven’t left elementary school yet. Think about that for a moment. What kind of decisions are you comfortable in delegating to kids in that age range?
- And to tie this to the OODA framework, when we compare these components to the abilities of children vs adults, we can see why we might want to take precautions when giving decision making and acting abilities to machines. Looking more specifically at the comparison, consider that the decision making engine for children is the amygdala, which is fully developed at birth. This decision making engine is driven by fear, emotion, impulse, aggression, and instinct. But for a fully grown adult, the decision making engine is the prefrontal cortex, and this is fully developed at age 25. (If you ever wondered why you can’t rent a car until age 25, that’s your hint.) It’s towards the forehead and it controls behavior, meaning that you can avoid having a tattoo on your forehead that say poor impulse control. It can prioritize inputs and provides impulse control. Now, an amygdala for a decision making engine might be fine if one’s sensing and sense-making are clean and unpolluted, but let’s look at that as well. For children, we have questionable inputs, like their peers, and memes, and social media. For adults, we might have similar inputs, but we’ve developed filters and better discernment of those inputs. And then for our algorithms, anyone who has a teenager probably can agree with me that their sense making may not make much sense. But for adults, we should expect them to be rational and logical and be able to explain their logic. So, consider now. With the current level of maturity in our machine learning systems and the type of hostile sensory inputs that we get from the Internet, and with unexplainable logic, what kind of decisions should we allow machines to make? To put it another way, if you combine ML/AI with RPA, is the system going to mimic the behaviors of fully grown adults or teenagers? Do we understand the behavior of these entities and do we have confidence that they will behave or make decisions in an appropriate manner? Where do we get that confidence from?
- To understand how we can gain that confidence, let’s now look at some lessons learned and from these lessons, determine how we can gain confidence and trust in our systems, even if some of the elements make up the system are not fully mature. As a way to represent what happens when we lack of maturity of a certain capability, I will depict the lessons learned as skipping steps in the sensing, sense-making, decision-making, acting process. When we skip steps, we need more guardrails. Let’s look at each of these scenarios with lessons learned from the security world and the guardrails that we might need for each.
- In this first scenario, we’re going straight from sensing to acting. This is like a reflex action, so basically like a stimulus response system. Mention that we want to do automated patching and automated blocking from threat intel, but we should make sure we have these guardrails in place first
- Mention that there should be warning signs if there is no decision making called out